Testing a new contact/crashed input: hit & cover

I tried a new input for contact & crashed: like the hitting shots input, but where the opponent covers the point with two (or more) checkers when hitting, without leaving another blot elsewhere.

I actually broke it into two pieces: one for the odds of being hit & covered in the opponent's home board, and one for everywhere else. (And of course, both inputs were applied to each player, so four new inputs.)

I started with a random-weights version of Player 3.2 and added the new inputs, and trained it using supervised learning on the GNUbg training set.

The player converged after 150 epochs to Contact ER 14.0, Crashed ER 12.4, and Race ER 2.08. This compares to 14.0, 12.6, and 2.08 for Player 3.2. So marginally better on Crashed ER but otherwise identical.

Since we know that Crashed ER does not matter much in terms of game play, this new player really is almost identical to Player 3.2.

The conclusion must be that these new inputs do not add noticeable value. So I will not include them going forward. The hunt continues!

Player 3.2 1-ply benchmark

I ran my 1-ply version of Player 3.2 through the GNUbg benchmarks: Contact ER 11.0, Crashed ER 10.8, Race ER 1.67. That compares favorably to the 0-ply scores of 14.0, 12.6, and 2.08, but still worse than the 0-ply GNUbg scores.

Bug in benchmark calculation: fixed

I just discovered a bug in my benchmark calculation: if the player's board was not found in the list of best-five rolled out boards in the benchmarks, it was assuming an equity error of zero instead of the worst equity error in the list. So it was making those edge cases look much better than they should have, and skewing the average ERs a bit better.

Below is a table of the corrected benchmark results for the nine players in the original results, plus four new ones:

Player	GNUbg Contact ER	GNUbg Crashed ER	GNUgb Race ER	PubEval Avg Ppg	Benchmark 2 Avg Ppg
GNUbg 0-ply	10.5	11.0	1.01	0.630	-
Player 3.2	14.0	12.6	2.08	0.547	0.131
Player 3.2q	33.7	26.8	2.40	0.146	-0.282
Player 3.1	14.9	14.2	2.08	0.548	0.106
Player 3	14.9	14.2	2.67	0.550	0.108
Player 2.4	18.2	19.3	1.98	0.480	0.072
Player 2.4q	38.3	41.1	4.70	0.119	-0.283
Player 2.1	18.7	19.8	2.01	0.460	0.069
Player 2	20.5	30.0	2.09	0.442	0.021
Benchmark 2	21.5	23.7	5.44	0.432	0
Benchmark 2 (10)	42.7	37.5	13.20	0.064	-0.418
Benchmark 2 (40)	26.2	25.9	5.94	0.330	-0.067
Benchmark 1	23.0	24.5	9.66	0.351	-0.101
PubEval	44.1	49.7	3.54	0	-0.437

Redoing the one-variable regressions on the corrected & expanded data:

Metric	PubEval Ppg vs Contact ER	PubEval Ppg vs Crashed ER	PubEval Ppg vs Race ER	BM2 Ppg vs Contact ER	BM2 Ppg vs Crashed ER	BM2 Ppg vs Race ER
Slope	-0.0182	-0.0161	-0.0268	-0.0189	-0.0165	-0.0275
Intercept	0.8065	0.7638	0.4631	0.3969	0.3468	0.0362
R-Squared	98.8%	83.5%	22.5%	98.0%	80.6%	22.3%

Redoing the multivariate linear regression:

Benchmark	Intercept	Contact ER Slope	Crashed ER Slope	Race ER Slope	R-Squared
PubEval	0.8037	-0.01980	+0.00133	+0.00202	98.9%
Benchmark 2	0.3945	-0.02049	+0.00209	-0.00246	98.4%
Benchmark 2 (only good players)	0.4102	-0.01756	-0.00063	-0.00694	98.5%

The main conclusion: Contact ER mostly determines cubeless money play score. The single regression of score against Crashed ER has a relatively high R^2, but that is only because Crashed ER is highly correlated with Contact ER. When properly separated with the multivariate regression it becomes clear that Contact ER is the only measure that really matters.

Player 3.2q: a new filter strategy

For multiple plies I was previously using Player 2.4q as a coarse strategy to trim down the list of moves that I need to run the more expensive 0-ply evaluation on.

I've now moved that to a new coarse strategy: Player 3.2q. This is just like Player 3.2 but with five hidden nodes instead of 120 for each of the three networks.

It scores Contact ER 33.7, Crashed ER 24.8, and Race ER 2.40. Significantly stronger than Player 2.4q. (Benchmark scores updated after fix to benchmark calculation.)

In 100k cubeless money games against Player 2.4q it scores +0.210ppg +/- 0.004ppg.

Player 3.2: adding a crashed network

In my earlier tests I could not find a significant benefit of using a crashed network, even following the GNUbg definition of crashed.

But that was using TD learning on self-play, and I wanted to see whether the improved training using supervised learning would find a significant benefit.

Indeed it did: a noticeable benefit, so I spawned my new best player, Player 3.2, which is like Player 3.1 but adds a crashed network.

Its benchmark scores: Contact ER 14.0, Crashed ER 12.6, and Race ER 2.08 (the same as Player 3.1). (Starting to get close to GNUbg 0-ply! Extrapolating from the regression on Benchmark 2, GNUbg should beat Player 3.2 by only around 0.05ppg.) (Benchmark scores updated after fix to benchmark calculation.)

Adding the crashed network made the biggest difference to the Crashed ER, with a noticeable but smaller improvement in Contact ER. This is a bit different to the experience of the GNUbg team, who saw a relatively small improvement in Crashed ER but a big difference in Contact ER after adding a crashed network, since the contact network could focus its optimization on a smaller set of game layouts.

That said, my player's performance is still reasonably far from GNUbg's. I think there's still some work to do on extra inputs to bridge the gap.

Player 3.2 summary:

• 120 hidden nodes.
• Trained using supervised learning on the GNUbg training databases, starting with the Player 3.1 weights (using the contact weights as the initial guess for the crashed weights).
• Contact, crashed, and race networks.
• One-sided bearoff database used when both players have all checkers in their home boards.
• Contact inputs as per Player 2.4, with Berliner prime and hitting shot inputs in addition to the original Tesauro inputs.
• Crashed inputs are the same as contact inputs.
• Race inputs are the original Tesauro inputs plus the 14 extra inputs added with Player 3.1.

I trained it for 128 epochs, with an alpha schedule of 1 until the 8th iteration, then 0.32 until the 20th iteration, then 0.1 until the 60th iteration, then 0.032 until the 100th iteration, then 0.01 afterward.

In 100k cubeless money games, Player 3.2 scores +0.021ppg +/- 0.004ppg against Player 3.1. 

In 40k cubeless money games against Benchmark 2 it scores +0.131ppg; the prediction from the multivariate regression is +0.152ppg.

In 40k cubeless money games against PubEval it scores +0.574ppg; the prediction from the multivariate regression is +0.547ppg.

Player 3.1: expanded race inputs

In the past I tried a couple extra inputs for the contact network; now I'm trying a new set of inputs for the race network.

This follows what Joseph Heled did for GNUbg's race network: instead of using a single input that represents the number of checkers borne off, using 14 separate inputs. The i'th input is 1 if the number of borne-off checkers is greater than or equal to i, and 0 otherwise. Splitting out the number into separate inputs like this is what the regular Tesauro inputs do for number of checkers on a point, and it lets the neural network discover more complex nonlinear dependencies.

Player 3.1 implements this expanded set of race inputs. Its contact network is the same as Player 3, so its Contact and Crashed ER are the same as that player (14.9 and 14.2). Its Race ER is improved: 2.08, vs 2.67 for Player 3. (Benchmark scores updated after fix to benchmark calculation.)

So a relatively small improvement, but a noticeable one worth around 0.004ppg in score.

Player 3.1 summary:

• 120 hidden nodes.
• Trained from scratch (uniform random inputs in [-0.1,0.1]) using supervised learning on the Race GNUbg training database, not TD learning. 
• Contact and race networks. No crashed network.
• One-sided bearoff database used when both players have all checkers in their home boards.
• Contact inputs as per Player 2.4, with Berliner prime and hitting shot inputs in addition to the original Tesauro inputs.
• Race inputs are the original Tesauro inputs plus the 14 extra inputs per player as described above.

I trained it for 110 epochs, with an alpha schedule of 1 until the 8th iteration, then 0.32 until the 20th iteration, then 0.1 until the 60th iteration, then 0.032 until the 100th iteration, then 0.01 afterward. Really it converged pretty quickly - after about 20 iterations. I let it run longer to see if would improve incrementally with smaller alpha, but it did not.

Player 3: training with supervised learning

I tried training a player using supervised learning on the GNUbg training databases, instead of training using the TD learning that I have used up until now.

I took a setup like Player 2.4 and trained it from scratch (uniform random weights in [-0.1,0.1]), one version with 80 hidden nodes and another with 120 hidden nodes.

I'm naming the 120-node version Player 3, since it has a significantly better performance than my previous best player. The step from TD learning to supervised learning made quite a large difference in performance, consistent with the experience of the GNUbg team.

Its benchmark scores: Contact ER 14.9, Crashed ER 14.2, and Race ER 2.67. (Benchmark scores updated after fix to benchmark calculation.)

Moving from 80 to 120 hidden nodes makes a significant difference, unlike in earlier tests on TD-trained players.

Player 3 summary:

• 120 hidden nodes. I also trained an 80-node version which performed noticeably worse. I may occasionally refer to the 80-node version as Player 3 (80).
• Trained from scratch (uniform random inputs in [-0.1,0.1]) using supervised learning on the GNUbg training databases, not TD learning. 
• Contact and race networks. No crashed network.
• One-sided bearoff database used when both players have all checkers in their home boards.
• Contact inputs as per Player 2.4, with Berliner prime and hitting shot inputs in addition to the original Tesauro inputs.
• Race inputs are the original Tesauro inputs.

The training approach: I combined the crashed & contact training sets into a single set, and kept the race training set as is. One "epoch" of training (borrowing Joseph Heled's terminology): randomly order the crashed/contact training set and train the contact net with that; then randomly order the race training set and train the race net. Alpha is kept constant for an epoch, but can vary between epochs. If Contact ER does not improve over an epoch, the training sets are (randomly) re-ordered.

I trained the 80- and 120-node versions using supervised learning on the GNUbg training databases. The schedule for alpha was 1 until epoch 8, 0.32 until epoch 30, 0.1 until epoch 60, 0.032 to epoch 100, and 0.01 afterwards.

I trained the 80-node version for 112 epochs. It scored Contact ER 14.8, Crashed ER 12.3, and Race ER 1.59.

The 120-node version was trained for 100 epochs. It scored Contact ER 13.9, Crashed ER 11.8, and Race ER 1.56 - significantly better than the 80-node version.

Some head to head match results for Player 3 (the 120-node version), 100k cubeless money games (standard error 0.004ppg):

• Against the 80-node version: +0.018ppg
• Against Player 2.4: +0.047ppg
• Against Benchmark 2: +0.120ppg. The estimate from the multivariate regression is +0.129ppg.
• Against PubEval: +0.546ppg. The estimate from the multivariate regression is +0.529ppg.