The next test on the Benchmark 2 player is to see how the performance depends on the number of hidden nodes. In general one expects performance to improve as the number of hidden nodes increases, but that is balanced by a higher computational overhead and potentially more difficulty converging during training.
The Benchmark 2 player described earlier used 80 hidden nodes.
10 Hidden Nodes
I trained another version with 10 hidden nodes (from initial small random weights this time) and benchmarked it against the trained 80-node version. It converged relatively quickly: in about 75k training runs (all at alpha=0.1):
Its converged performance against the 80-node benchmark, however, was quite poor. In a 10k-game match it scored -0.417ppg on average and won 36.9% of the games. It won +0.056ppg against pubeval in a 10k-game match and 51.7% of its matches. (Note: pubeval results were updated after the pubeval player was fixed.)
40 Hidden Nodes
The next version had 40 hidden nodes. Benchmarked against the 80-hidden node version in training:
It used alpha=0.1 for the first 200k runs then switched to alpha=0.02, and seemed quite converged by 250k training runs.
In a 10k-game match against the 80-node benchmark it scored -0.064ppg and won 47.5% of the games. It scored +0.325ppg against pubeval in a 10k-game match and won 60.7% of the games. (Note: pubeval results were updated after the pubeval player was fixed.)
120 Hidden Nodes
The final test was for 120 hidden nodes. Benchmarked against the 80-hidden node version in training:
As with the other training, I started with alpha=0.1 and dropped to alpha=0.02 after 200k runs. The network looks converged after roughly 300k training runs.
The equity difference vs the 80-node benchmark was looking quite small so for a comparison I ran 30k games to reduce the statistical error (vs 10k games for the previous two tests). In those games the 120-node player score -0.018ppg and won 49.0% of the games.
Conclusions
The 80-hidden node benchmark seems to be the best of the 10, 40, 80, and 120-node choices. It is clear for the 10- and 40-node versions that adding more nodes improves performance. It is somewhat surprising that moving to 120 nodes showed a decline in performance, though the difference was small. It seems like convergence in the hidden node direction is around 80 nodes for this type of setup. I will continue to use the 80-node version of Benchmark 2 as a future benchmark.
The Benchmark 2 player described earlier used 80 hidden nodes.
10 Hidden Nodes
I trained another version with 10 hidden nodes (from initial small random weights this time) and benchmarked it against the trained 80-node version. It converged relatively quickly: in about 75k training runs (all at alpha=0.1):
Its converged performance against the 80-node benchmark, however, was quite poor. In a 10k-game match it scored -0.417ppg on average and won 36.9% of the games. It won +0.056ppg against pubeval in a 10k-game match and 51.7% of its matches. (Note: pubeval results were updated after the pubeval player was fixed.)
40 Hidden Nodes
The next version had 40 hidden nodes. Benchmarked against the 80-hidden node version in training:
It used alpha=0.1 for the first 200k runs then switched to alpha=0.02, and seemed quite converged by 250k training runs.
In a 10k-game match against the 80-node benchmark it scored -0.064ppg and won 47.5% of the games. It scored +0.325ppg against pubeval in a 10k-game match and won 60.7% of the games. (Note: pubeval results were updated after the pubeval player was fixed.)
120 Hidden Nodes
The final test was for 120 hidden nodes. Benchmarked against the 80-hidden node version in training:
As with the other training, I started with alpha=0.1 and dropped to alpha=0.02 after 200k runs. The network looks converged after roughly 300k training runs.
The equity difference vs the 80-node benchmark was looking quite small so for a comparison I ran 30k games to reduce the statistical error (vs 10k games for the previous two tests). In those games the 120-node player score -0.018ppg and won 49.0% of the games.
Conclusions
The 80-hidden node benchmark seems to be the best of the 10, 40, 80, and 120-node choices. It is clear for the 10- and 40-node versions that adding more nodes improves performance. It is somewhat surprising that moving to 120 nodes showed a decline in performance, though the difference was small. It seems like convergence in the hidden node direction is around 80 nodes for this type of setup. I will continue to use the 80-node version of Benchmark 2 as a future benchmark.
No comments:
Post a Comment