Wednesday, January 25, 2012

Multiple ply benchmark

Joseph Heled, one of the key GnuBG developers and the one responsible for most of the neural network pieces, has run some experiments with multiple plies against pubEval to see a comparison.

For 1k 1-game matches, 2-ply beat pubEval 75% of the time. For 3k 1-game matches, 1-ply beat pubEval 74% of the time. For 10k 1-game matches, 0-ply beat pubEval 71.9% of the time.

Those were single-game matches where all wins counted for 1 point, so not money games where gammons and backgammons count extra.

The 1- and 2-ply matches were with smaller numbers of trials, so have more statistical uncertainty attached. But roughly, adding multiple plies should improve performance by 2-3% in probability of win.

I haven't seen comparisons of 1- or 2-ply GnuBG vs its 0-ply cousin, but it seems reasonable to assume that they will win by comparable margins (2-3%).

No comments:

Post a Comment