Using PyTorch for the neural networks

 I've been thinking a bit about whether GPUs are useful for neural network calculations with backgammon bots. It's not totally clear: there is overhead in shipping memory to the GPUs as a cost, measured against the benefits of parallelization.

That said, it is not very hard to check: there are nice open source packages available for neural networks that already support GPU calculations.

A simple one that seems decent for our purposes is PyTorch. This is a Python package for representing neural networks, and, as desired, it supports GPUs through CUDA (which works if you've got an NVIDIA GPU, as I do on my desktop machine).

There are two possible benefits of the GPU. First, evaluation of the network might be faster - so that it's quicker to do bot calculations during a game. Second, the training might be faster - that doesn't show up in the post-training game play, but could let you experiment more easily during training with different network sizes and configurations.

For the training calculations, the GPU really only helps with supervised learning - for example, training against the GNUBG training examples - but not with TD learning. That's because TD learning requires you to simulate a game to get the target to train against - you don't know the set of inputs and targets as training data before the TD training starts.

I constructed a simple feed forward neural network with the same shape as the ones I've discussed before: 196 inputs (the standard Tesauro inputs with no extensions), 120 hidden nodes (as a representative example), and 5 output nodes.

Then I just timed the calculation of network outputs given the inputs. The PyTorch version was noticeably slower even than my hand-rolled Python neural network class (which uses numpy vectorization to speed it up - not as fast as native C++ code but with a factor of 2-3). This is true even though the PyTorch networks do their calculations in C++ themselves. These calculations didn't use the GPU, just the CPU, in serial.

Then I tried the PyTorch calculations parallelizing on the GPU. I've got an NVIDIA GeForce GTX 1060 3GB GPU in my desktop machine, with 1,152 CUDA cores. It was slower, with this number of hidden nodes, than the CPU version of the PyTorch calculation. So the memory transfer overhead outweighed the parallelization benefits in this case.

I tried it with a larger network to see what happened - as the number of hidden nodes goes up, the PyTorch evaluations start to outperform my numpy-based evaluations, especially when using the GPU

So for post-training bot evaluations, it doesn't seem like using the GPU will give much improvement, if we're using nets of the size we're used to.

The GPU does, however, make a massive difference when training! In practice it's running about 10-20x faster than the serial version of the supervised training against the GNUBG training data. I'm excited about this one. And I can always take the trained weights off the PyTorch-based network and paste them onto the numpy-based one, which executes faster post training.

Custom GPTs can run only Python code

I just discovered that OpenAI's custom GPTs cannot run compiled code like C++ - just Python.

That's an interesting limitation of these things, and one that I suspect will disappear in the not-so-distant future, probability to do with how simple the temporary environments need to be that execute the code.

In any case, it means my backgammon bots - for now! - will need to be pure Python. That said, the Python environment does have most of the standard numerical Python packages like numpy and pandas, and more advanced packages like scipy and sklearn that themselves contain some machine learning functionality. So maybe I'll try to build an sklearn-based neural net and see if that works better than my hand-rolled one that uses numpy vectorization.

The tutorbot begins to take shape

 I've named my custom GPT the "Backgammon Tutorbot". 

It's now getting a bit better. You can tell it a position in text, like "show me the starting position", and it knows what that means, shows you a proper image of the position, and internally knows what checker layout it corresponds to.

You can then (slowly!) step through a game. If you ask it, for example, "show me the position after a 5-1", it'll call the backgammon bot Python code to figure out the best move for a 5-1, then show you an image of the resulting position, with a bit of commentary. And it remembers the new position as the current one.

Next, if you say "show me the position after the opponent rolls a 6-2", it'll figure out the opponent's best 6-2, then show you an image of that position, with some commentary.

And so on, until the game is over (or you hit the GPT-4 time cap, which I did a bunch).

So it's getting better at the mechanics of representing and advancing a real game. It's quite interesting to use the chat interface as the UI rather than a traditional application. In some ways it's much slower - but it's also chatting in regular English, with a lot of flexibility about how you actually ask your questions. The chatbot is very good at dealing with that kind of flexibility.

But it's still not very good at commentary yet. I've tried uploading one doc with human discussions of opening moves, but that's very early still, and no results to report.