Neural-net-extracted Go problems from pro games

[lightvector]

I just finished a new website today that I hope people will find interesting! I trained neural nets to model players of different ranks, and then took a large database of pro games and used them to extract positions when the next move would likely be instinctual for a typical human pro but less instinctual for a typical weaker player.

The result was this collection of Go problems:
https://neuralnetgoproblems.com/

Among other things, it seems to have a high density of "basic shape problems" from local open-space fights. Keep in mind that they are all auto-generated so may not be as clean as typical tsumego, and the ranks are only very approximate. Also feel free to PM me if you're having trouble getting it to load, getting things to work right on all browsers is hard

I'm excited to see what people make of it.

(also posted to reddit r/baduk: https://www.reddit.com/r/baduk/comments ... pro_games/)

[dfan]

This is a great idea and I'm excited to try it out!

[Kirby]

That looks really interesting; thank you!

Could you explain a bit about how you accomplished that? Specifically I'm interested in:

- How did you train the network?

- How did you generate the problems?

- How did you assign a rank?

Thank you.

The about page has a link to his github repo, which has some interesting information:
https://github.com/lightvector/GoNN

[lightvector]

That looks really interesting; thank you!

Could you explain a bit about how you accomplished that? Specifically I'm interested in:

- How did you train the network?

- How did you generate the problems?

- How did you assign a rank?

Thank you.

I used human pro games plus a lot of games from online servers by different ranks of players. I trained the network by using the architecture similar to the one in the latest commit in my Github that Kirby linked, except I made the neural net slightly bigger and added the rank and server combination as a one-hot-encoded input feature. Except with a dimensionality reduction gadget to force the neural net to embed the 60+ different server/rank combinations into a lower-dimensional space. It mostly did learn a smoothish embedding where nearby ranks were usulaly embedded to nearby points, which was a great sign that it really was learning how differently-ranked players played, since that means it managed to learn which ranks were similar to which other ranks entirely on its own without being told anything about the ordering of the ranks.

Armed with a neural net that "understood" what kinds of moves each rank of player was likely to make in a given position, I then filtered pro games for moves that the neural net thought were clear for pros (the neural net predicted a pro would be very likely to make exactly a specific move, and indeed the pro did make that move) and where the neural net thought they would not so clear for different weaker ranks of players. I had to playtest and iterate on the criteria for filtering quite a bit.

The final nominal ranks that the problem sets are labeled with are only loosely related to the ranks used in the filtering criteria. They were mostly determined by further testing with a few volunteers, along with some by-hand fudging, particularly on the lower end of the scale, which was initially much too hard.

[daal]

Incredible! Great work!

[jlt]

Very interesting! Two questions:

1. How many problems are there in your database?

2. How did you determine that a move is "possible/unlikely, might have been the pro move in other situations"?

[lightvector]

* Around 127000 problems.

* The neural net assigned a nontrivial amount of probability to a typical pro playing those moves, even though they were not the game move and the amount of probability assigned was far less than the game move.

[Calvin Clark]

Impressive. There seems to be quite the variance for a given level. This showed up as a 3d problem, but if to are on the board, I have doubts. (Problem 3d-13598.sgf)

But this is definitely a useful project, and I hope you and your collaborators can take it further. Wasn't the chess.com tactics trainer created by computer analysis? (Pre-neural net engines, I think).

[lightvector]

That seems to be a case where the neural net is just wrong that the move would be unlikely for a lower-ranked player. I'm not sure I can do much about that, except to train better nets if I were to try to regenerate the problem sets. Neural nets "blunder" too I guess. It does seem to be different between different rotations of the neural net in this specific case. Maybe averaging across rotations would be a thing to try.

I did notice that there was high variance, which is why I labeled it as "Avg difficulty" rather than simply "Difficulty". Although I agree this is a little too much variance.

If I were to make up a reason why in retrospect still having some fraction of easy problems is good, it would be that in a real game, many of the moves *are* easy, so maybe it's good to to keep you on your toes and not always be expecting something non-obvious.

Edit: Going ahead and trying a problem generation run where I average multiple rotations of the neural net together, to see if it results in some reduction of the frequency of this.

[Knotwilg]

I tried out a few problems at 14k level and 1d level.

First of all, it's a fantastic piece of work! This might just be my new favorite online resource.

One feature that may even improve it is: the level of the problem is a bit of a giveaway to me. In a 14k level problem, I know that we're looking for the basic move. A 1d problem may have a more intricate solution and the first move I think of is probably not the one to play.

Would it be feasible to offer random level problems?

[lightvector]

Sure! Hopefully the new problem generation run will be finished by the end of this weekend (it takes a while to complete), and at that point I'll see if it somewhat reduces the incidence of "super-easy" problems in harder sets that Calvin noticed, although surely there will still be plenty of it. But given that I'm trying to do that, seems worth also adding a button that puts all of the problems together in a giant pool, or something like that.

I'll also push a few bugfixes at that time too, as right now you can get into some strange glitched states with mixing clicking on the board and redo after undo. Hopefully it should be up by the end of this weekend, if not, then the first half of next week.

[Bill Spight]

Let me add my voice to the chorus:

Wonderful idea! Great job!

[lightvector]

Done! Regenerated the problems (although I find any difference hard to notice), fixed some UI bugs, and added an "all" button that simply has all the problems from every other set mixed together.

[Ian Butler]

Fantastic project. Bravo!
Real helpful set of whole board middle game problems.
Thanks a lot for making this!