On using Lizzie + LZ/KataGo/minigo etc

yoyoma · #1

Figured this topic could use it's own thread. The simplest way is:

"o" to load game record file (note Tygem etc autosaves game records where the program is installed)
"a" to analyze the game, pick your playouts based on how much patience you have ;-)

I pointed out to Bill Spight that if the Bot doesn't consider the move actually played, you can just play it and force it to analyze that move. In this use case you already have the entire game record, so of course the Bot is forced to analyze the actual game moves, and it will give accurate(tm) winrate evals, along with how big a mistake it thinks your actual move is.

But you can also just scroll through the game with pondering on ("space"), and for practical use the finer points of how many playouts each move gets don't matter. You'll quickly notice when the bot is confused about a situation and needs more playouts -- the winrate will start fluctuating wildly every time you go to a new move, particularly if the bot did not expect it.

The bots are very strong, but not perfect, and it's not clear what exactly the winrates the bot gives means, or how accurate they are. Well humans are still very quick to learn generalizations, so if you just use the bots for awhile and you will get a natural feeling for what it means when a bot gives a 60% or 90% winrate. Also you'll notice that as the game gets closer to the end the winrates will get more extreme.

I've been using KataGo, and it supports variable komi. Using Lizzie you can use "e" to open the GTP terminal, and type "komi 7.5" etc. Using this method you can ask KataGo winrates for various values of komi:

8.5 W 55%
7.5 W 55%
7.0 W 50%
6.5 W 45%
5.5 W 45%

KataGo thinks 7.0 is fair komi. Note for area rules score differences are always(tm) odd, e.g. white ahead by 5, 7, or 9. This makes the winrates for komi of 7.5 and 8.5 the same.

Also note KataGo calls a 2 point mistake a 10% hit to your winrate! Keep this in mind when judging how serious a mistake is when analyzing games.

For my own games, I get the idea that reading is the most important factor in winning or losing games. It also happens that I play in the wrong part of the board or have the wrong whole board plan, but more often I don't see some tactic that allows me to cut, or allows me to tenuki etc.

The bots also reinforce my opinion that while debating the finer points of early opening theory is fun, it isn't particularly important to your winrate. Maybe it's because I already learned the basics of opening theory, but many of the debates I see about if a move should be high or low, or 3-space or 4-space extension simply don't matter that much. Move 2 most certainly did not lose the game! :razz:

I think this is a great thing because it allows everyone to pick the moves that they like. Don't worry if the bots say early 3-3 is the best, and you can keep splitting the side if you prefer that and accept a 2% winrate difference. But following trends is fun too so I've been playing early 3-3 every chance I get! OTOH I've traditionally played mostly 3-4 points in my opening, and I continue to do so. I just like it better than 4-4.

Bill Spight · #2

May I add my experience with the Elf commentaries on the GoGoD database games?

One thing the Elf team did not do was to calculate differences in winrate estimates. Nor did they replicate the game tree search. I don't know what their pruning criteria were, but they plainly pruned branches, and that was in general a good thing, IMO. No point losing ourselves in a maze of calculations, as Znosko-Borovsky put it.

If the play in the game was Elf's top choice, they indicated that, and sometimes added variations. If it was not Elf's top choice, they always included a variation with that choice, along with the Black winrate estimate and the number of playouts. The game play also has a winrate estimate and number of playouts, but the two may not be related. For instance, sometimes the game play was not on Elf's radar, and has 0 playouts. Well, you can't get a good winrate estimate from 0 playouts. Where does that estimate come from? Inspection reveals that it comes from the winrate estimate of Elf's reply to the game move. How confident can we be of that estimate? The number of playouts reflects the confidence we can place in the estimate. There is no general agreement as to how confident we can be with a certain number of playouts, but, for the purpose of analysis, I have my doubts about fewer than 10k playouts. With analysis I am not just interested in finding a good play, but in comparing different plays. a distinct task. With fewer than 100 playouts, Elf seems to take the winrate estimate from Elf's reply, just as it does with 0 playouts. With several hunddred playouts Elf takes the estimate from the move itself, not Elf's reply. I do not know the threshold number above which ELf does that.

But I have noticed that with relatively few playouts, but still above the threshold, Elf's winrate estimate for Elf's reply can be quite different from the estimate given for the game move. In such cases I trust the estimate from Elf's reply more than the estimate from the game play. For one thing, it has more playouts, usually much more. For another, it is one ply (go move) deeper in the tree. Using these estimates can produce quite a different picture from the winrate estimates from each move of the game.

Bill Spight · #3

Update.

I have been watching the ELf winrate estimates for plays in the game that have few playouts, and have a better sense of when the Elf team used the winrate estimate for Elf's reply instead. It seems like the threshold number of playouts is about 500. I have seen them keep the original estimate with 523 playouts, while with 494 playouts the given estimate was the same as that for Elf's reply.

IOW, 500 playouts seems not to be enough for the Elf team to have confidence in Elf's winrate estimate.

How many reviews have we seen that used winrate estimates with many fewer playouts than 500?

Edit: Update2.

There seems to be a playout threshold for plays that the ELf team report in a variation, i.e., plays with fewer than 1500 playouts do not seem to be reported.

Bantari · #4

yoyoma wrote:

Also note KataGo calls a 2 point mistake a 10% hit to your winrate!

Not sure how significant it is. When I play equally-skilled player, most mistakes will even out, and those which do not, will decide the game - no matter how large they are. If I play against a hypothetical perfect player, even 1 point mistake loses the game. And there is a whole spectrum in-between. But I might be looking at it all wrong.

To be honest - I am not really sure what this means. If both play perfectly from now on, one has 10% better chance? Or if both continue to make mistakes at the current level, then one has 10% better chance? Or what? I am really not up to date on my AI, so its an honest question.

Or is it just a relative evaluation to distinguish better moves from best ones? In which case the actual number (like 10%) might also be not significant - as long as it helps organize candidate moves from worst to best, we might use 1.1%, 2.7%, etc.. instead of 10% or 20%.

Somebody can explain?

dfan · #5

If both play at the level of the engine from now on, one has 10% better chance.

spook · #6

ZBaduk shows score estimations of KataGo since recently.
And from what I can tell,

KataGo will estimate W+1.25 point for the first move of the game.
Once a player falls behind more than 10 points, it will estimate his chances to be around 15%
And once he falls behind more than 20 points, it will estimate a winrate of below 5%

LeeLa Zero often seems to be more extreme than KataGo.

However, all of this may be a matter of configuration settings as well.
There are a lot of configuration settings of KataGo which I do not fully understand yet.

Attachment:

katago score estimations.jpg [ 378.92 KiB | Viewed 4358 times ]

On using Lizzie + LZ/KataGo/minigo etc

Who is online