Life In 19x19

When Facebook released their Elf OpenGo bot they also included 12 games versus 4 top Korean pros. These seem to have been rather neglected, so I thought I'd analyse one using the LeelaElf converted weights. As well as the go instruction this can see how close our LeelaElf conversion is to the true Elf: does it play the same moves and give the same win% that were included in the released games (Elf record uses a -1 to +1 scale whereas LeelaZero/Lizze a 0% to 100% so Elf pro record mistake of 0.05 to 0.09 is a 2% LZ mistake).

The games (except first 2) "were played using the v0 pretrained model (publicly available for download). For each move, ELF OpenGo used 2 threads with 40000 rollouts per thread (grouped into batches of 16). This took around 50 seconds per move on a V100 GPU.". For comparison my GeForce 1060 to do 80k playouts with LeelaElf takes 380 seconds (8 times). To enable a decent review pace I generally only did ~10k playouts, but if it looked like another choice might overtake the current top would do more.



And the winrate graph. White's biggest mistakes to 20% were:
- bottom right kick
- hoshi pincer
- hanging connection instead of cut tesuji
- top left approach
- double approach
- triple approach/surround As for comparing Elf and LeelaElf, the only significant good move Elf found that LeelaElf didn't in the part I analuysed was the q12 pincer, though LeelaElf agreed it was good when shown it. They both shared the strange blindspot of not seeing the monkey jump kill at top left (which does cast some doubt on the previous evaluations: black could live if didn't solid connect before and fall back, but then white would capture the one stone in sente which is quite a big difference to strength of groups). They were in pretty good agreement at finding the pros mistakes, though exact win% drops varied somewhat.

About LeelaElf not finding q12 on its own, I thought this could be because of significant differences between the LeelaZero and Elf engines running similar weights, but according to someone more familiar with them they should actually be very similar, but with usual non-deterministic algorithm (and rotation symmetry) caveats. So I repeatedly ran LeelaElf on the position before q12 for 80k nodes to see what it would choose:
1) for review q6 (or q11 I forget)
2) q12
3) q12
4) q12 from 2k
5) q12 (q6 until 10k, q11 starts to look promising before q12, q12 #1 at 18k)
6) q12 (q6 to 1k, then q11 #1, q6 back to #1 around 15k, q12 #1 at 35k and remains)
7) q12

So seems I got an unluckly roll of the LeelaElf dice when I was reviewing, averaging over multiple trials LeelaElf does find and want to play q12 like real Elf did. (Assuming that my exploration of the game tree didn't somehow bias the search when I got to q12, maybe it did).

One thing that I do when considering strategic plays recommended by programs (i.e., with no other stone adjacent or diagonally adjacent to it) is to also look at the adjacent plays. With programs doing a whole board search they may miss local improvements.

A good strategic heuristic is to consider the whole board and come up with a general area of play, and then to consider where to play within that area. In effect (or actuality) you have two searches, a global search followed by a local one. That works pretty well for humans.

Edit: If you are trying to create a program that plays well, a dual search heuristic might be inefficient, consuming too much time by comparison with a single global search. But if you are analyzing a position you do not have the same kind of time constraints to worry about. Different goals.

Here's a Lizzie picture of LeelaElf thinking about q12, it does consider it, but only has a few playouts and lower win% than q11 and q6 so doesn't devote many new playouts to it so doesn't realise the true win% is higher. So it seems LZ is quite sensitive to the initial (and high error) win% assigned to the other candidate moves, if you are unlucky and they are too low compared to the current #1 it won't put enough effort into investigating them and discovering they are actually good. This test it did actually come to prefer q12 by 50k playouts, but I remember when I was reviewing q12 started at 48% or 49% and never overtook q6/11.

I think it is funny how lifelike the AI response is.

Their answer always also depends on how you ask.