LeelaZero adventures on Fox

[sorin]

This game was interesting, demonstrated the importance of getting the timing of a peep wrong.

What are the variations in the SGFs from your online games using LZ - are they auto-generated by Lizzie somehow (if so, how to do that)? Or did you add the variations manually, and then ran Lizzie in auto-analyze mode?

[Uberdude]

What are the variations in the SGFs from your online games using LZ - are they auto-generated by Lizzie somehow (if so, how to do that)? Or did you add the variations manually, and then ran Lizzie in auto-analyze mode?

I made them manually with pondering on, choosing top candidates moves of LZ. Some are me exploring the promising blue outline circle candidates, i.e. a move with lower playouts than LZ's filled blue first choice with most playouts, but with a higher winrate; so I will play that move and allow LZ to devote further playouts to it and if the winrate stays high once it has a similar number of playouts then I'll play it (e.g. l12 atari at move 88 leading to the trade in preference to initial choice of peaceful j13); sometimes the winrate will drop, typically because LZ then finds a strong counter to the move for the opponent that makes it not as good as it first thought (e.g h15 at move 34, or e8 at move 32: that had higher winrate than k17 with a few playouts but that's only because its instinct was that black would answer with nobi and then it was a good exchange, but it soon decided black would hane in response and then it wasn't a good exchange). So this is me manually increasing LZ's MCTS exploration factor so that it can find moves in a 30 second (or whatever) byo yomi that it would naturally find as best only with more time. When I usurp Lizzie's blue circle I think I do usually pick a move which is a stable better move, though occasionally I can hurriedly pick a weaker move which further analysis reveals as not as good as the original choice. Other times it's just me satisfying my curiosity of "what would LZ do if this happened?": I'm not just a dumb move-relaying monkey but a go player who uses this as a learning opportunity (for those unaware, shift and left/right arrow in Lizzie allows re-ordering of the game tree which is very useful to move the game line to lizzie's mainline to keep the sgf tidy and winrate graph synced). h13 peep on move 57 was me after the game exploring "Is this a good time to peep?". Answer is yes, LZ's recommended move for black at that point was l16 rather than n18 but it also thinks h13 is good as seen from the "(0.4%)" annotation, meaning after that move that player's winrate increased by 0.4% which is more than the 0.2 of n18 block (it's not unusual for winrate to have a few-tenths of % adjustment after move played vs before), and the variation shows LZ's expected continuation.

[sorin]

(for those unaware, shift and left/right arrow in Lizzie allows re-ordering of the game tree which is very useful to move the game line to lizzie's mainline to keep the sgf tidy and winrate graph synced)

Thanks for sharing your methods. You are quite a power-user of Lizzie - I had no idea about this feature!
I wish Lizzie had an option to automatically create a variation for the top winrate move when that differs from the current move.
The current way, it seems that the only option is to annotate each move with a +/- percentage, but that doesn't tell what the better move is (which would be very nice to know when there is a big %-age drop for instance) - one has to reload the game in Lizzie to find that out.

Thanks also for sharing your adventures with LZ - it is very interesting!

[Uberdude]

This super-157 is a plucky little fighter: it managed to hold its own running on my 1060 and even increase winrate as black from 47 to 55% * in middlegame against another bot player using the more-than-4-times as powerful dual 1080Ti (and I presume a 40-block network) in a 15 second game before one of my usurping blue circle decisions (t11 over s11, it didn't expect white t8) turned out to be bad (but maybe it would have lost anyway). There was a big semeai that ended in seki.

* However, this seems to be a misjudgement of the position, at move 204 LZ #198 thinks black is down at 28% already. I think the smaller block networks do have more trouble with semeais involving big groups. Todo: would be interesting to plot winrate graphs from various networks for this game.

[Uberdude]

So LZ 157a and #198 have quite different opinions about who was winning that game in the middlegame. First graph is 157a on 10k playouts, 2nd is 198 on 1k (which is a bit lower than equal time) with extra playouts at a few key moments for greater accuracy. Basically 157a thought the fight was pretty even and even sometimes good for it, 198 says white always good. I believe 198 more. It's also weird that after the seki is clearly formed 157a still wanted to keep playing mutual dame in the seki, did it not understand it was seki (such moves having equal winrate to other moves better for points elsewhere I can understand, but better point moves having winrate drops is weird, maybe it thinks only chance to win is if it plays self-atari it the seki and opp ignores? Or is the seki just too big for 15 blocks?). And third is elfv1 on 2k playouts. Elf tends to like black more than LZ as we see here to start with, but once the big fight gets well and truly underway it also thinks white takes the lead before 157a realised it was in trouble but still thinks there's quite some swings to almost even.

big seki 157a 10k winrate.PNG (193.62 KiB) Viewed 19189 times

big seki 198 1k winrate.PNG (226.13 KiB) Viewed 19189 times

big seki elfv1 2k winrate.PNG (205.12 KiB) Viewed 19188 times

P.S. Yeah I should export these winrates as numbers to graph them all together.

[Uberdude]

Played another game against a pro, karroyjack Zhao Yifei 3p goratings #622. LZ #198 embarassed itself with a ladder delusion. And it wasn't just due to low playouts, it appears to me 198 has got worse at ladders versus older 40-block networks!


For move 49 198 wants to play s15, it doesn't want to ladder the white stones (which doesn't work), but seem to think white WILL think the ladder works and thus needs to tenuki this sente move to prevent the ladder, meaning black can then capture the cutting stones in the corner to make all his groups safe. A very short horizon effect? 194 on the other hand wants to defend at q18 for a long time, but it too goes for s15 after 25k playouts.

karroy 198.PNG (1.59 MiB) Viewed 21526 times

[Bill Spight]

Crossing eyes and dotting tees:

[Uberdude]

Highlight of this game (network #199, 20s) was the deep invasion/reduction at move 40. LZ thumbs its nose at the black influence (top right joseki is rare because usually you don't want to give black the strength of the ponnuki even though it allows you to capture on the right side and keep corner) and lives with ease. I added quite a few variations of how LZ as white would deal with my various attempts (with LZ's help) as black to attack/kill it. LZ says it's best for black to just admit it's hard to punish and that territory's gone, tenuki to c9, let white spend another move to strengthen it and get another tenuki. Also 130-132 a nice example of "raising a flag" to create centre eyespace.

[splee99]

Move 64: 84% winrate. It's so hard even for top pro to calculate the chance of winning at this early stage.

[Uberdude]

Yes, the ability of these bots to detect a half point win 200 moves away is amazing.

[Uberdude]

Played another pro, "friedrice", Taiwanese-born Japanese pro Han Zenki 8p, goratings #490 at 3049. LZ #200 running on my 1060 GPU at 3 x 20 seconds was usually getting 1 or 2 thousand playouts per move but more when sequences went as expected (up to 10k or so). I didn't do any promising blue-circle jiggery-pokery, though there were a few instances where soon after playing a move LZ decided another was better (e.g. e12 over d12, r3 over k8). It beat him easily, impressive direction of play and haengma in a big fight from the first unsettled joseki. g4 was a sharp move which set up weakness in black's shape and would have preferred him to g5 defend and allow white d9 hane at head of two. The cheek of ignoring l5 was another highlight and subsequent sabaki, the way of living on lower side and right corner gave white moves like o5, p7 peep, j5 cut which meant it was confident to save the middle stones when he attacked at l9. This game was a good illustration of how LZ really doesn't like to connect on dame, and likes to force the opponent to connect on dame (but he refused and played p2 so LZ rescued the middle stones). LZ was at 96% by move 100 and it would have been about 25 points win if it went to counting (resign at 162). So still not played a top pro, but shows that LZ on mid-range hardware is a lot stronger than this level of pro (at least when no ladder delusion).

friedrice winrate.PNG (190.68 KiB) Viewed 21079 times

[Uberdude]

I tried GX88 against a human 9d on Fox. It was interestingly: messed up big time in the opening by playing a move which was only good if it had a ladder to counter opponent's resistance (which it barely considered so didn't read) but didn't. But impressively it managed to come back from 15% to win.

After about 10k playouts at move 24 it decides s5 is better than r7 so wouldn't get into the ladder mess, but if I play r7 even after 100k playouts it doesn't give enough consideration to r6 (only 24 playouts) to realise the ladder doesn't work and white playing there will totally stuff it. Once I play white r6 it needs only 1k playouts to realise the ladder doesn't work, but it still suffers from a delusion in that it expects black s6, white q7, s7 connect, and then white q8 extend to defend against a ladder atari which doesn't work. Of course white can instead play s4 to capture the corner stones because there's no ladder, and once played it again sees a big drop in winrate.

gx88 ladder.PNG (197.3 KiB) Viewed 20947 times

[Uberdude]

A game with LZ #202 as white, interesting blind spot in that after k13 wedge (n10 kosumi directly was other main choice) it never dreamed black would atari from outside, but as soon as black did that winrate plunged (although there is a throw in tesuji to capture m16 white loses many points from sacrificing the 4 stones. I analysed with a minigo weight (000513-revenge_converted.txt.gz) and it wanted to kosumi directly, but it too was blind to k12 atari outside. However, good-old 157, and 157b (the 157 with extra 40 block training) did find the k12 counter in just a few seconds, demonstrating my experience that although they are in general weaker they tend to explore a broader range of moves. Elfv1 also found it. k10 was a good move it only just found in time that turned the fight in its favour.

[Uberdude]

Tried out minigo v16 000513-revenge on Fox (black). Likes to start at 3-3, looks strong, though did have quite a few blindspots where it didn't see opponent's move and then winrate drops when they play it. An interesting mix of solid play and asking the opponent where the 2 eyes of groups are. Also some fun wedges of one point jumps at moves 115 and 121.

minigo.PNG (220.17 KiB) Viewed 20818 times

[Uberdude]

I also tried the 20-block minigo v15 990 cormorant. Won one ame, but this one was an embarassing ladder mess up. It then played sente moves in the hope the opponent would unnecessary honte capture the ladder and ignore the sente move, so I resigned. (Comments from lizzie say Elf but I used minigo and then looked at elf's view).