Uberdude wrote:
I independently analaysed the game to find the Leela similarity metric: I got 98% for Carlo (aka 49 out of 50 matched) and 80% for his opponent Reem (aka 40 out of 50 matched). Some people on facebook were reporting 93% for Reem, which would indicate this was the kind of game with many obvious moves so 98% is not damning, which I did not reproduce. Some caveats: Leela (I used version 0.11 which was released a month or two before the game in question) is not deterministic and can change its preferences as you let it analyse longer (I was usually around 30k - 50k nodes), so someone else could get different results. My feeling from watching the analysis is that 80% could go up a bit but not as high as 93%, also an odd number is impossible with a denominator of 50. Also does anyone know how strong Leela 0.11 is under these or other conditions?
In addition, Stanislaw Frejlak (Polish 6d) made a good analysis of the game:
Quote:
I analysed the game against Reem Ben David. Tl;dr from what I saw I would not accuse Carlo and I wonder where the 98% claims come from.
I know that the methodology I used might be not sufficient. I hope the official methodology was much more thourough and sofisticated. And also that this game is not the only evidence for such a decision.
And first of all, I find the rumours about Reem getting 93% completely ridicolous (Reem didn't play like Leela at all). Ok, let's go.
Thorough description of my analysis of the game:
Firstly I watched the game on my own trying to guess next moves. Then I uploaded it to Leela looking at her suggestions. Many of the moves in this game are really obvious and I had the same first intuition as Leela and so Carlo did. All other moves I checked giving Leela time to calculate from 30 000 up to 100 000 nodes. After that time Leela seemed to be decided not changing her views anymore.
Of course the opening is not the best part to analyse. Anyone studying with a program can learn its josekis. Anyway I started from the very beginning. So, the first move I found strange was 11 and Leela is really suggesting it. 13 I would rather push through but I checked weiqi.tools and pros like both options. 16 is a mistake, Leela agrees. 17 is strange, Leela agrees, but still it's an opening, still doesn't really matter. 29 is not played by pros at all. Leela prefers it a bit at F14 but she says Carlo's move is also good.
Josekis finished, we're entering a middle-game. Move 33 I found really strange (even more than white's 16). I thought that maybe this is this Leela style, but nope, Leela doesn't even think of this one. Of course she peeps C14. Everyone peeps... 37 is interesting. Probably many people would descend without thinking as Carlo did. But Leela suggests peeping! 38 is slow... My Leela prefers F3 for black now. Of course Carlo jumped, every human jumps. For 43 I considered G13. Leela prefers Carlo's move though. To 45 Leela approves. I also do. 51 probably I would not play. I didn't know where to play. Maybe I would try K16 starting a fight which is bad for black. Typical for my style... Leela suggests the move Carlo played. 52 is crude. 53 I'd consider pulling back, Leela and Carlo prefer hane. 55 I would connect at M17. Leela would push at M15. Carlo bumped at K17 which I find too simple. With 59 fun starts. In a fast game probably most of us would jump at J11. Leela also would but after some time she found E13 and realized it is very good. Carlo also played E13. It's the same with 61, not really obvious to exchange it but they both like it.
Ok, then white cut off two stones which surprised me. It's very slow, Leela agrees. From this point on, the game is undoubtedly good for black. White tries to surround the smallest moyo in the world with his thickness while black is occupying four corners. It makes the analysis harder, because black just needs simple moves. 75 they both like. However, Leela would tiger after it. Carlo of course extended as every human would. 79 Leela doesn't push through. I wouldn't either. Carlo did. 84 is super slack. Darn, it wasn't any software what decided a result of this game but white's moves like 16, 70, 84. Black's 85 I find strange. Why so calm? I thought Leela will recommend it but she doesn't! She plays a checking extension like a normal go player. But Carlo is playing safe knowing he's ahead. 87 they both like. Yeah, it's big.
After 96 everything seems more or less settled and Black's ahaed. Now black tigers twice which looks too bad (aji-keshi). Knowing that computer softwares like to play stupid exchanges (u remember Lee vs. Alpha game 2, moves 117 and 150? ^^), I expected it to be a Leela's suggestion. It is not. Firstly Leela wanted to keima at N14. Then she changed her mind to K13. The same after the tiger exchanges. However, Carlo plays N14. Move 105 I think I would eat a stone. Carlo blocked. Leela would extend at R13. Move 111 Leela answers to white's move. Carlo decides to take two stones. Funfact: after atari on two stones Leela would love to go back to O13. Carlo doesn't. Move 117 - in the first instinct I'd triangle. Leela considers connecting. But then she saw a wedge. True, wedge is powerful. Carlo simply triangled though. 119 they both peep. Good move. Then however, Leela doesn't think of keima [Uberdude: in my analysis Leela did find keima]. Of course she jumps. Carlo plays keima though. 123 Carlo attaches, and so Leela does. I didn't see it. It's really powerful. 125 Leela pushes at P12. Carlo turns instead. 127, 133 they agree. 135 Leela would just connect. The sequence starting from 139 is powerful and they both play it. But it feels natural also. 151 Carlo calmly protects his shape. Leela prefers to peep at N9.
Conlusion: moves 11, 29, 51, 59, 61 and 123 seem more or less suspicious. On the other hand moves 33, 37, 55, 85, 97, 117, 121 doesn't seem to be Leela-style. In general the game was simple and good for black from early middle-game so it can hardly provide strong proofs.
If we have no official study published I encourage others to do such analysis on your own. When we compare our results, we'll probably learn more.
I have just seen Frejlak's analysis, and I find it very interesting.
Here is short comment:
Quote:
Of course the opening is not the best part to analyse. Anyone studying with a program can learn its josekis.
I agree on opening and analysis, it is not best part to compare game to AI. However, since there are now new josekis, analyzing it is getting complicated and beyond my abilities. It is good Frejlak took a look at it.
Quote:
Move 33 I found really strange (even more than white's 16). I thought that maybe this is this Leela style, but nope, Leela doesn't even think of this one.
There is several similar examples in analysis, it seems that Stanislav focuses only on Leela's best move. He run analysis from 30k to 100k, and we know from duration of Metta's moves that most of his moves were fast. Move 33 actually is recommended by Leela as E choice (low), but it is still recommended, and early in analysis is shown in similar color as suggestions 2-5.
Quote:
Conlusion: moves 11, 29, 51, 59, 61 and 123 seem more or less suspicious.
They are all Leela's A choice, and moves 51 and 59 are important middle game moves.
Quote:
On the other hand moves 33, 37, 55, 85, 97, 117, 121 doesn't seem to be Leela-style. In general the game was simple and good for black from early middle-game so it can hardly provide strong proofs.
Actually all those moves are recommended by Leela. They are not A choice, but definitely are moves that would Leela play. Although some of the moves were not top pick, deviations in them are very small so they are not bad either.
Finally, Frejlak has focused only on Leela's top move and missed others, and also did not take into account wider picture.