Life In 19x19

All positions are Black to play. The last play is marked.

First position: Second position. Position 3 Position 4 Position 5 Position 6 Edit: Added the times for White's previous plays, in response to Javaness2.

Edit: Added choices and % losses from Ales Cieply's analysis with Leela 11.

Position 7 Position 8 Edit: Added plays and percentages from the analysis by Cieply's Leela 11.

Position 9 Position 10 Position 11 Position 12 ----
No more positions. These 12 positions include all 10 which took Carlo 49 seconds or more, plus a couple of others. OC, we don't know why they took so long, but Botvinnik's idea that they were relatively difficult for him is a good working hypothesis.

Just to state the obvious, the kgb metric is fine up to a point. It would be nicer to have not just the time B spend on Move X but also the time W spend on the previous move (or moves if they were forced). What we can't tell is when B or W left the board to go to the bathroom, or nipped off to the kitchen to get a biscuit, or called up a professional on the telephone. As we don't know about such behaviour, we have to be careful about the kgb.

Bill,
nice topic for analysis.

I have to write again regarding time, since it is online game, we don't know what players were doing.
You might say that he was thinking for some time, but he could be reading news like I do, getting up, phone, etc.

Same goes for fast plays, it does not necessary mean that he didn't think on it.
If you run Leela in analysis mode (open empty sgf and start from beginning), you can go forward some moves. Also Leela will show you next moves on the bar, or in analysis window. Therefore, if opponent spent even little time thinking (10+ s), next moves could already be prepared.

During analysis of important moves I did almost similar as alleged cheater would do.

----

And finally, trying to go move by move and think would it be possible that 4d+ would play them is not good enough.
He could play most of them, but we have to take a look at bigger picture.
Main question is overall similarity of play to Leela, in suspicious and live games.

In 2 supsicious games, there is several important tenukis. He played all of them top Leela's choice (apart from one forcing sequence).
Theoretically, he could do it.
But how he could not hit one in WAGC games?

And finally, how all of his moves in 2 games were within 2% of Leela's top choice?
Human players rarely can do it, in short fighting games. Not in long games.

I think it is really worthwhile to pursue this approach. However, in the Italian appeal they demonstrate that Leela is not quite deterministic when deciding on its top move (or sometimes even on whether the move is among the top 3). Thus, my first question to your presentation would be - did you check whether in these chosen positions Leela "always" suggest the same move? I would run Leela let's say 5 times on each chosen position and with various playouts settings. We do not know for sure what computer CM used. We can just assume the information provided by the Italians (about 100k nodes in 30s) is OK. When playing with Leela I found that the estimated winrates are more settled than the Leela's top move suggestions, which also led me to choosing the type of analysis based on delta histograms rather than looking at selected significant moves. Both approaches have their merits and flaws ...

AlesCieply wrote:I think it is really worthwhile to pursue this approach. However, in the Italian appeal they demonstrate that Leela is not quite deterministic when deciding on its top move (or sometimes even on whether the move is among the top 3).

That's why I said what Frejlak's Leela said and what Bojanic's Leela said. Bojanic's rsgf file also gives win rate estimates.

AlesCieply wrote:When playing with Leela I found that the estimated winrates are more settled than the Leela's top move suggestions,

Good to know.

Looking at the rsgf files, it seems to me that the variability in the estimated win rates crucially depends upon the number of rollouts. (Big Duh! ) So to use Leela for evaluation, not for play, I would want to have the same number of rollouts for each possible choice. Preferably at least 10,000 each.

Edit: Today's top bots are optimized for play, not for evaluation.

Javaness2 wrote:Just to state the obvious, the kgb metric is fine up to a point. It would be nicer to have not just the time B spend on Move X but also the time W spend on the previous move (or moves if they were forced).

Interesting idea.

What we can't tell is when B or W left the board to go to the bathroom, or nipped off to the kitchen to get a biscuit, or called up a professional on the telephone. As we don't know about such behaviour, we have to be careful about the kgb.

Claro que si.

Bojanic wrote:I have to write again regarding time, since it is online game, we don't know what players were doing.
You might say that he was thinking for some time, but he could be reading news like I do, getting up, phone, etc.

Same goes for fast plays, it does not necessary mean that he didn't think on it.

Ditto.

If you run Leela in analysis mode (open empty sgf and start from beginning), you can go forward some moves. Also Leela will show you next moves on the bar, or in analysis window. Therefore, if opponent spent even little time thinking (10+ s), next moves could already be prepared.

Ditto.

Bojanic wrote:And finally, trying to go move by move and think would it be possible that 4d+ would play them is not good enough.

It is not good enough to demonstrate innocence. But that is not where the burden of proof lies.

Bill Spight wrote:

AlesCieply wrote:I think it is really worthwhile to pursue this approach. However, in the Italian appeal they demonstrate that Leela is not quite deterministic when deciding on its top move (or sometimes even on whether the move is among the top 3).

That's why I said what Frejlak's Leela said and what Bojanic's Leela said. Bojanic's rsgf file also gives win rate estimates.

For moves I considered important, I watched how analysis progressed, and noted when some choice first appeared.

There is so much to say about this analysis, so I am going to make some renarks in some random order.

1. We do know the specifications of that computer: unless you have reasons to think somebody is lying which I find disturbing. I even find it disturbing that it is suggested not knowing something that is known.

Anyway is has an NVIDIA GTX 960M, which is the dominant factor in speed. CPU and memory are far less important. The speed gain over the CPU is a factor of at least 10.

2. The time taken to move contains a wealth of information that has not been used in a thorough analysis. In the Italian appeal only the weaknesses of the original "98% method" were shown, but they failed to provide additional evidence that goes against the allegation of cheating. A statistical analysus can show that both players speeded up, resp. slowed down in the same periods: i.e. they could both experience the same situations as difficult or easy. This is IMO strong evidence of not cheating, because it is very difficult to simulate that behaviour.

3. It is an error to use Leela 0.11 to evaluate a situation when you investigate cheating with the same program because that will easily lead to circular reasoning. The win rate given by Leela 0.11 are very different from AQ (which existed at the time of the game) and from Leela Zero and Leela ELF.

4. Position 5: black 51 was played in 2 seconds, too fast to consult a computer. You could object that he could have peeked earlier, but both players were playing at a rather high speed in this sequence, so there was very little time to use an AI. Furthermore if you would use the Leela application in analysis mode it does NOT suggest this move until after white 50. In other words, there is firm evidence that B could not have cheated to play this move.

5. Position 6: Black 71 @ J9. Here we see the limits of the evaluation function of Leela 0.11, because it does not "see" the whole forced sequence that follows, where the difference increases from 2 to 10%. AQ sees a 8% difference immediately. So that classifies White 70 as a very bad move.

6. Edit again: sorry - I made an error in the move number.

thanks.

The following diagram shows AQ's evaluation after Blacks move. This diagram shows how Blacks "winning %" changes after his answer to Whites moves. So if it jumps up- White's move was not so good and Black replied took advantage from that.

The diagram is from 11-04 this year, when AQ was sill about as strong as LeelaZeo and stronger than Leela 0.11

Jan.van.Rongen wrote: 3. It is an error to use Leela 0.11 to evaluate a situation when you investigate cheating with the same program because that will easily lead to circular reasoning. The win rate given by Leela 0.11 are very different from AQ (which existed at the time of the game) and from Leela Zero and Leela ELF.

It is wrong to use Leela in analysis of case where it is suspected that it was used?
Interesting reasoning. What is next, to use some random player's games instead of Metta's?

Jan.van.Rongen wrote:4. Position 5: black 51 was played in 2 seconds, too fast to consult a computer. You could object that he could have peeked earlier, but both players were playing at a rather high speed in this sequence, so there was very little time to use an AI. Furthermore if you would use the Leela application in analysis mode it does NOT suggest this move until after white 50. In other words, there is firm evidence that B could not have cheated to play this move.

Analysis window of Leela.
Please note that on bottom bar before W50 was played you can see L17 suggestion almost immediately. Therefore, black had enough time, and your "firm evidence" is wrong.

Jan.van.Rongen wrote:The following diagram shows AQ's evaluation after Blacks move. This diagram shows how Blacks "winning %" changes after his answer to Whites moves. So if it jumps up- White's move was not so good and Black replied took advantage from that.

The diagram is from 11-04 this year, when AQ was sill about as strong as LeelaZeo and stronger than Leela 0.11

The attachment move_diagram.jpeg is no longer available

And this is the diagram of game Metta-Ben David in Leela 0.11, which shows remarkably small differences. This diagram is very remarkable on it's own - black made very few mistakes, that were rather small.
It is difficult how to explain not only such small deviations, but stream of Leela's top choices in human game of this length.