Life In 19x19

EdLee wrote:Hi Javaness,

that Stewpot likes cruises

Noob question: what's this reference ?

A certain FIDE official

Lizzie (LeelaZero UI) version 0.5 was just released and has a new feature to bulk analyse the game (like go review partner) and give a winrate graph so I used this on a few games. It agrees that Carlo dominated Dragos, the big drop being the bulge and empty triangle connection. Middle line is 50%, black winning is at top. In the endgame LZ identifies a few of Carlo's mistakes but Dragos usually didn't punish them (e.g. Carlo plays a move which isn't sente enough so Dragos should make his own sente yose gain somewhere else but instead he just meekly answered) so these are little upward spikes in the graph. When humans are getting smashed by stronger bots (e.g. Lee Sedol vs AlphaGo or Haylee vs LeelaZero) the winrate graph is often like this, a one-way descent to defeat, whereas human games are typically more of a to-and-fro affair with mistakes from both making the leader change throughout the game. Of course sometimes one human just dominates the other and is in control the whole game, so whilst a descending graph like this could arouse suspicion it is far from proof that a bot was used (particularly when the good moves played by the winning player are not those suggested by the bot he is suspected of cheating with, as with the move 74 sequence).

Here are somewhat counterexamples of me beating/doing well against a stronger player. First vs Victor Chow 6/7d in the online league in Jan 2015 (so it should be easy to believe I didn't cheat with Leela as it didn't exist then!). At the end it thinks white won but that's because LZ assumes 7.5 komi but the league is 6.5 and I won by 0.5. Or me vs Kim Seongjin 7d at the London Open 1.5 years ago. I was white so doing fairly well, see if you can guess where I went into byo-yomi after my 90 minutes main time (he'd used 10 minutes)! The "98%" game vs Reem was interesting, as many commentators though Carlo was leading from fairly early so could play simply to win (Reem too soft in surrounding lower side moyo said Stanislaw), and that could explain the high similarity metric. Actually LZ thinks the lead switched a few times in middlegame and it wasn't a domination like vs Dragos. One of Carlo's moves it is most critical of was the empty triangle connection for move 117, instead of the wedge tesuji which Leela 0.11 also found (but his move still counted as a top 3 match as it was #3 and not >5% worse) . Lukan mentioned the game vs Csaba (round after vs Reem) looked like a different player. LZ (#145, 500 playouts/move for the graph) agrees that it was a much more to-and-fro game than vs Dragos, and a quite a bit more than vs Reem. Csaba was black and won by 2.5 so the 7.5 vs 6.5 komi difference will be a distortion near the end. I'm planning to do a top 3 Leela (0.11) similarity analysis on it later (mostly for interest, we already know this is a poor metric to use as evidence of cheating). [Edit: added missing graphs]

Uberdude wrote:The "98%" game vs Reem was interesting, as many commentators though Carlo was leading from fairly early so could play simply to win (Reem too soft in surrounding lower side moyo said Stanislaw), and that could explain the high similarity metric.

That's similar to what I said, but not exactly. I did think that Carlo took an early lead, and that a lot of the matching moves were simple, or part of one lane road sequences. Therefore, many of the matches with Leela did not mean much, if anything.

Actually LZ thinks the lead switched a few times in middlegame and it wasn't a domination like vs Dragos. One of Carlo's moves it is most critical of was the empty triangle connection for move 117, instead of the wedge tesuji which Leela 0.11 also found (but his move still counted as a top 3 match as it was #3 and not >5% worse) .

Carlo vs Reem LZ winrate.PNG

I guess this graph suggests that cheating by picking one of Leela's top three moves is not such a great strategy.

I took a look at the Dragos (white) vs Mijodrag game with LeelaZero/Lizzie, it was rather interesting both from this cheating case and a simple studying go perspective. Mijodrag's current rating is now 3d, but he used to be 5d so should have some stronger ideas but probably lost sharpness over the years as many older players have. Still, a bad loss for Dragos so weakens the "Dragos is super strong 6d so highly unlikely a 3/4d could beat him without cheating" idea (that's not to say the manner of the win was more dominating in Carlo's case). As Lukan said Dragos's o15 was a massive blunder (-59%) and he would be winning with the net, but LZ also thought black messed up earlier in the fight so could have been winning if he played it better. Here's the game and win graph; replaying the game quickly I didn't have a feeling for who was winning as both played such a boring territory style and I couldn't be bothered to count, but LZ actually thought the general trend was upwards towards a black lead. However, there was a lot of zigzags, and that's because at nearly every move for quite some stretches of the game LZ thought both players were missing the urgent point concerning the strength of the white wall on the right, q11 for black and p12 for white (I checked a few times with LeelaElf and it agrees). So everytime black doesn't play q11 his winrate drops a load, because it expects white to play the best move at p12, and then when Dragos doesn't black's win goes back up by about the same amount. It's like when you are watching DDKs playing and both fail to live/kill a 50 point group, but on a higher level.


To hammer home how important LZ considers this area, here's the times it thinks it's better than the move they played (better move with a 1, game move marked with X):

just noticed, Carlo did pretty well in the world amateur tournamnent beating stronger opponents, perhaps food for thought?

http://europeangodatabase.eu/EGD/Tourna ... n=14713996

Uberdude wrote:there was a lot of zigzags, and that's because at nearly every move for quite some stretches of the game LZ thought both players were missing the urgent point concerning the strength of the white wall on the right, q11 for black and p12 for white (I checked a few times with LeelaElf and it agrees).

One nice thing about playing over ancient games was discovering that, even when play was generally quite territorial, top players often played for central strength. I think that even 300 years ago top players would have found Black Q-11 or White P-12.

dsatkas wrote:just noticed, Carlo did pretty well in the world amateur tournamnent beating stronger opponents, perhaps food for thought?

http://europeangodatabase.eu/EGD/Tourna ... n=14713996

For what it's worth, the win against France was a forfeit.

Hello, after 13 pages of posts over this argument, I think the
statistical analysis of the "method" used to accuse Carlo Metta can
help. You can find it here: http://frmor.net/misc.html. Hoping it
stops this unhelpful loosing of time, with kind regards, Mirco.

That's a pretty compelling document, and my interpretation is that Carlo is owed a huge apology from the go community.

It's very readable and amazingly clear, given how advanced the statistics involved are.

My only concern is that the author is very close to Carlo (his PhD adviser).

It would definitely be good to get some independent corroboration/replication of the results posted in the document.

I found the following paragraph to be of interest:

Francesco Morandin wrote: We analysed another number that Leela provides: the probability that a strong amateur player would play this move. This number is called network probability. We put this in a table [12], where the choices of Metta are underlined and in red. We found that Metta’s choices were 49 out of 50 times in the human top 3 (and 46 out of 50 times in the human top 2). The only time his choice was “not human” (move 121) was a mistake according to Leela (Leela ranked this move in position 7 after indefinite time).

In my earlier criticism I pointed out that similarity (confirmatory evidence) is very weak evidence, and that contrast (disconfirmatory evidence) was necessary. I was unaware that Leela herself provided contrasting evidence, the contrast being between Leela's choices and the choices of strong amateurs. Picking one of Leela's choices which is not one of the human choices would be evidence of cheating. (Such evidence seems to be more available in chess, where top engines often play non-human moves.) There are other ways of finding disconfirmatory evidence, but they involve analyzing a large number of games. Here was disconfirmatory evidence that Leela put on the plate. However, that evidence was nil. There was no case where Metta picked a choice of Leela's that was not also a human choice.

To be sure, other evidence of cheating might have been developed, but was not.

MircoF wrote:I think the statistical analysis of the "method" used to accuse Carlo Metta can help. You can find it here: http://frmor.net/misc.html.

The document says:

Francesco Morandin, page 2 wrote:The average similarity from those 105 runs was 93.6%

I am surprised that Uberdude found 98% in his (single) test, whereas Francesco Morandin found 98% in only 2 runs out of 105.

Uberdude wrote:I independently analysed the game to find the Leela similarity metric: I got 98% for Carlo (aka 49 out of 50 matched) and 80% for his opponent Reem (aka 40 out of 50 matched)

Anyway, the whole discussion in this thread shows that evidence of cheating is very weak. In post #234, it was observed that the sequence that a strong player found the most remarkable in the most remarkable win of Carlo Metta during PETGC was actually not suggested by Leela.

I do not make an argument for or against evidence of cheating with this post.

I just want to mention the fact, that a cheating player can choose to not play the highest win probability moves, but just moves recommended by the network probablity.

(So moves similar to network probability are no argument that no cheating happened.)

Gomoto wrote:I do not make a argument for or against evidence of cheating with this post.

I just want to mention the fact, that a cheating player can choose to not play the highest win probability moves, but just moves recommended by the network probablity.

(So moves similar to network probability are no argument that no cheating happened.)

True. But that is not the claim. The claim, which bears the burden of proof, is that cheating happened.

jlt wrote:

MircoF wrote:I think the statistical analysis of the "method" used to accuse Carlo Metta can help. You can find it here: http://frmor.net/misc.html.

The document says:

Francesco Morandin, page 2 wrote:The average similarity from those 105 runs was 93.6%

I am surprised that Uberdude found 98% in his (single) test, whereas Francesco Morandin found 98% in only 2 runs out of 105.

Uberdude wrote:I independently analysed the game to find the Leela similarity metric: I got 98% for Carlo (aka 49 out of 50 matched) and 80% for his opponent Reem (aka 40 out of 50 matched)

Me too. I included the excel spreadsheet I used earlier on the thread for analysis/verification. I could check again but I think it may be explained by my analysis being with 20-30k rather than 50k playouts (which I only settled on in later runs) and Leela starts off liking the high network probability human moves and switches to better less human moves with more playouts. Also I didn't record the winrate from top deviation in the spreadsheet to start so might have been a bit sloppy there in including some moves as matches which should have been rejected by the 5% clause. Plus as Francesco noted by doing the analysis manually there is the possibility for the human operator (even unintentionally) to introduce some bias in letting the sims run a bit longer/shorter than intended to make a move a match or not.

jlt wrote: Anyway, the whole discussion in this thread shows that evidence of cheating is very weak. In post #234, it was observed that the sequence that a strong player found the most remarkable in the most remarkable win of Carlo Metta during PETGC was actually not suggested by Leela.

Now that I have a PC with gpu I let Leela analyse for 2.5m sims and q15 did rise from choice #18 to #5. So Leela did start to think about it more but was still far off local moves around the cutting stone (net and p7 peep top iirc).

Btw, this evening (7pm GMT) I'll be playing 1st board in the UK vs Italy league A/B promotion/demotion match against Matias Pankoke 5d. Italy have already won 2 boards, including Carlo beating Chris (our strongest but not highest ranked player), of the 5-game match so it'll be tough with us outranked on all 3 boards...

Hello,

I have read almost entire discussion, and i would like to point out to some facts that seems to be omitted here.

- In A league are strong amateur players, who are also very intelligent.
- results, and superficial analysis, can confirm that players accused of cheating did not use illegal software in all games.
- if we assume that some player decides to cheat, he will probably do it in a way not to arouse too much suspicion. After all, those are players with lot of playing experience, and, as we mentioned, are very intelligent.
- Conclusion would be that player who wants to cheat, will do it only in games where he would not arouse suspicion, i.e. 100 rating points around him.
- Also, since all players are strong players, they can use Leela only in critical moments, and continue to play by themselves after achieving advantage.
- Monitoring Leela for position analysis and some variations could help player to maintain enough lead.

Therefore, with minimal assistance, player can gain enough assistance to win a game.
And worst of all - it would be almost impossible to prove without a doubt if someone used help.


Analysis given in this topic focus on entire game (after opening, of course). There is a lot of moves that Leela could play, and lot of moves that human would play.
Taking into account points given above, it is not surprising at all. It would be surprising is someone used program for entire game, which would be idiotic to say the least.

-----

We must make several observations:
- After appearance of strong programs, PGETC management did nothing to prevent their usage.
- After suspicion that some players used it, very little investigation was done.
In current situation, regularity of games played on internet cannot be guaranteed.

After preliminary analysis of all games, I am afraid that even more players use computer assistance in a manner that would be difficult to prove.

-----

PS for those who are more versatile in analyzing, could you take a look at this game:
http://pandanet-igs.com/system/sgfs/627 ... 1496176456
Especially white's mistake.