Life In 19x19
http://lifein19x19.com/

“Decision: case of using computer assistance in League A”
http://lifein19x19.com/viewtopic.php?f=10&t=15538
Page 31 of 36

Author:  Bill Spight [ Mon Jun 18, 2018 9:37 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Simba wrote:
So yes - my final words here at least for the foreseeable future:

Thank you for participating in our discussion of this serious issue. :)


Simba wrote:
I still believe without even the slightest shadow of a doubt that CM cheated against me, and find the opposite viewpoint to be generally very naive and idealistic.


I hope that you read this note and respond soon.

When you arrived I expressed my hope that you would say why you hold this strong belief. Perhaps I missed something, but all I heard was that you felt overwhelmed by Metta's play, and that it seemed that he did not play move 156 by copying Leela 11. Is there anything more? Without hearing a case that Metta cheated in this game what is there to say?

Edit: I know that you spent a lot of time on move 156 and the difference between Leela 11 and Leela Zero. But unless you assume that Metta was cheating at that time there is nothing there. And even if you do assume that he was cheating, then all you can say is that he was probably not copying Leela 11. In either case move 156 provides no evidence of cheating. I think that the anonymous accuser sowed alarm and confusion by preying upon fear and suspicion.

Author:  Bill Spight [ Mon Jun 18, 2018 10:14 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Charlie wrote:
Looking at Uberdude's analyses of the game Carlo played against Chris and at the game itself, move 156 looks pretty normal to me. Looking at the times in the SGF makes it seem even more normal.

Carlo spent nearly 4 minutes on move 154. Does anyone honestly believe that he needed that long for that hane? No way -- he was obviously thinking about the group at G18 which only has one eye and pretty much zero chance of making another one. When black descended to H18, Carlo played J18 in two seconds -- he had prepared it.


Important point.

Author:  Javaness2 [ Mon Jun 18, 2018 10:43 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Bill Spight wrote:
Important point.
Important entirely speculative point

Author:  Bojanic [ Mon Jun 18, 2018 11:52 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A

MircoF wrote:
Having the lunch together, he told us of our ranking system that is the best in the world. I wanted to say it to Ales that I know has developed it and thanks him for the work he has done for the European go community.

Yes, as we had chance to see on this topic, Ales is very interesting character, author of best rating system in the world by day, and witch hunter by night.

Author:  AlesCieply [ Mon Jun 18, 2018 12:15 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Bojanic wrote:
Yes, as we had chance to see on this topic, Ales is very interesting character, author of best rating system in the world by day, and witch hunter by night.

Thanks Mirco and Milos for the flattering words :oops: but you are quite wrong here. The EGF rating system is definitely not the best one, I am well aware of its flaws. However, for how it started (as a quite simple model and my attempt to prove wrong a few friends who claimed that ELO-like ratings would not work in go) it turned out to be reasonably effective. On whether you are wrong or correct on the other side of myself I let others to decide. :)

Author:  AlesCieply [ Mon Jun 18, 2018 1:11 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Jan.van.Rongen wrote:
So IMO my question is still valid: what was your reaction to the much more detailed and much more solid Italian expert report?

I appreciate you have learned quite a lot about my real (not what you previously thought) involvement in the case. I can confirm the appeals committee including myself was consulted when the league manager was making his decision and at that point I thought the cheating was sufficiently proven as we got reports from several different sources that came to the same conclusion. I would like to add that (despite what many people on this forum think) the original judgement was not about only one game and about the 98%. In fact, more games were checked including some provided by Carlo as his games played at regular tournaments. In addition, I have myself seen two more analysis but maybe only at some later time (would have to check my email to see when exactly I got them) and I also heard "expert opinion" on some of the games from very top European players, not only from Lukan who contributes here.

The content of the Italian appeal was a real surprise for me. I have read it several times, I can see some minor flaws in the reasoning there but I admit a well done job on their part. My reaction was that I installed Leela and started to check everything myself. That's about when I started to think about doing my own analysis to see if it can prove anything or not. If they can do it in chess why not in go was my thought. I did read several papers by Ken Regan then and decided to try the delta-histograms that were presented in the "New evidence report". At the same time we were also concluding our discussions in the appeals committee. At the point we decided to close the matter my analysis was still in its beginnings. It was me who formulated the appeals committee decision (everybody can still check it at the PGETC pages) so you can clearly see what was my opinion then.

I could say more to it but do not find it that relevant. While working on the analysis I found about the Shakhov-Metta game and that made it for me. Before that (and due to the Italian analysis) I have had some doubts about the accusation and I am aware that my analysis is not perfect. However, the analysis presented in my report cannot be disregarded (or disproved) so easily on the basis of the Italian appeal. The fact that the Italian appeal clearly showed the flaws of the original analysis does not mean that one cannot come with a better one. If they managed to do it in chess why not in go. Surely, it will take time but I feel we are progressing.

Author:  Bill Spight [ Mon Jun 18, 2018 1:33 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

AlesCieply wrote:
I would like to add that (despite what many people on this forum think) the original judgement was not about only one game and about the 98%. In fact, more games were checked including some provided by Carlo as his games played at regular tournaments. In addition, I have myself seen two more analysis but maybe only at some later time (would have to check my email to see when exactly I got them) and I also heard "expert opinion" on some of the games from very top European players, not only from Lukan who contributes here.


Many thanks for providing that information. :D

Quote:
If they can do it in chess why not in go was my thought.


Indeed. :)

Quote:
The fact that the Italian appeal clearly showed the flaws of the original analysis does not mean that one cannot come with a better one. If they managed to do it in chess why not in go. Surely, it will take time but I feel we are progressing.


Yes, we need to do research and develop the tools to detect cheating. That will take time. Meanwhile, because of the urgency, IMO we need to develop ways to make use of human expertise, which is available now.

Author:  Bojanic [ Mon Jun 18, 2018 1:34 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Uberdude wrote:
I independently analaysed the game to find the Leela similarity metric: I got 98% for Carlo (aka 49 out of 50 matched) and 80% for his opponent Reem (aka 40 out of 50 matched). Some people on facebook were reporting 93% for Reem, which would indicate this was the kind of game with many obvious moves so 98% is not damning, which I did not reproduce. Some caveats: Leela (I used version 0.11 which was released a month or two before the game in question) is not deterministic and can change its preferences as you let it analyse longer (I was usually around 30k - 50k nodes), so someone else could get different results. My feeling from watching the analysis is that 80% could go up a bit but not as high as 93%, also an odd number is impossible with a denominator of 50. Also does anyone know how strong Leela 0.11 is under these or other conditions?

In addition, Stanislaw Frejlak (Polish 6d) made a good analysis of the game:
Quote:
I analysed the game against Reem Ben David. Tl;dr from what I saw I would not accuse Carlo and I wonder where the 98% claims come from.

I know that the methodology I used might be not sufficient. I hope the official methodology was much more thourough and sofisticated. And also that this game is not the only evidence for such a decision.

And first of all, I find the rumours about Reem getting 93% completely ridicolous (Reem didn't play like Leela at all). Ok, let's go.

Thorough description of my analysis of the game:
Firstly I watched the game on my own trying to guess next moves. Then I uploaded it to Leela looking at her suggestions. Many of the moves in this game are really obvious and I had the same first intuition as Leela and so Carlo did. All other moves I checked giving Leela time to calculate from 30 000 up to 100 000 nodes. After that time Leela seemed to be decided not changing her views anymore.

Of course the opening is not the best part to analyse. Anyone studying with a program can learn its josekis. Anyway I started from the very beginning. So, the first move I found strange was 11 and Leela is really suggesting it. 13 I would rather push through but I checked weiqi.tools and pros like both options. 16 is a mistake, Leela agrees. 17 is strange, Leela agrees, but still it's an opening, still doesn't really matter. 29 is not played by pros at all. Leela prefers it a bit at F14 but she says Carlo's move is also good.

Josekis finished, we're entering a middle-game. Move 33 I found really strange (even more than white's 16). I thought that maybe this is this Leela style, but nope, Leela doesn't even think of this one. Of course she peeps C14. Everyone peeps... 37 is interesting. Probably many people would descend without thinking as Carlo did. But Leela suggests peeping! 38 is slow... My Leela prefers F3 for black now. Of course Carlo jumped, every human jumps. For 43 I considered G13. Leela prefers Carlo's move though. To 45 Leela approves. I also do. 51 probably I would not play. I didn't know where to play. Maybe I would try K16 starting a fight which is bad for black. Typical for my style... Leela suggests the move Carlo played. 52 is crude. 53 I'd consider pulling back, Leela and Carlo prefer hane. 55 I would connect at M17. Leela would push at M15. Carlo bumped at K17 which I find too simple. With 59 fun starts. In a fast game probably most of us would jump at J11. Leela also would but after some time she found E13 and realized it is very good. Carlo also played E13. It's the same with 61, not really obvious to exchange it but they both like it.

Ok, then white cut off two stones which surprised me. It's very slow, Leela agrees. From this point on, the game is undoubtedly good for black. White tries to surround the smallest moyo in the world with his thickness while black is occupying four corners. It makes the analysis harder, because black just needs simple moves. 75 they both like. However, Leela would tiger after it. Carlo of course extended as every human would. 79 Leela doesn't push through. I wouldn't either. Carlo did. 84 is super slack. Darn, it wasn't any software what decided a result of this game but white's moves like 16, 70, 84. Black's 85 I find strange. Why so calm? I thought Leela will recommend it but she doesn't! She plays a checking extension like a normal go player. But Carlo is playing safe knowing he's ahead. 87 they both like. Yeah, it's big.

After 96 everything seems more or less settled and Black's ahaed. Now black tigers twice which looks too bad (aji-keshi). Knowing that computer softwares like to play stupid exchanges (u remember Lee vs. Alpha game 2, moves 117 and 150? ^^), I expected it to be a Leela's suggestion. It is not. Firstly Leela wanted to keima at N14. Then she changed her mind to K13. The same after the tiger exchanges. However, Carlo plays N14. Move 105 I think I would eat a stone. Carlo blocked. Leela would extend at R13. Move 111 Leela answers to white's move. Carlo decides to take two stones. Funfact: after atari on two stones Leela would love to go back to O13. Carlo doesn't. Move 117 - in the first instinct I'd triangle. Leela considers connecting. But then she saw a wedge. True, wedge is powerful. Carlo simply triangled though. 119 they both peep. Good move. Then however, Leela doesn't think of keima [Uberdude: in my analysis Leela did find keima]. Of course she jumps. Carlo plays keima though. 123 Carlo attaches, and so Leela does. I didn't see it. It's really powerful. 125 Leela pushes at P12. Carlo turns instead. 127, 133 they agree. 135 Leela would just connect. The sequence starting from 139 is powerful and they both play it. But it feels natural also. 151 Carlo calmly protects his shape. Leela prefers to peep at N9.

Conlusion: moves 11, 29, 51, 59, 61 and 123 seem more or less suspicious. On the other hand moves 33, 37, 55, 85, 97, 117, 121 doesn't seem to be Leela-style. In general the game was simple and good for black from early middle-game so it can hardly provide strong proofs.

If we have no official study published I encourage others to do such analysis on your own. When we compare our results, we'll probably learn more.

I have just seen Frejlak's analysis, and I find it very interesting.

Here is short comment:
Quote:
Of course the opening is not the best part to analyse. Anyone studying with a program can learn its josekis.

I agree on opening and analysis, it is not best part to compare game to AI. However, since there are now new josekis, analyzing it is getting complicated and beyond my abilities. It is good Frejlak took a look at it.

Quote:
Move 33 I found really strange (even more than white's 16). I thought that maybe this is this Leela style, but nope, Leela doesn't even think of this one.

There is several similar examples in analysis, it seems that Stanislav focuses only on Leela's best move. He run analysis from 30k to 100k, and we know from duration of Metta's moves that most of his moves were fast. Move 33 actually is recommended by Leela as E choice (low), but it is still recommended, and early in analysis is shown in similar color as suggestions 2-5.

Quote:
Conlusion: moves 11, 29, 51, 59, 61 and 123 seem more or less suspicious.

They are all Leela's A choice, and moves 51 and 59 are important middle game moves.

Quote:
On the other hand moves 33, 37, 55, 85, 97, 117, 121 doesn't seem to be Leela-style. In general the game was simple and good for black from early middle-game so it can hardly provide strong proofs.

Actually all those moves are recommended by Leela. They are not A choice, but definitely are moves that would Leela play. Although some of the moves were not top pick, deviations in them are very small so they are not bad either.

Finally, Frejlak has focused only on Leela's top move and missed others, and also did not take into account wider picture.

Author:  Gobang [ Mon Jun 18, 2018 1:35 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Conclusion:

There are arguments for and against the idea that a player has cheated at online Go.

None of them prove anything. (If you use probabilities, then it is easily proven that every lotto winner must be cheating, since the chance of guessing the numbers correctly is astronomically small. Am I saying that lotto winners are cheats? No, I am saying that statistics are not enough to prove anything.)

Let's keep enjoying online go, but let's not take it too seriously and invest too much emotion into it. There is no way of knowing if you are playing against the human you are supposed to be playing against, version X Y or Z of Leela, Crazy stone, a visiting professional player, a space alien, a combination of all of these, etc...

We have a great hobby. Let's not spoil it.

This has made face to face games more valuable and desirable. I think that is a good thing. To me a face to face game is the most enjoyable kind of game.

I wish everyone who is organizing and attending the 2018 EGC all the best. (I hope I will some day be able to attend an EGC). We are all human, with all the baggage that comes with it.

Author:  jlt [ Mon Jun 18, 2018 1:45 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Bojanic wrote:
Quote:
Move 33 I found really strange (even more than white's 16). I thought that maybe this is this Leela style, but nope, Leela doesn't even think of this one.

There is several similar examples in analysis, it seems that Stanislav focuses only on Leela's best move. He run analysis from 30k to 100k, and we know from duration of Metta's moves that most of his moves were fast. Move 33 actually is recommended by Leela as E choice (low), but it is still recommended, and early in analysis is shown in similar color as suggestions 2-5.


Metta spent 133 seconds on move 33.

Author:  AlesCieply [ Mon Jun 18, 2018 1:51 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Bill Spight wrote:
Is there anything more? Without hearing a case that Metta cheated in this game what is there to say?

Here you can find what the player thought before "the anonymous intervention":
http://www.britgo.org/news/2018/teams0605

For me the relevant point is that he felt overwhelmed by playing much stronger player than 4d. I guess anyone long enough in go and sufficiently strong can see/feel whether his opponent is one stone or several stones stronger. I take Simba's testimony as another indication that CM's play on internet is well above his skills shown at regular tournaments.

Author:  bugsti [ Mon Jun 18, 2018 1:52 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Bojanic wrote:

There is several similar examples in analysis, it seems that Stanislav focuses only on Leela's best move. He run analysis from 30k to 100k, and we know from duration of Metta's moves that most of his moves were fast. Move 33 actually is recommended by Leela as E choice (low), but it is still recommended, and early in analysis is shown in similar color as suggestions 2-5.



After 200k simulations, move 33 is 17th choice for Leela 0.11 on my pc, with only 6 simulations for that move and very bad value :scratch:

Author:  Bojanic [ Mon Jun 18, 2018 1:59 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

jlt wrote:
Metta spent 133 seconds on move 33.

Please note that regarding move time there are several factors which should be taken into account:
- since input of moves into Leela and moves into client is not automatic, and we do not what was player doing during that time (eating, drinking, phoning...), relations of time spent with time necessary for Leela is not directly applicable.
- we can not even consider spent time as top limit for time used in Leela, since it is possible to do further analysis of next moves on opponent's time.

But it is necessary to point that order of Leela's suggestions is changing during time, and we do not have info at what time player would chose move.

Author:  Bill Spight [ Mon Jun 18, 2018 2:47 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Gobang wrote:
(If you use probabilities, then it is easily proven that every lotto winner must be cheating, since the chance of guessing the numbers correctly is astronomically small. Am I saying that lotto winners are cheats? No, I am saying that statistics are not enough to prove anything.)


If you use probabilities improperly, that is. The lottery fallacy is well known. :)

Quote:
We are all human, with all the baggage that comes with it.


Indeed. :)

Author:  Bill Spight [ Mon Jun 18, 2018 2:53 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Bojanic wrote:
Quote:
On the other hand moves 33, 37, 55, 85, 97, 117, 121 doesn't seem to be Leela-style. In general the game was simple and good for black from early middle-game so it can hardly provide strong proofs.

Actually all those moves are recommended by Leela. They are not A choice, but definitely are moves that would Leela play. Although some of the moves were not top pick, deviations in them are very small so they are not bad either.

Finally, Frejlak has focused only on Leela's top move and missed others, and also did not take into account wider picture.


Actually, focusing on the top choice is the most sensitive measure of agreement with Leela. Adding other choices makes the statistics look impressive, but also makes them less sensitive.

Edit: Yes, I know that Frejlak is not making a statistical argument. :)

Author:  EdLee [ Mon Jun 18, 2018 3:34 pm ]
Post subject: 

Quote:
The lottery fallacy is well known. :)
Hi Bill, is this it ?
Or perhaps this.

Author:  Bill Spight [ Mon Jun 18, 2018 4:30 pm ]
Post subject:  Re:

EdLee wrote:
Quote:
The lottery fallacy is well known. :)
Hi Bill, is this it ?
Or perhaps this.


Well, it's not the gambler's fallacy. (Actually, there are two gambler's fallacies, which are opposites. ;))

The prosecutor's fallacy sounds like it, but at least one form of the prosecutor's fallacy lies in ignoring background knowledge (probabilities).

Here is a link to a statement of the lottery fallacy. :) http://lucidphilosophy.com/18-lottery-fallacy/

Author:  Gobang [ Mon Jun 18, 2018 4:48 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

The most plausible argument to me in all this is the opinion of a player. I know what it feels like to play someone 2 stones weaker than I am and what it feels like to play with someone who is 2 stones or more stronger. There is no mistaking the difference. I am nowhere near 6d, but at that level the difference between 4d and 6d+ must be as clear as day and night. If a 6d player says that Metta cheated, then I think he probably did. The opinion of a 6d holds more weight to me than some comparison between Metta's moves and Leela's suggestions.

Lukan: "I have found another simple reason, why Carlo should be unseated from EGC main referee's position. The Italian side experssed, that he is depressed and afraid to speak on public. Following this logic, will not he be afraid to speak on public also on EGC, where he has to control 1000 players?"

How can it be that he still wants to be a referee?

Simba: "- I won't touch PGETC again if he isn't convicted and thrown out as per the rules, nor will apparently the entirety of Serbia based on what someone said a few pages ago, and I'm sure a few other strong players won't either. If the officials would rather keep this cheater in exchange for a bunch of legitimate players, then that's a shame but it's their prerogative."

Perhaps players who care about PGETC should get together and encourage PGETC to reform and legitimize itself and somehow recover from the damage caused by this debacle.

Author:  Bill Spight [ Mon Jun 18, 2018 5:36 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Gobang wrote:
The most plausible argument to me in all this is the opinion of a player. I know what it feels like to play someone 2 stones weaker than I am and what it feels like to play with someone who is 2 stones or more stronger. There is no mistaking the difference.


A four stone difference is perceptible. But the variability in results is also large.

Quote:
If a 6d player says that Metta cheated, then I think he probably did.


Skill at go is not the same as skill in cheating detection. Hardly anybody has been trained it that. We are learning, however. :)

Author:  RobertJasiek [ Mon Jun 18, 2018 9:19 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A

Gobang wrote:
the difference between 4d and 6d+ must be as clear as day and night


As I have explained in forum/viewtopic.php?p=232862#p232862 , it is not clear at all for only one game or a few games. It is only clear for many played games.

Page 31 of 36 All times are UTC - 8 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/