“Decision: case of using computer assistance in League A”

[theoldway]

One thing we would like to know is how often Leela makes similar blunders. Here is also where the choices of other strong programs, such as Zen, Golaxy, or Leela/Efi could be helpful to know. If they also choose that blunder, then the fact that Metta did too is not so significant. It would be a blunder that is easy to make, even if you are playing well. The choices of other strong amateurs would also be relevant, by the same token.

It seems that also Zen 7 and Leela Zero make this blunder. Maybe it's not that easy to spot as real game tsumego. It's kind of tricky.

[Gobang]

Therefore, it is not just handful of moves, it is entire middle game in two online and two live games.

Regarding other live games of Carlo Metta, please note that some of them were played after objection that he used Leela, therefore it would be less likely that he might use Leela again in same manner (entire game), or if at all.

Chess players needed years and more than 200,000 games of thousands of players to develop, test and estabilish an anti-cheating protocol. You took a bunch of online games of one player (not randomly, but those you've observed as similar to Leela) and 3 live games played from 7 to 12 months later than the online games.

Are we supposed to take you seriously?

Please take your seat in the Salem witch trial, you deserve it.

I find your post inappropriate and ungrateful to people who are trying to make real sense of this situation. Your attitude strengtheners my opinion that Carlo's and anybody else's online games are completely meaningless and that trying to analyse and prove cheating one way or another is not going to lead to any satisfactory results.

Someone analyses games and proves beyond reasonable doubt that cheating has occurred. This is met by protest and derision. A debate ensues. Since there is no admission of guilt, there is no real result. The whole exercise has been a waste of time.

Given this situation I can not see how Pandanet can continue hosting online tournaments, at least tournaments that anyone takes seriously. I am sure they have better things to do with their time than to run around after cheaters and get stuck in disputes once cheating is discovered. They have nothing to gain.

I find this comment: "Regarding other live games of Carlo Metta, please note that some of them were played after objection that he used Leela, therefore it would be less likely that he might use Leela again in same manner (entire game), or if at all." interesting. How is he supposed to "use Leela again" in a live game?

Finally, I find it fascinating that anyone embroiled in this kind of controversy would find it appropriate to be chief referee for a major go tournament, regardless of guilt or innocence.

[Bojanic]

Gobang,
My obvious lapsus.
I menat online games, pgetc after round 4.

[Tryss]

Someone analyses games and proves beyond reasonable doubt that cheating has occurred. This is met by protest and derision. A debate ensues. Since there is no admission of guilt, there is no real result. The whole exercise has been a waste of time.

I'm sorry, but I didn't see here a proof that cheating occured beyond reasonable doubt.

And my opinion on this matter is that it's probably impossible to prove it beyond reasonnable doubt in this case.

Finally, I find it fascinating that anyone embroiled in this kind of controversy would find it appropriate to be chief referee for a major go tournament, regardless of guilt or innocence.

But I can agree to this, he should step down from this position. And as you say, that's regardless of guilt or innocence.

[Gobang]

I'm sorry, but I didn't see here a proof that cheating occured beyond reasonable doubt.

No need to be sorry. It is your opinion and you have every right to it. "Reasonable doubt" will always be subjective.

[Uberdude]

One thing we would like to know is how often Leela makes similar blunders. Here is also where the choices of other strong programs, such as Zen, Golaxy, or Leela/Efi could be helpful to know. If they also choose that blunder, then the fact that Metta did too is not so significant. It would be a blunder that is easy to make, even if you are playing well. The choices of other strong amateurs would also be relevant, by the same token.

It seems that also Zen 7 and Leela Zero make this blunder. Maybe it's not that easy to spot as real game tsumego. It's kind of tricky.

Assuming you mean the Metta-Ruzicka game, that's what I posted about. LeelaElf makes the same oversight, as did Frank Jansen 6d in a very similar shape in another game. Btw, I think it's better to call this an oversight (of not seeing the black cutting move killing through shortage of liberties) than a blunder, as to me a blunder implies a very bad move where a good one was available, but here the situation is white is dead (or a desperate ko connection out depending on the outside) and there is no good move. The mistake/blunder is allowing yourself to get into this shape earlier.

[Bill Spight]

Therefore, it is not just handful of moves, it is entire middle game in two online and two live games.

Regarding other live games of Carlo Metta, please note that some of them were played after objection that he used Leela, therefore it would be less likely that he might use Leela again in same manner (entire game), or if at all.

Chess players needed years and more than 200,000 games of thousands of players to develop, test and estabilish an anti-cheating protocol. You took a bunch of online games of one player (not randomly, but those you've observed as similar to Leela) and 3 live games played from 7 to 12 months later than the online games.

Are we supposed to take you seriously?

Please take your seat in the Salem witch trial, you deserve it.

I find your post inappropriate and ungrateful to people who are trying to make real sense of this situation.

I agree. Bojanic has conducted a careful and important investigation. Whatever his errors, he deserves our respect and gratitude.

Your attitude strengtheners my opinion that Carlo's and anybody else's online games are completely meaningless and that trying to analyse and prove cheating one way or another is not going to lead to any satisfactory results.

Analyzing a few games statistically is unlikely to lead to any definitive results. As theoldway indicates, it took a lot of data to develop and test an appropriate statistical methodology for detecting cheating at chess. For go we are just getting started on that task.

Someone analyses games and proves beyond reasonable doubt that cheating has occurred. This is met by protest and derision. A debate ensues. Since there is no admission of guilt, there is no real result. The whole exercise has been a waste of time.

"Reasonable doubt" will always be subjective.

I sense your frustration. But the journey of 1,000 miles begins with a single step.

[RobertJasiek]

I find it fascinating that anyone embroiled in this kind of controversy would find it appropriate to be chief referee for a major go tournament, regardless of guilt or innocence.

Suppose innocence and the referee's honest impartiality, that is, if he thinks to be partial when meant to act or decide, assume he temporarily retreats from his duty with respect to a task during which partiality has an impact.

A referee has the right to be a referee. As a referee, he deserves respect as a referee. A referee has the right to be, or have been, involved in a dispute on his own because he, as a player, has the same rights to justice as those players he judges on. He just must maintain impartiality, i.e., if the case he judges on is significantly related to the case he is, or has been, judged on, he must temporarily retreat to dissolve the particular conflict of partiality.

(There are exceptions, such as in the German Championships, where the appeals committee members necessarily belong to the players so every such referee is necessarily partial. Christoph Gerlach set the precedent that, in this tournament, this kind of partiality is allowed because each possible substitute referee is also necessarily partial because the other players' results also affect his own tournament results relative to the other players. For practical reasons, there are no extra persons forming the appeals committee.)

Let me repeat: As a referee, he deserves respect as a referee. Do not cast unjust doubts on his integrity as a referee. As a referee, he deserves this respect not to be loaded with unjust doubts. What applies to a referee also applies to a chief referee, member of an appeals committee or member of the EGF Rules (etc.) Commission (third instance, although I do not know whether Panda tournaments are EGF tournaments and subject to it).

A player involved in a dispute has the right to question impartiality of his referee(s). In that case, the referee(s) must honestly and sincerely consider whether / inhowfar they are indeed impartial. If they find (and explain) their impartiality, then the player must respect them to judge on his case. (See also the precedent EGF Rules Committee judging about Zeijst - Dinerstein: a questioned impartiality was considered, rejected and explained.)

[jlt]

A few observations on Bojanic's analysis. For the game Carlo Metta – Reem Ben David, he analyzed middle game tenuki moves 51, 59, 65, 87, 97, 101.

• I still have Champion Go 1.1.4 on my smartphone. This program is quite weak compared to Leela or to the two players, probably at most 5k EGF. Among these 6 moves, it found move 65, so we can assume that Carlo Metta could easily find move 65 by himself.
• If we want to make Bojanic's method more scientific, we should give a precise definition of tenuki that a computer could check. I don't know if this has already been done, but in any case I am not able at the moment to provide such a definition.
• Let's call a "non obvious move" a tenuki move played between moves 50 and 150 (or maybe 30 and 180) and which is not suggested by a weak program (could be GnuGo or another program).
• Find at least 100 random online games played before the existence of strong AI, with sufficiently long time settings (no blitz), and with both players being mid-dan. For each of these games, determine the number of "non obvious moves", and how many of these are Leela's suggestions.
• From these data, estimate how likely it was for Carlo Metta to have such a high degree of similarity a) for the match Metta-Reem b) for this match and all his preceding PGETC. If the likelihood is very small, like 1:1000000 or even 1:100000, this will be a very strong indication of cheating although not a hard proof. If not, the case should be dismissed.
• More important than solving this particular case is to design a scientific method to prevent future cheating. We don't know for sure if cheating has already occurred in online or OTB tournaments, but we know that cheating will occur in the future. Cheating discovered by statistical methods can justify at least that a match be replayed and monitored, while someone found guilty of using a hidden electronic device should be banned from tournaments for several years.
• There are very few cases where just statistical methods (without material proofs) could be enough, for instance when a player improves very quickly, beats several dan players at a tournament but is unable to solve 15 kyu tsumegos or tesujis.
• In online tournaments, if matches are not monitored or recorded by webcams, it will always be possible to cheat in a subtle way, like using a strong software just once or twice during the game. This will be undetected by statistical methods, just like when athletes are injected just enough EPO/growth hormone/testosterone/whatever so that their blood or urine analysis looks normal; these athletes should be considered innocent until material proof is found.

[RobertJasiek]

Especially in unsettled positions, there is more than one possible definition of local versus elsewhere. One possible definition is:

An empty intersection is _local_ to a two-eye-alive string if the opponent cannot prevent the player from both becoming part of one two-eye-formation of his.

Note that such a definition does not define local to an uncapturable seki string. Also note that local to one particular two-eye-alive string is not the same as local to an unequivocal group of a player's two-eye-alive strings. For definitions of 'prevent', 'two-eye-alive' and 'two-eye-formation', see my webpage.

[Gobang]

I still find myself struggling to see the point of expending time and effort towards trying to create an anti cheating system for online Go games. Who would want to spend time doing that? Who would pay for it? Would it really prove anything? The worst case scenario is that it would legitimize cheating. A cheater can work out a way to beat the system and then proudly proclaim that his results are honest.

I find the lengthy discussion of cheating methods and cheat detection systems somewhat distasteful. It has some academic interest but the practical meaning is doubtful. It is truly sad that this situation has come up.

Regarding players caught red handed cheating, for example by using a hidden device in a live tournament. They should not be allowed to play another tournament game ever again.

[Bojanic]

Hello jlt,
thank you for constructive contribution to this topic.

A few observations on Bojanic's analysis. For the game Carlo Metta – Reem Ben David, he analyzed middle game tenuki moves 51, 59, 65, 87, 97, 101.

Just to note - fighting sequences in between are also analyzed.
Tenuki moves are interesting for research because they are not forced.

[*] I still have Champion Go 1.1.4 on my smartphone. This program is quite weak compared to Leela or to the two players, probably at most 5k EGF. Among these 6 moves, it found move 65, so we can assume that Carlo Metta could easily find move 65 by himself.

It is possible, of course, but we are taking into account several tenuki moves and fighting sequences.
He could have found one (although in WAGC he failed to find any), but almost every move to be similar to top choice is too much of coincidence.

[*] If we want to make Bojanic's method more scientific, we should give a precise definition of tenuki that a computer could check. I don't know if this has already been done, but in any case I am not able at the moment to provide such a definition.

It is certainly a bit subjective, but I think that definition that it could be a move that is played without a contact with previous group of stones.

[*] Find at least 100 random online games played before the existence of strong AI, with sufficiently long time settings (no blitz), and with both players being mid-dan. For each of these games, determine the number of "non obvious moves", and how many of these are Leela's suggestions.

Finding a good sample of games would be a problem, especially if player is improving.
In last year, for Metta we have only records of 2 live games, and less than 10 internet (of which half could not be used in analysis).

[*] More important than solving this particular case is to design a scientific method to prevent future cheating. We don't know for sure if cheating has already occurred in online or OTB tournaments, but we know that cheating will occur in the future. Cheating discovered by statistical methods can justify at least that a match be replayed and monitored, while someone found guilty of using a hidden electronic device should be banned from tournaments for several years.

Please note that in this situation we have two games where there are almost complete similarities in middle game with Leela, and it is very difficult to prove without reasonable doubt.
How could you prove if someone used program only for one fighting? Or to analyze his moves. It is impossible.

[*] In online tournaments, if matches are not monitored or recorded by webcams, it will always be possible to cheat in a subtle way, like using a strong software just once or twice during the game. This will be undetected by statistical methods, just like when athletes are injected just enough EPO/growth hormone/testosterone/whatever so that their blood or urine analysis looks normal; these athletes should be considered innocent until material proof is found.

I don't think that camera is enough, esp if player has help. We have seen player in China who cheated in live game, which is substantially more difficult.
Player could open window, or on his phone, and watch online game analysis.
Or receive moves via message system.

[Javaness2]

My gut feeling is that it is so difficult to detect cheating in PGETC, that putting in place complex and costly measures is just not practical, nor perhaps even possible.
Probably it, and all other online events, will become unrated again.
We rely on honour in the preliminary stages, just as we always did.
In the finals, the team which finishes last will not have any cash prize.

[John Fairbairn]

I still find myself struggling to see the point of expending time and effort towards trying to create an anti cheating system for online Go games. Who would want to spend time doing that? Who would pay for it? Would it really prove anything? The worst case scenario is that it would legitimize cheating. A cheater can work out a way to beat the system and then proudly proclaim that his results are honest.

It may not be the main focus here, but many pro games are played online. E.g. China is so big that it makes sense to play some games that way.

I find the lengthy discussion of cheating methods and cheat detection systems somewhat distasteful. It has some academic interest but the practical meaning is doubtful. It is truly sad that this situation has come up.

Some of the length of the discussion is explained by the need to counter those who rushed to judgement based on intuitive suspicions and flimsy statistics. Preserving standards of justice is rather more than academic.

Regarding players caught red handed cheating, for example by using a hidden device in a live tournament. They should not be allowed to play another tournament game ever again.

There is an ambivalent attitude towards cheating. I think there may be a biological explanation. Cheating among primates seems to be a way that (say) physically weaker individuals can overcome physical stronger ones. So we recognise the need to cheat and can even applaud especially clever scams. Against that we have notions of fair play that seem to be imposed by moralists from outside and which can vary from culture to culture. All this, plus the threat of legal action, seems to put a severe brake, in various sports, on how harsh sanctions should be.

Personally, I would like to see very harsh sanctions, but what follows from that is a need to get the decision absolutely right in the first place.

For whatever reason, the history of cheating shows us that the typical response is to set up elaborate and expensive mechanisms to combat it - just look at the huge international bureaucracy of the anti-doping organisations in athletics and other sports. Even in go, it is believed that Chinese rules were changed in Ming times just to combat the cheaters who were making the number of captured stones increase or decrease. In Japan, the wide use of coded comments by which strong bystanders could help weaker accomplices win huge sums in gambling games led to harsh penalties (which have occasionally been enforced) that prevent pros from playing games for stakes. Even a non-go oriented country like Argentina has applied the full force of their law against Korean immigrants gambling at go.

Yet none of this stops cheating. We have had a case in recent times of a 9-dan team captain sanctioned because of match fixing.

So we can expect in go the further rise of measures we are already seeing - e.g. phone lockers and doping tests.

The huge changes and inconveniences that cheaters cause for the honest participants are in some ways akin to terrorism.

But the "honest" people are just as much terrorists if they base their reactions on mere suspicions. If the usual trajectory is followed without resistance, we'll be seeing highly regrettable mutations of the suspicions about one player become "all Italians are cheats" or "all AI programmers are cheats." Then it becomes hard to look a cheat in the eye and tell him what he is doing is despicable.

[Gobang]

The huge changes and inconveniences that cheaters cause for the honest participants are in some ways akin to terrorism.

+1. I could not agree more.

In a way I guess I have been living in "cloud cuckoo land". I am new to serious Go and the Go tournament scene. I was under the illusion that it was more important to Go players to treat the game, themselves and their opponents with dignity and respect than it was to simply win games.