Can you determine the use of AI based on 4 games?

[jlt]

@Sampi: you quoted me out of context. I just analyzed game 1 with KataGo and said that there is no evidence of cheating based solely on concordance with AI moves (I insist on the word "solely").

On the other hand, Antti Törmänen analyzed the game further and concluded that Black was probably a cheater. He did not just count the number of blue moves, but also noted that some surprising moves were considered by the AI, and that most of the game is high dan level while several moves are kyu mistakes.

I am ready to accept the verdict obtained after a thorough analysis by a skilled, experienced and impartial player like Antti.

And by the way, several months ago, I and other people showed Antti a game we found suspicious, and after the same kind of analysis he concluded there was probably no cheating, and we accepted his conclusion immediately.

[Sampi]

Some games of the same player where he is known to not be using AI (eg before strong bots if he was same strength years ago, in person game records where more easily believed to have not been cheating) for comparison allow stronger conclusions.

I don't think it's necessary but if it will help you come to the obvious conclusion then here you go:





Both games are a few days after the games I already posted and it's unlikely the person cheated.

[Bill Spight]

Some games of the same player where he is known to not be using AI (eg before strong bots if he was same strength years ago, in person game records where more easily believed to have not been cheating) for comparison allow stronger conclusions.

I don't think it's necessary but if it will help you come to the obvious conclusion then here you go:

If the conclusion is obvious, why did you ask the question?

[Uberdude]

These responses are really disappointing and make me sad as hell. I'm no sure if you people are blind, complicit or just plain dumb.

For clarity, do you include my responses in this?

The play is obviously stronger than any of the KGS ranks next to his name. Without checking vs AI, I'd say first game he hit all the opening points AI likes, which a mid-dan player like myself who has trained with AI can identify, though the chance of getting a dozen such decisions right in a row diminishes. I'd be tempted to atari at a10 before defending, but is that bad human atari love because a11 is also sente? If bot doesn't want to that's another point towards bot cheating, though of course strong humans can also resist ataris.

More information about the account also helps. If it's a new account for a KGS+ tournament that could plausibly be a strong Chinese amateur, who happened to have a ddk ? rank because he misunderstood automatch or timed out his first game then I'm less likely to think it's cheating than if it's an established stable ddk KGS user with hundreds of games.

Also, differential analysis with multiple bots is very useful, as I and Bill have said previously. If a player gets 80% of some matching metric with LZ v157, 83% with Elf, 73% with LZ 240, 84% with KataGo it's more plausible they are a really strong human player than if they get 95% with LZ v157, 60% with Elf, 64% with LZ 240, 72% with KataGo. A high match with the default network with Lizzie is even more useful information.

[Bill Spight]

These responses are really disappointing and make me sad as hell. I'm no sure if you people are blind, complicit or just plain dumb.

Has it crossed your mind that maybe we are smarter than you?

I don't see any evidence of cheating based solely on concordance with AI moves.

If you don't see any evidence, that's because you didn't bother to check.

If you say that, it's because you didn't bother to read jlt's note. Of course he checked. Are you calling him a liar?

What I do care about is how cheating is detected and handled, specially in tournaments where there's prizes. What really frustrates me is that this person is so blatant and obvious, and organizers (and apparently people in this forum) can't make up their minds.

You don't care enough about it to avoid jumping to conclusions yourself. Is that how you want tournament organizers to act?

I am afraid, from your remarks, that you did not ask your question honestly. Perhaps you were looking for agreement to bolster your own conclusions, and willing to dismiss out of hand people who disagree with you. Frankly, your question was deceptive. It wasn't really a question. Some people might call that cheating. What do you think? That's a real question, BTW.

[Sampi]

@Uberdude @Bill Spight @jlt
Hey guys I'm sorry if what I said was insulting, I'm just frustrated by the lack of action. The cheating does seem obvious to me (and I thought it was to everybody else) so I was surprised when nobody voted on the poll and nobody plain out called the cheater out (which is what I would've done).
I understand this is a public forum and you guys have no obligation to check the games or even offer a verdict. Again, I want to apologize, I'm not angry at you, I'm angry at the situation. You're not dumb at all, in fact I should thank you for your help.

[Sampi]

I am afraid, from your remarks, that you did not ask your question honestly. Perhaps you were looking for agreement to bolster your own conclusions, and willing to dismiss out of hand people who disagree with you. Frankly, your question was deceptive. It wasn't really a question. Some people might call that cheating. What do you think? That's a real question, BTW.

That is exactly right. I was just looking for agreement. I didn't mean to make my question deceptive, I was trying to sound as neutral as possible.

[Bill Spight]

Dear Sampi,

Thank your for your prompt and conciliatory reply. I sympathize with your frustration. And I believe that you did not have any deceptive intent when you posed your question.

Most people do not realize how weak confirmatory evidence is. I did not realize it myself until I did some research into the Hempel's Raven paradox some 30 years ago. The hypothesis that all ravens are black is logically equivalent to the hypothesis that everything is either black or not a raven. Thus, any observation of something black or something that is not a raven is confirmatory evidence for that hypothesis. The idea that observing a brown cow or black ink is evidence that all ravens are black seems preposterous. But that's the point. It is confirmatory evidence for that proposition, but it is so weak that we do not consider it evidence at all. What we need to do is to look for disconfirmatory evidence.

That's why we need other evidence than the fact that moves in a game record match some choices by a bot. Matching evidence is confirmatory, and therefore weak. What we need in addition is evidence that is not confirmatory. For instance, if top pros match bots' choices only up to 60% of the time, and usually less, then a matching rate significantly higher than 60% requires explanation. It is the difference between how often the player matches the bot's plays and how often the pros match the bot's plays that provides the disconfirmatory evidence that we need. But, as I said, evidence outside the player's game records is required to reach a conclusion.

[jlt]

More information about the account also helps. If it's a new account for a KGS+ tournament that could plausibly be a strong Chinese amateur, who happened to have a ddk ? rank because he misunderstood automatch or timed out his first game then I'm less likely to think it's cheating than if it's an established stable ddk KGS user with hundreds of games.

The KGS account only has 1 rated game and 5 unrated, so I initially thought that maybe he was a high dan sandbagger. I hadn't checked on OGS, where there is a 3-4 kyu account with the same nickname. Both users on KGS and OGS speak spanish so they are probably the same. So with that additional information, I am now much more inclined to think that he used an AI in his KGS games.

[gennan]

I'm just frustrated by the lack of action. The cheating does seem obvious to me (and I thought it was to everybody else) so I was surprised when nobody voted on the poll and nobody plain out called the cheater out (which is what I would've done).

I saw this topic, but I didn't look into those games, because just checking some games doesn't mean much for the more general issue of cheating IMO. A lot was already discussed in the related topic viewtopic.php?f=10&t=17528. We could repeat the issues discussed in that other thread, but I didn't see the point of doing that.

The OP of that other thread seems to have abandoned it. I got the impression he also got frustrated by the lack of agreement (more than you).
But it's not that we don't want to do anything about it. Rather, when you're going to build a real policy with sanctions, it's important to use tested and objective methods that get it right in edge cases. For developing such methods, much more work is needed than checking some individual cases.

Another thing, we here are not policy makers for go servers or go associations (are we?). So any agreement or disagreement you find here is not going to have much effect on anything. Discussing it here is more like a litmus test for when you're going to build your case to the real policy makers (assuming that is your intention).

[spook]

There are several small things we could do, to make it more difficult to cheat.
However, there is no guarentee that all ways of cheating can be stopped.

When a user constantly switches between applications, that's suspicious.
e.g. go clients should perhaps offer some transparency about the minimizing/maximizing of windows.
(Even for web clients, there is a visibilityChange event which could be used).

It guarentees nothing, but at least, it makes it more difficult for cheaters.
They would need at least 2 devices to succeed.

[Tryss]

Except that you really don't need to maximize/minimize windows to have the bot and your game on the same window :

SplitScreen.jpg (1.15 MiB) Viewed 5646 times

[Adin]

The OP of that other thread seems to have abandoned it.

I didn't abandon it, just had nothing to add to the latest posts.

Another thing, we here are not policy makers for go servers or go associations (are we?). So any agreement or disagreement you find here is not going to have much effect on anything. Discussing it here is more like a litmus test for when you're going to build your case to the real policy makers (assuming that is your intention).

That's very much not true. For example I am involved in policy making on KGS and what I read has some influence on me. And I can tell you that the other thread was also read by the top management of KGS. So each of the posts have an impact. Or to give another example when playing or just observing some games one might notice AI cheating and has a choice to either report it or not get involved. In the end the playing environment is created by each of us. One can write ten philosophical posts about it with no conclusion because everything is relative and how do we know what is the nature of truth and who is right? Or one can write to the administration of his favorite server requesting more action against AI cheaters. If the community feels strongly and more importantly acts against AI cheating then things will happen. Otherwise very few things will happen, just like now.

[vogalinha]

I'm the player on the 1st game, first of all I'd like to thank Antti for graciously analyzing the game in detail.

I'll just give more background information on the game. This game was played between me vogalinha (Alexandre Amaro) strong 6d/weak 7d on fox and a self-claimed 5k for the Latin America vs China qualifiers.

The game ended up a bit short as I felt very strongly that the opponent was using an AI, after messing up again in the last fight and even weakening my corner I was a bit annoyed so I just resigned.

Perhaps to give a bit more of a nudge in the "he's a cheater" direction, I'd like to point out that on game 4 there was a bit of a dispute in the chat as he thought the position on the bottom right corner was seki. Now tell me, how can someone who plays as in game 1 not now basic life and death shapes?

Based on that I'm nearly sure that he was using AI. You could argue that maybe he's really strong at the opening as he plays with the AI a lot, I find this to be hard to believe, but who knows.