I just picked it at random off the list of players. I find it difficult to imagine that it is easy to find a particular graph if you trim off the kyu/dan rankings as well.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

In this era of big data, we are already seeing algorithms that estimate important things, such as the quality of classroom teaching or the propensity to commit crimes, which are used to make consequential real world decisions, but which are untested. In addition, they are proprietary, so that it is, as a practical matter, impossible for them to be independently verified. Yet they are used, faute de mieux, and may simply enshrine current flawed judgements, at best.

I think that it is not only possible, but easy to come up with an algorithm that would distinguish between the plays of go bots and the plays of human pros, in human games played before 2015. Not no more. We humans have learned from the bots, and we are still learning from them.

After the sprint cheating scandal on IGS in the 1990s, I would not have held an online tournament without monitoring for prize money. At all. And now, everybody can consult their own expert.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

KGS dan ranks differ from real world EGF ranks by up to 6 ranks without any cheating. That is how unreliable KGS ratings are. For kyus, expect larger differences. Therefore, before you start assessing cheating, first consider the usual variation.

Furthermore, take into account that players can have very different strengths in offline and online play (2 or 3 real world ranks difference easily).

So before cheating, you must tolerate ca. 10 ranks difference between KGS and EGF.

Before AI cheating, also consider more variation due to new throw-away accounts (win or throw away and create a next account).

If a long 10k KGS is 8d a week later, consider both a rating system anomaly or AI cheating.

Hence, before even dreaming of detecting AI cheating, correct the rating system, limit each person to one account and verify him by his identity document!

You gotta be much more specific than that. Matching is a form of confirmatory evidence, which is of extremely low value. In addition, people will count all sorts of plays as matches, as we saw in the Metta controversy.

To put it in terms of police testimony in American courtrooms these days, a high matching rate with Leela is consistent with improvement in playing ability. (It is also consistent with cheating, which is perhaps more like police testimony. )

Edit: Since nobody has posted another note, let me clarify. At this point in time, matches with Leela or any top bot, my indicate skill at go, but cannot be taken to indicate playing like a bot instead of a human. Especially if the match is loose, and not only of the bot's top choice. But show me that the player plays Leela Zero's top choice 95% of the time, that's good evidence of superhuman play. It is unlikely that even KataGo or Elf can do that!

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

10 ranks difference for people who play frequently sounds impossible.

In the SDK range, a difference of 4 ranks (e.g. 6k EGF and 2k KGS) is quite frequent, 5 ranks can happen but 10 ranks? Do you know actual examples of people who play frequently and are, say, 2d KGS and 9k EGF?

Right, for frequent players, the variation is smaller. However, even for them intermediate variation of their KGS graphs (4 ranks up or down) can occur for a day or some months. It does not affect all players.

How can we stop referees cheating? Like skipping the match, and then afterwards just making up the rules as they go along.

_________________
North Lecale

Give them training and pay them some money.

Rule for referees #1. Always make your rulings from the book.

Rule for organizers #1. Write the book.

Edit: And, I suppose, this:

Rule for referees #0: Show up.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

The flip side to this conversation is that while people can mimic an AI and jump 12 stones overnight, they can also cheat more subtly. Being a stone and a half stronger in important games by consulting the AI at a critical juncture or two and choosing a human looking suggestion is not going to be detectable from statistical analysis. So while there's value in weeding out cheating, I think it has to be assumed to be happening in any un-monitored tournament.

I'd also note that if you have a reliable algorithm for detecting AI moves, you can use adversarial learning to train a bot that doesn't look like an AI anymore. Maybe the limitation forces it to be a couple stones weaker, but that's still plenty of room to be stronger than us mere humans.

I agree that it needs to be done, but it seems like an excessive burden on admins. Their conclusions will always be challenged, and they have to live with the nagging doubts that they might be in the wrong. The only reasonable solution imo is what SoDesuNe suggested - making a server-side cheating detector bot. We have bots. We have programmers. We have models from the chess community. It seems that the problem is big enough for KGS (the AGA) to put some money into this instead of foisting the responsibility onto volunteers.

_________________
Patience, grasshopper.

I wonder if somebody can construct a ballpark estimate of how much more difficult it is to detect cheating in Go than it is in Chess.
Multiply by 3 for the length of the game? Something for the average available choices, you probably end up finding it is 1000 times more difficult.

_________________
North Lecale

There is no single level of difficulty. Detecting and proving the at most one cheating move per game is extremely much more difficult than if all moves are copied.

One sentence I'd rather never hear or read is “I thought about this move, but I feared that it would both seem very unusual and be a top bot choice, so I'd be suspected of cheating.”

_________________
A good system naturally covers all corner cases without further effort.

Well I do worry about that sometimes when I play on FlyOrDie (a site where many games can be played, including chess and go). They seem to have some bot detection. At least they flag some players suspected of cheating, but admins are rarely seen in the go group and they don't seem to know how the game is played, so I guess their bot detection is automatic.

I don't know how good their cheating detection is, so I do regularly play openings that bots won't play. I do that on purpose, because I don't want to be unjustly flagged as a cheater.

In the 2019 Annual General Meeting 2019 of the EGF, a Russian group proposed to investigate bot detection for online tournaments.

See item 13.d of 2019 AGM Minutes and the text of their proposal. I don't know the current status of their project.

You should never change your game out of fear of being considered a cheater. It's never about one move, or ten moves or even one entire game. It's about a lot of your moves matching a bot in a lot of games. Which is extremely unlikely.

Correlations between mistakes due to thinking time, playing when tired or the fact of play being online with playing strength can (but need not for each player) be much higher than a correlation between bot assistance and playing strength. Therefore, establishing correlations is hard.

E.g., in real world tournaments with long thinking times, I can reduce my blunder rate to 1/10 move per game. In KGS play, my blunder rate is more like 2 moves per game, as it had been at ca. 3 kyu level in real world tournaments with long thinking times. In real world tournaments with long thinking times, my worst moves should be dan level. In KGS play, my worst moves (overlooking atari etc.) are 30 kyu level.

Even if all that noise of 35 ranks difference in mistakes could always be detected and accounted, identifying moves as mistakes versus perfect play and the degrees of their severity is extraordinarily hard even for non-basic endgames.

I guess what they want to do: set some hypotheses: given a particular bot and its plays or evaluation - versus human plays. Assume the bot to play and evaluate correctly. Assume different human play or play depreciated by the bot analysis to be "mistakes". Measure some variance of such "mistakes". Such analysis will be utter nonsense!

Repetition of this mantra does not establish truth.

1) Many moves are "obvious" so should agree to bot moves.

2) Somebody having studied much with a particular bot can have many more same moves.

Hence it is "extremely likely" and not "extremely unlikely".

Here's a game I recently played, where I made only minor mistakes and my opponent (1k) cruised to victory without giving me any chance. Usually my games feature major swings when evaluated with AI, or occasionally there's a big mistake which turns it into a lopsided game. One sided games with only minor positional mistakes are very unusual for 1k-1d games.

What do you think?

Fairly unclear, but I'd tend toward considering it a honest game. I'd say 80% chance of it being honest. Also remember that a single game taken out of context is almost never conclusive.