Page 3 of 10
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 9:37 am
by Javaness2
I wonder if somebody can construct a ballpark estimate of how much more difficult it is to detect cheating in Go than it is in Chess.
Multiply by 3 for the length of the game? Something for the average available choices, you probably end up finding it is 1000 times more difficult.
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 10:47 am
by RobertJasiek
There is no single level of difficulty. Detecting and proving the at most one cheating move per game is extremely much more difficult than if all moves are copied.
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 4:01 pm
by Harleqin
One sentence I'd rather never hear or read is “I thought about this move, but I feared that it would both seem very unusual and be a top bot choice, so I'd be suspected of cheating.”
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 10:44 pm
by gennan
Harleqin wrote:One sentence I'd rather never hear or read is “I thought about this move, but I feared that it would both seem very unusual and be a top bot choice, so I'd be suspected of cheating.”
Well I do worry about that sometimes when I play on FlyOrDie (a site where many games can be played, including chess and go). They seem to have some bot detection. At least they flag some players suspected of cheating, but admins are rarely seen in the go group and they don't seem to know how the game is played, so I guess their bot detection is automatic.
I don't know how good their cheating detection is, so I do regularly play openings that bots won't play. I do that on purpose, because I don't want to be unjustly flagged as a cheater.
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 10:54 pm
by gennan
In the 2019 Annual General Meeting 2019 of the EGF, a Russian group proposed to investigate bot detection for online tournaments.
See item 13.d of
2019 AGM Minutes and the text of their
proposal. I don't know the current status of their project.
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 11:08 pm
by Adin
You should never change your game out of fear of being considered a cheater. It's never about one move, or ten moves or even one entire game. It's about a lot of your moves matching a bot in a lot of games. Which is extremely unlikely.
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 11:28 pm
by RobertJasiek
Correlations between mistakes due to thinking time, playing when tired or the fact of play being online with playing strength can (but need not for each player) be much higher than a correlation between bot assistance and playing strength. Therefore, establishing correlations is hard.
E.g., in real world tournaments with long thinking times, I can reduce my blunder rate to 1/10 move per game. In KGS play, my blunder rate is more like 2 moves per game, as it had been at ca. 3 kyu level in real world tournaments with long thinking times. In real world tournaments with long thinking times, my worst moves should be dan level. In KGS play, my worst moves (overlooking atari etc.) are 30 kyu level.
Even if all that noise of 35 ranks difference in mistakes could always be detected and accounted, identifying moves as mistakes versus perfect play and the degrees of their severity is extraordinarily hard even for non-basic endgames.
I guess what they want to do: set some hypotheses: given a particular bot and its plays or evaluation - versus human plays. Assume the bot to play and evaluate correctly. Assume different human play or play depreciated by the bot analysis to be "mistakes". Measure some variance of such "mistakes". Such analysis will be utter nonsense!
Re: On handling online cheating with AI
Posted: Mon Jun 01, 2020 11:39 pm
by RobertJasiek
Adin wrote:It's about a lot of your moves matching a bot in a lot of games. Which is extremely unlikely.
Repetition of this mantra does not establish truth.
1) Many moves are "obvious" so should agree to bot moves.
2) Somebody having studied much with a particular bot can have many more same moves.
Hence it is "extremely likely" and not "extremely unlikely".
Re: On handling online cheating with AI
Posted: Tue Jun 02, 2020 12:55 am
by Knotwilg
Here's a game I recently played, where I made only minor mistakes and my opponent (1k) cruised to victory without giving me any chance. Usually my games feature major swings when evaluated with AI, or occasionally there's a big mistake which turns it into a lopsided game. One sided games with only minor positional mistakes are very unusual for 1k-1d games.
What do you think?
Re: On handling online cheating with AI
Posted: Tue Jun 02, 2020 1:52 am
by Adin
What do you think?
Fairly unclear, but I'd tend toward considering it a honest game. I'd say 80% chance of it being honest. Also remember that a single game taken out of context is almost never conclusive.
Re: On handling online cheating with AI
Posted: Tue Jun 02, 2020 2:05 am
by jlt
Here is a idea to detect cheating, which needs to be refined.
I looked at one of my recent long games vs. 6k EGF. Most of the game was quite balanced. Let's define a "mistake" as a move that loses more than 5 points according to 15-block KataGo after 1000 playouts. Then among moves 31-180, we made 55 mistakes, so we made mistakes 37% of the time.
Given a move between 31 and 180 in an even game, let X=1 if a player makes a mistake, and X=0 otherwise. Let's assume that X follows the binomial law with parameter p=0.37 (this is probably wrong but let's assume that for simplicity).
Now, suppose an EGF 6k player plays 50 games, and that between moves 31 and 180, makes mistakes only 10% of the time. That's (p-0.1)*50*75/sqrt(p*(1-p)*50*75) which is about 34 standard deviations less than expected. The probability to play at that level is about 1/sqrt(2 Pi) * integral(exp(-x^2/2), x=34.. infinity) which is about 10-253.That's extremely unlikely.
(Edit: the approximation by the integral is wrong since the binomial distribution looks like a gaussian curve only around the mean, but in any case the probability is extremely small.)
Now, before banning a user, more investigation is needed, my percentage 37% was based on just one game, I have no idea whether the percentage of mistakes is similar in other EGF 6k games.
Re: On handling online cheating with AI
Posted: Tue Jun 02, 2020 3:02 am
by jlt
We can see that Knotwilg's game is of higher quality. Between moves 31 and 180,
Black made 2 mistakes larger than 5 points and 13 mistakes between 2.1 and 5 points.
White made 1 mistake larger than 5 points and 9 mistakes between 2.1 and 5 points.
In that case I would conclude that White probably didn't cheat. Or if he did, cheating was done in such a subtle way that is indetectable by game analysis.
Re: On handling online cheating with AI
Posted: Tue Jun 02, 2020 3:25 am
by Bill Spight
Adin wrote:It's about a lot of your moves matching a bot in a lot of games. Which is extremely unlikely.
Is it?
There are a lot of plausible ideas that fail empirically.
Re: On handling online cheating with AI
Posted: Tue Jun 02, 2020 4:09 am
by gennan
Adin wrote:You should never change your game out of fear of being considered a cheater. It's never about one move, or ten moves or even one entire game. It's about a lot of your moves matching a bot in a lot of games. Which is extremely unlikely.
That assumes that the bot detector is high quality. A bad detector may give many false positives. I have no way of knowing the quality of their detector.
Re: On handling online cheating with AI
Posted: Tue Jun 02, 2020 4:13 am
by Uberdude
Back on the PGETC case (2 years ago! shame we don't have better automated systems like we talked about way back then), the matching metric used as evidence someone was cheating, subsequently overturned on appeal, was "game move was in top 3 choices of Leela 0.11". Here's some stats for that matching metric, and top 1 choice, I collected. Note the pros were lower than mid and high dan amateurs, probably because they are stronger than old Leela (this was before superhuman LeelaZero, KataGo etc). The 98% that lead to the original conviction was the highest of the sample, but of course has a massive selection bias that that was the one chosen to investigate. The game Carlo crushed Dragos, which many also thought he played cheatingly strong, he had a lower top 3 matching metric (78%) than I did (80%, 84%) in games I know for sure I didn't cheat.
Code: Select all
+-----------------+------+----------------+------+---------+---------+---------+---------+
| Black | Rank | White | Rank | B top 3 | W top 3 | B top 1 | W top 1 |
+-----------------+------+----------------+------+---------+---------+---------+---------+
| [Carlo Metta] | 4d | Reem Ben David | 4d | * 98 | 80 | * 72 | 54 | http://pandanet-igs.com/system/sgfs/6374/original/WWIWTFDSGS.sgf
| Andrey Kulkov | 6d | [Carlo Metta] | 4d | 80 | * 86 | 68 | * 62 | http://pandanet-igs.com/system/sgfs/6314/original/AMTRMFSDAB.sgf
| Dragos Bajenaru | 6d | [Carlo Metta] | 4d | 74 | * 78 | 50 | * 60 | http://pandanet-igs.com/system/sgfs/6354/original/JRZPCWSANY.sgf
| [Andrew Simons] | 4d | Jostein Flood | 3d | 80 | 88 | 54 | 62 | http://pandanet-igs.com/system/sgfs/6612/original/XSJUGZZTOX.sgf
| Geert Groenen | 5d | [Daniel Hu] | 4d | 74 | 66 | 40 | 46 | http://britgo.org/files/pandanet2016/mathmo-GGroenen-2017-01-10.sgf
| [Ilya Shikshin] | 1p | Artem Kachan. | 1p | 56 | 76 | 38 | 60 | http://pandanet-igs.com/system/sgfs/6384/original/RYSGTEGMXT.sgf
| [Andrew Simons] | 4d | Victor Chow | 7d | 84 | 76 | 44 | 44 | http://britgo.org/files/pandanet2014/RoseDuke-Egmump-2015-01-13.sgf
| Cornel Burzo | 6d | [A. Dinerstein]| 3p | 74 | 66 | 40 | 48 | http://pandanet-igs.com/system/sgfs/6349/original/SCNSFSJXTI.sgf
| Jonas Welticke | 6d | [Daniel Hu] | 4d | 54 | 64 | 34 | 42 | http://britgo.org/files/pandanet2017/mathmo-iryumika-2017-12-12.sgf
| [Park Junghwan] | 9p | Lee Sedol | 9p | 74 | 64 | 64 | 38 | http://www.go4go.net/go/games/sgfview/68053
| Lothar Spiegel | 5d | [Daniel Hu] | 4d | 66 | 58 | 48 | 42 | http://britgo.org/files/pandanet2016/mathmo-Mekanik-2017-04-25.sgf
| Gilles v.Eeden | 6d | [Viktor Lin] | 6d | 82 | 70 | 56 | 46 | http://pandanet-igs.com/system/sgfs/6616/original/FMKVQBHBBV.sgf
+-----------------+------+----------------+------+---------+---------+---------+---------+
As a chart at
https://www.lifein19x19.com/viewtopic.p ... 60#p229460