Can you determine the use of AI based on 4 games?

[Sampi]

Based on only 4 games, is it possible to determine the use of AI?

Game 1:


Game2:

Game 3:

Game 4:

[Bill Spight]

What do you mean, the use of AI?

What is the evidence of using AI in go, except becoming a better player?

As for using AI to cheat, I suppose the giveaway is relatively poor reading skill.

[jlt]

Online cheating is very difficult to detect, read for instance

https://www.nordicgodojo.eu/post/33/how ... -online-go

[Uberdude]

Put the games in the embedded eidogo player and forum members will more likely examine them.

I presume you mean using AI during the game, rather than to study before the game.

Some games of the same player where he is known to not be using AI (eg before strong bots if he was same strength years ago, in person game records where more easily believed to have not been cheating) for comparison allow stronger conclusions.

[Harleqin]

I think that just from a game by itself, you cannot say whether AI was consulted. If you know a player's recent games, you may be able to make a case that the play in the game is improbable, but I suspect this will be not convincing enough for anything approaching dan levels. It becomes hopeless if players use AI only in some moves.

So, proctors seem like the only solution to me.

Using a VR set as a “proctor”, as Antti proposed, is interesting, but also seems too easy to spoof to me. Besides, I am a bit concerned about the development of surveillance technologies.

Another option could be a two-webcam setup (one from the front, one from the back), but again, it seems easy to spoof.

In both VR set and two-webcam setup I would feel quite uncomfortable that any movement I make or do not make might be seen as suspicious. It's the classic surveillance conundrum: do you really want to put everyone under suspicion just to catch the few bad actors? I don't want to get philosophical here, but the answer should generally be “no”.

In conclusion, I think that actual human proctors are the only way forward. Proxy venues, as Marcel proposed, seem like a very useful thing to me.

There is one more option: play only bots, or care not whether your opponent is one, but this is of course not realistic for tournaments.

[Uberdude]

The play is obviously stronger than any of the KGS ranks next to his name. Without checking vs AI, I'd say first game he hit all the opening points AI likes, which a mid-dan player like myself who has trained with AI can identify, though the chance of getting a dozen such decisions right in a row diminishes. I'd be tempted to atari at a10 before defending, but is that bad human atari love because a11 is also sente? If bot doesn't want to that's another point towards bot cheating, though of course strong humans can also resist ataris.

Game 3 loss ending looks like a misclick/too eager click of expecting W to answer at h16 (0 point gote) and then playing g6 as next planned move without realising white actually played h4 sente. This could happen with a non-cheating human not paying attention, but switching between a bot and the game board probably makes such mistakes more likely.

Game 4 move 32 is a move I learnt from bots, likewise 34 not being slow but good.

[Bill Spight]

The play is obviously stronger than any of the KGS ranks next to his name.

Right. We have a sandbagger, to be sure. But an AI aided sandbagger? Quien sabe?

I'd be tempted to atari at a10 before defending, but is that bad human atari love because a11 is also sente? If bot doesn't want to that's another point towards bot cheating, though of course strong humans can also resist ataris.

The sagari to a11 is superior. White's reply is the ugly connection at b9. Ugh! As for the protection, I did not learn the keima connection until I was 4 kyu.

But what about ? That's a human looking move, isn't it? It induces White to make good shape at D-06, weakening the nearby Black stone at D-08. And what does Black gain? Isn't it better to leave D-06 as an option for Black?

[jlt]

15-block Katago likes both: defend directly or atari first. Katago likes . In fact, most black moves are among Katago's suggestions, with a few exceptions like , , but there are no big mistakes.

[Bill Spight]

15-block Katago likes both: defend directly or atari first. Katago likes . In fact, most black moves are among Katago's suggestions, with a few exceptions like , , but there are no big mistakes.

Hmmmm. Recently didn't we find around 80% concordance between KataGo and LZ? A nearly 100% concordance is amazing.

Previously I have pointed out the weakness of confirmatory evidence, in this case, agreement between the human's play and the bot's choice. The evidence here is not that agreement, per se, but the difference between that rate of agreement and the rate of agreement between two top bots, or the rate of agreement between a top bot and a professional.

[jlt]

It depends what you mean by "considered". The hane is sixth in terms of policy, and loses 3.5% compared to B3, so it was "considered" by Katago only in a weak sense.

[Bill Spight]

It depends what you mean by "considered". The hane is sixth in terms of policy, and loses 3.5% compared to B3, so it was "considered" by Katago only in a weak sense.

Thanks for the clarification.

I brought up concordance (exact match between the game play and the bot's top choice) because it is something on which we have hard, if currently limited, evidence. But it should not be hard to gather, as far as professional play is concerned.

For instance, in the Elf commentaries all of Elf's top choices are tagged as tesuji, TE[1]. So you can just count how often that tag appeears in the game record, excluding variations, and compare with the number of plays. Take GoGoD 2018-05-28l, with Meng Tailing, 6 dan, (W) vs. Kang Tongyun, 9 dan. There were 269 plays, 116 of which were also Elf's top choice, for a concordance of 43.1%. It only took me a few minutes to get that number. OC, that is just one game, but IIRC the concordance rate between pros and top bots is around 50%.

Edit: Not that it is a good way to cheat, but apparently some people just do play the AI's top choice. If you are checking for that kind of cheating, you want to make it difficult to find a match, and then if you find a higher rate of matching than top humans have, you have some evidence.

[jlt]

In game 1, Black chose Katago's top choice 37% of the time, so if Black is a high dan amateur player I don't see any evidence of cheating based solely on concordance with AI moves.

[Javaness2]

Just 4 games is really unlikely to be enough to determine if a player was using an AI or not.
But why the question?

[Sampi]

These responses are really disappointing and make me sad as hell. I'm no sure if you people are blind, complicit or just plain dumb.

I don't see any evidence of cheating based solely on concordance with AI moves.

If you don't see any evidence, that's because you didn't bother to check.
Seriously, the moves are mostly (I'd say around 95%) LeelaZero suggestions. Do moves 103 to 109 in game 4 look normal? How about moves 244 to 252 in that same game? How about checking the moves against the LeelaZero that came with the previous version of Lizzie (you know, the one everyone has, the easiest to get)?

I don't care about punishing this person specifically, I honestly don't, we all make mistakes. What I do care about is how cheating is detected and handled, specially in tournaments where there's prizes. What really frustrates me is that this person is so blatant and obvious, and organizers (and apparently people in this forum) can't make up their minds. Check any pro game against LeelaZero and you see at least some divergence (I'd say at most 90%) and nobody plays the moves in exactly the same order.

I fear some people just checked parts of some of the games and once they saw a divergence in 1 or 2 moves, decided it was impossible to know for sure. This person was very blatant, and even after these games specifically were brought to the attention of organizers (and you guys), there's still doubt? How?

If it's this easy, I might as well start cheating in online tournaments. People are too easy to fool.

[Kirby]

If it's this easy, I might as well start cheating in online tournaments. People are too easy to fool.

With the various AI programs available today, I don't think it's hard to cheat. Given the strength of widely available AIs, I think we've crossed that bridge.

So you can probably go ahead and cheat and win a tournament without getting caught. But why? I guess if it's for prize money, there's some incentive there. But there're probably easier ways to make money.

Tournaments are fun for me because you can test your strength, and get the real pressure of playing a serious game where the other person is trying to win, too. I don't think I'd get that fun if I used AI, because I'd know that the person playing wasn't me.