It is currently Thu Mar 28, 2024 1:41 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 147 posts ]  Go to page Previous  1 ... 4, 5, 6, 7, 8
Author Message
Offline
 Post subject: Re: On handling online cheating with AI
Post #141 Posted: Thu Jun 11, 2020 3:31 pm 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
NordicGoDojo wrote:
I have analysed a considerable number of games with KataGo looking at the size of a player's 'average mistake' – not in terms of winrate-%, but points, as KataGo is able to do. I think this method is more robust than looking at the winrate, because even a very small mistake can cause a big winrate change when the game is close.


Good point. In addition, while evaluation in terms of points only is theoretically problematic, because it ignores the value of having the move (sente), it is much closer to how humans evaluate positions than winrate estimates are.

Quote:
After analysing a few players, Ke Jie's average mistake seemed to be in the range of -0.5 points per move. For my own games, I got around -0.8 points; then for Shūsaku I got around -1.2 points, at which point I started to get suspicious. Then I checked a few European 6d players, who came to around -1.5 points; and then I came upon a game by Lukas Podpera 7d and Tanguy Le Calvé, which had an average mistake of only -0.3 points per move for both players.

Investigating further, I realised that the size of the average mistake depends on the 'nature' of the game: fighting-oriented games inevitable lead to higher average mistakes and peaceful games lead to lower average mistake. This is why Ke Jie's -0.5 points per move is impressive. Even if we analysed winrate-% rather than KataGo-points, I believe we would get the same conclusion.


Interesting. I think for this kind of approach we need to profile a large number of players, not just in terms of average or median mistakes, but of the whole distribution of errors. Furthermore, since the value of sente changes during the game, we should look at profiles of errors at different stages of the game, which might also reflect different styles of play.

Ideally, we would be able to come up with a finite number of profiles of honest play. Then not fitting any of those honest profiles would be evidence of possible cheating. I.e., you got some 'splainin' to do.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #142 Posted: Thu Jun 11, 2020 6:11 pm 
Beginner

Posts: 19
Liked others: 2
Was liked: 8
Rank: 1p
Bill Spight wrote:

Good point. In addition, while evaluation in terms of points only is theoretically problematic, because it ignores the value of having the move (sente), it is much closer to how humans evaluate positions than winrate estimates are.



What do you mean? E.g., if you pass at the start of the game, KataGo thinks you made a loss of roughly 13 points.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #143 Posted: Thu Jun 11, 2020 8:30 pm 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
NordicGoDojo wrote:
Bill Spight wrote:

Good point. In addition, while evaluation in terms of points only is theoretically problematic, because it ignores the value of having the move (sente), it is much closer to how humans evaluate positions than winrate estimates are.



What do you mean? E.g., if you pass at the start of the game, KataGo thinks you made a loss of roughly 13 points.


Sorry, I misspoke. It's human evaluation in terms of points that does not take the value of the move into account.

The problem with evaluation by points alone is that the payoff is not in terms of points.

Edit: But the value of sente still affects the error function for points.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


Last edited by Bill Spight on Fri Jun 12, 2020 2:52 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #144 Posted: Fri Jun 12, 2020 12:04 am 
Dies in gote

Posts: 52
Liked others: 2
Was liked: 12
Knotwilg wrote:
If you have learned so much that you can stay close to AI choices throughout the whole game, then you've reached professional strength. To do that in a very short time span is suspicious. To play mostly at AI level and make a few kyu level blunders, is also suspicious.


Now that is an assumption :-)

Either way, it is a bit controversial to which extent one can act on 'suspicion' as has been discussed in this thread.

Perhaps we should not look at 'punishing' suspicious players with 'flags' and what not - just give them an untimely promotion.

Then we can focus our attention on cheaters at the highest ranks, where it should be easier to find 'superhuman play'.

(yes, this of course also has its downsides..)

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #145 Posted: Fri Jun 12, 2020 1:40 am 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
Is cheating really a big problem at levels other than high dan ranks? Do some of you frequently encounter players that you suspect of using AI?

(I am not talking about tournaments, only about usual games on a server.)

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #146 Posted: Fri Jun 12, 2020 4:24 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
jlt wrote:
Is cheating really a big problem at levels other than high dan ranks? Do some of you frequently encounter players that you suspect of using AI?

(I am not talking about tournaments, only about usual games on a server.)


I play about 10-20 (casual) games a week online an I've never encountered an opponent who I suspected of cheating. This may depend on the server though.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #147 Posted: Mon Aug 03, 2020 9:07 am 
Lives in gote

Posts: 474
Liked others: 120
Was liked: 157
Rank: igs 4d+
jlt wrote:
Do some of you frequently encounter players that you suspect of using AI?
Here is the graph of a player I saw on IGS. After more than a year of being 4d and having a 50% win/loss ratio, (s)he is now up to 7d and 80% wins since mid-july 2020. His/her results in the last 20 games, played over 4 days against opponents ranked 5d+ to 8d, are 19 wins - 1 loss (on time).
I checked randomly a dozen of his/her games with Leelazero: almost all followed the same pattern, with the player at hand taking an early lead an never releasing it.

But I'm sure someone will come forward and say that it is possible to improve so much and so fast because he knows the cousin of a friend of his sister-in-law who did, and that we can do absolutely nothing against cheating online because we can never know for sure.


Attachments:
graph.jpg
graph.jpg [ 44.34 KiB | Viewed 4071 times ]

This post by Shenoute was liked by: jlt
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 147 posts ]  Go to page Previous  1 ... 4, 5, 6, 7, 8

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group