It is currently Fri Mar 29, 2024 2:46 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 147 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 8  Next
Author Message
Offline
 Post subject: Re: On handling online cheating with AI
Post #41 Posted: Tue Jun 02, 2020 2:05 am 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
Here is a idea to detect cheating, which needs to be refined.

I looked at one of my recent long games vs. 6k EGF. Most of the game was quite balanced. Let's define a "mistake" as a move that loses more than 5 points according to 15-block KataGo after 1000 playouts. Then among moves 31-180, we made 55 mistakes, so we made mistakes 37% of the time.

Given a move between 31 and 180 in an even game, let X=1 if a player makes a mistake, and X=0 otherwise. Let's assume that X follows the binomial law with parameter p=0.37 (this is probably wrong but let's assume that for simplicity).

Now, suppose an EGF 6k player plays 50 games, and that between moves 31 and 180, makes mistakes only 10% of the time. That's (p-0.1)*50*75/sqrt(p*(1-p)*50*75) which is about 34 standard deviations less than expected. The probability to play at that level is about 1/sqrt(2 Pi) * integral(exp(-x^2/2), x=34.. infinity) which is about 10-253.That's extremely unlikely.

(Edit: the approximation by the integral is wrong since the binomial distribution looks like a gaussian curve only around the mean, but in any case the probability is extremely small.)

Now, before banning a user, more investigation is needed, my percentage 37% was based on just one game, I have no idea whether the percentage of mistakes is similar in other EGF 6k games.


Last edited by jlt on Tue Jun 02, 2020 3:50 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #42 Posted: Tue Jun 02, 2020 3:02 am 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
We can see that Knotwilg's game is of higher quality. Between moves 31 and 180,

Black made 2 mistakes larger than 5 points and 13 mistakes between 2.1 and 5 points.
White made 1 mistake larger than 5 points and 9 mistakes between 2.1 and 5 points.

In that case I would conclude that White probably didn't cheat. Or if he did, cheating was done in such a subtle way that is indetectable by game analysis.


This post by jlt was liked by: Knotwilg
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #43 Posted: Tue Jun 02, 2020 3:25 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Adin wrote:
It's about a lot of your moves matching a bot in a lot of games. Which is extremely unlikely.


Is it?

There are a lot of plausible ideas that fail empirically. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #44 Posted: Tue Jun 02, 2020 4:09 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Adin wrote:
You should never change your game out of fear of being considered a cheater. It's never about one move, or ten moves or even one entire game. It's about a lot of your moves matching a bot in a lot of games. Which is extremely unlikely.

That assumes that the bot detector is high quality. A bad detector may give many false positives. I have no way of knowing the quality of their detector.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #45 Posted: Tue Jun 02, 2020 4:13 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Back on the PGETC case (2 years ago! shame we don't have better automated systems like we talked about way back then), the matching metric used as evidence someone was cheating, subsequently overturned on appeal, was "game move was in top 3 choices of Leela 0.11". Here's some stats for that matching metric, and top 1 choice, I collected. Note the pros were lower than mid and high dan amateurs, probably because they are stronger than old Leela (this was before superhuman LeelaZero, KataGo etc). The 98% that lead to the original conviction was the highest of the sample, but of course has a massive selection bias that that was the one chosen to investigate. The game Carlo crushed Dragos, which many also thought he played cheatingly strong, he had a lower top 3 matching metric (78%) than I did (80%, 84%) in games I know for sure I didn't cheat.

Code:
+-----------------+------+----------------+------+---------+---------+---------+---------+
|      Black      | Rank |     White      | Rank | B top 3 | W top 3 | B top 1 | W top 1 |
+-----------------+------+----------------+------+---------+---------+---------+---------+
| [Carlo Metta]   |  4d  | Reem Ben David |  4d  |    * 98 |      80 |    * 72 |      54 |   http://pandanet-igs.com/system/sgfs/6374/original/WWIWTFDSGS.sgf
| Andrey Kulkov   |  6d  | [Carlo Metta]  |  4d  |      80 |    * 86 |      68 |    * 62 |   http://pandanet-igs.com/system/sgfs/6314/original/AMTRMFSDAB.sgf
| Dragos Bajenaru |  6d  | [Carlo Metta]  |  4d  |      74 |    * 78 |      50 |    * 60 |   http://pandanet-igs.com/system/sgfs/6354/original/JRZPCWSANY.sgf
| [Andrew Simons] |  4d  | Jostein Flood  |  3d  |      80 |      88 |      54 |      62 |   http://pandanet-igs.com/system/sgfs/6612/original/XSJUGZZTOX.sgf
| Geert Groenen   |  5d  | [Daniel Hu]    |  4d  |      74 |      66 |      40 |      46 |   http://britgo.org/files/pandanet2016/mathmo-GGroenen-2017-01-10.sgf
| [Ilya Shikshin] |  1p  | Artem Kachan.  |  1p  |      56 |      76 |      38 |      60 |   http://pandanet-igs.com/system/sgfs/6384/original/RYSGTEGMXT.sgf
| [Andrew Simons] |  4d  | Victor Chow    |  7d  |      84 |      76 |      44 |      44 |   http://britgo.org/files/pandanet2014/RoseDuke-Egmump-2015-01-13.sgf
| Cornel Burzo    |  6d  | [A. Dinerstein]|  3p  |      74 |      66 |      40 |      48 |   http://pandanet-igs.com/system/sgfs/6349/original/SCNSFSJXTI.sgf
| Jonas Welticke  |  6d  | [Daniel Hu]    |  4d  |      54 |      64 |      34 |      42 |   http://britgo.org/files/pandanet2017/mathmo-iryumika-2017-12-12.sgf
| [Park Junghwan] |  9p  | Lee Sedol      |  9p  |      74 |      64 |      64 |      38 |   http://www.go4go.net/go/games/sgfview/68053
| Lothar Spiegel  |  5d  | [Daniel Hu]    |  4d  |      66 |      58 |      48 |      42 |   http://britgo.org/files/pandanet2016/mathmo-Mekanik-2017-04-25.sgf
| Gilles v.Eeden  |  6d  | [Viktor Lin]   |  6d  |      82 |      70 |      56 |      46 |   http://pandanet-igs.com/system/sgfs/6616/original/FMKVQBHBBV.sgf
+-----------------+------+----------------+------+---------+---------+---------+---------+


As a chart at https://www.lifein19x19.com/viewtopic.p ... 60#p229460

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #46 Posted: Tue Jun 02, 2020 4:33 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Uberdude wrote:
Back on the PGETC case (2 years ago! shame we don't have better automated systems like we talked about way back then), the matching metric used as evidence someone was cheating, subsequently overturned on appeal, was "game move was in top 3 choices of Leela 0.11". Here's some stats for that matching metric, and top 1 choice, I collected. Note the pros were lower than mid and high dan amateurs, probably because they are stronger than old Leela (this was before superhuman LeelaZero, KataGo etc).

Code:
+-----------------+------+----------------+------+---------+---------+---------+---------+
|      Black      | Rank |     White      | Rank | B top 3 | W top 3 | B top 1 | W top 1 |
+-----------------+------+----------------+------+---------+---------+---------+---------+
| [Carlo Metta]   |  4d  | Reem Ben David |  4d  |    * 98 |      80 |    * 72 |      54 |   http://pandanet-igs.com/system/sgfs/6374/original/WWIWTFDSGS.sgf
| Andrey Kulkov   |  6d  | [Carlo Metta]  |  4d  |      80 |    * 86 |      68 |    * 62 |   http://pandanet-igs.com/system/sgfs/6314/original/AMTRMFSDAB.sgf
| Dragos Bajenaru |  6d  | [Carlo Metta]  |  4d  |      74 |    * 78 |      50 |    * 60 |   http://pandanet-igs.com/system/sgfs/6354/original/JRZPCWSANY.sgf
| [Andrew Simons] |  4d  | Jostein Flood  |  3d  |      80 |      88 |      54 |      62 |   http://pandanet-igs.com/system/sgfs/6612/original/XSJUGZZTOX.sgf
| Geert Groenen   |  5d  | [Daniel Hu]    |  4d  |      74 |      66 |      40 |      46 |   http://britgo.org/files/pandanet2016/mathmo-GGroenen-2017-01-10.sgf
| [Ilya Shikshin] |  1p  | Artem Kachan.  |  1p  |      56 |      76 |      38 |      60 |   http://pandanet-igs.com/system/sgfs/6384/original/RYSGTEGMXT.sgf
| [Andrew Simons] |  4d  | Victor Chow    |  7d  |      84 |      76 |      44 |      44 |   http://britgo.org/files/pandanet2014/RoseDuke-Egmump-2015-01-13.sgf
| Cornel Burzo    |  6d  | [A. Dinerstein]|  3p  |      74 |      66 |      40 |      48 |   http://pandanet-igs.com/system/sgfs/6349/original/SCNSFSJXTI.sgf
| Jonas Welticke  |  6d  | [Daniel Hu]    |  4d  |      54 |      64 |      34 |      42 |   http://britgo.org/files/pandanet2017/mathmo-iryumika-2017-12-12.sgf
| [Park Junghwan] |  9p  | Lee Sedol      |  9p  |      74 |      64 |      64 |      38 |   http://www.go4go.net/go/games/sgfview/68053
| Lothar Spiegel  |  5d  | [Daniel Hu]    |  4d  |      66 |      58 |      48 |      42 |   http://britgo.org/files/pandanet2016/mathmo-Mekanik-2017-04-25.sgf
| Gilles v.Eeden  |  6d  | [Viktor Lin]   |  6d  |      82 |      70 |      56 |      46 |   http://pandanet-igs.com/system/sgfs/6616/original/FMKVQBHBBV.sgf
+-----------------+------+----------------+------+---------+---------+---------+---------+


As a chart at https://www.lifein19x19.com/viewtopic.p ... 60#p229460

From that table, I would say that the data is inconclusive, because there is not a lot of correlation between the player grade and the match percentage.
For example, Carlo Metta 4d, Reem Ben David 4d, Jostein Flood 3d and Andrew Simons 4d all have a higher match than than the average 6d+ players.
I suppose that might be explained by these players training a lot with Leela, perhaps specifically to prepare for this tournament.

The 98% in the first row is very high, but his opponent also had a high top 3 match in that game. So it could still be explained by this particular game happening to be similar to the kind of game that Leela plays, perhaps because both players trained a lot with Leela.

So my feeling is that these number are insufficient to draw a solid conclusion. I think the game content should at least be scrutinized in more detail by some human experts (do they exist?).

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #47 Posted: Tue Jun 02, 2020 4:44 am 
Gosei

Posts: 1494
Liked others: 111
Was liked: 315
In a combination of naive thought, and vague recollection of what Ken Regan was doing, don't you need something like a playing strength histogram for all ply.

_________________
North Lecale

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #48 Posted: Tue Jun 02, 2020 5:30 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Javaness2 wrote:
Don't you need to look at the percentage of matches, rather than the number of matches?
In a combination of naive thought, and vague recollection of what Ken Regan was doing, don't you need something like a playing strength histogram for all ply.


The metric in my table was % of matches between some moves numbers I can't recall, maybe 30 and 130. I don't think it's a good way of detecting cheaters. I'd have thought, now we have KataGo able to give measurements of mistakes in points rather than winrate (which is pretty useless if one player leading "big"), that the kind of point loss histogram like moha made in https://www.lifein19x19.com/viewtopic.p ... 89#p255489 is the way to go if you want to compare distributions. Even better is some way to focus how well they played the difficult rather than easy moves (e.g. moves matching LZ that don't match Leela classic, which can be taken as an approximation of a strong human amateur pre AI), but even that doesn't cope with how humans play like AI now (but most obviously in the opening, so perhaps only look at moves e.g. 40+).

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #49 Posted: Tue Jun 02, 2020 6:08 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
On telling bot play from human play

OC, on average the bot plays are better. And perhaps the question should be how good a player's plays are. Ken Regan developed a way of giving chess plays ELO ratings.

Following are 12 examples where one play was made by a human, but a bot (Elf in this case, as the examples come from the Elf commentaries) not only gave a different play as its top choice, but gave the human play very few rollouts. To control for the quality of the play, their winrate estimates are within 2.0%. Since winrate estimates of plays with few rollouts are unreliable, Elf took the estimate from its winrate estimate of its choice of reply to the human play.

Example 1

Click Here To Show Diagram Code
[go]$$Wc Move 2
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . a . . . |
$$ | . . b . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]


Of a or b, which did Elf choose, which did the human choose? The winrate difference is 1.2%. Elf's choice got 23k rollouts, the human play got 2. (Not 2k, 2. ;))

This one is easy, if you have studied bot preferences. The bot prefers the 4-4.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #50 Posted: Tue Jun 02, 2020 6:20 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 2

Maybe not so easy.

Click Here To Show Diagram Code
[go]$$Wc Move 6
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X , a . . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . O . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . O . . . . . . . . . . . . X b . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]


Who played the high approach? Who played the attachment?

Winrate difference, 1.6%. Elf' rollouts: 28k, human's rollouts: 1.

Elf chose the attachment. A different bot might pick the approach, I suppose.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #51 Posted: Tue Jun 02, 2020 6:33 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 3.

Click Here To Show Diagram Code
[go]$$Bc Move 23
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X X O . a . . O . . . . . . . . |
$$ | . . X , O . O . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . O . . . |
$$ | . . . , . b . . . , . . . X O , . . . |
$$ | . . O . . . . . . . . . . X O X O . . |
$$ | . . . . . . . . . . . . X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]


Who played the invasion, who played the approach?

Winrate difference: 1.9%. Elf's play got 21k rollouts, the human play got 183.

Elf played the invasion.

I might have gotten this one, as the approach is close to Black's strength in the bottom right.

Elf's main variation:

Click Here To Show Diagram Code
[go]$$Bcm23 Mainline
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . 2 . . . . . . . . . . . . |
$$ | . . . X X O . 1 . . O . . 3 . . . . . |
$$ | . . X , O . O a . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . 4 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . 8 . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . 6 . . . . . . . . . . . O . . . |
$$ | . . . , . . 7 . . , . . . X O , . . . |
$$ | . . O . 5 . . . . . . . . X O X O . . |
$$ | . . . . . . . . . . . . X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]

Instead of playing :b25: first to prepare the invasion, Elf starts with the invasion. We might consider the invasion a probe, as White might choose a different reply, such as a. Elf prefers the one space low approach to the bottom left corner.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #52 Posted: Tue Jun 02, 2020 6:48 am 
Dies in gote

Posts: 28
Liked others: 0
Was liked: 2
Rank: 1 kyu
What's the point of flooding this thread with endless diagrams? This is not what it's about.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #53 Posted: Tue Jun 02, 2020 6:52 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 4.

Click Here To Show Diagram Code
[go]$$Wc Move 28
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . a O . . . . |
$$ | . . . X X O . . . . O . X . X . . . . |
$$ | . . X , O . O . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . b . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . O . . . . . . . . . . . . O . . . |
$$ | . . . , . X . . . , . . . X O , . . . |
$$ | . . O . . . . . . . . . . X O X O . . |
$$ | . . . . . . . . . . . . X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]


Who crawled in the corner, who extended on the side?

Winrate difference: 0.6%. Elf's play got 53k rollouts, the human's play got 0.

Elf played the crawl.

Well the number of rollouts was perhaps a big clue. How many rollouts does it take to explore an extension on the side?

Elf's main variation:

Click Here To Show Diagram Code
[go]$$Wcm28 Mainline
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . 3 1 O . . . . |
$$ | . . . X X O . . . . O . X . X . 2 . . |
$$ | . . X , O . O . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . 9 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . 7 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . 6 . . |
$$ | . . . . . . . . . . . . . . 8 . . . . |
$$ | . . O . . . . . . . . . . . . O . . . |
$$ | . . . 4 . X . . . , . . . X O , . . . |
$$ | . . O 5 . . . . . . . . . X O X O . . |
$$ | . . . . . . . . . . . . X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]


After :w28:, :b29: took the 3-3 and then :w30: connected underneath. Black's approach with :b33: is interesting, aiming at :b35:, a key point for the joseki in the bottom right. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #54 Posted: Tue Jun 02, 2020 6:54 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Adin wrote:
What's the point of flooding this thread with endless diagrams? This is not what it's about.


The point of the diagrams is telling the difference between human plays and bot plays. Which is the point of the human matching the bot's plays or not, isn't it?

If matching is relevant, so is not matching.

Edit: Besides, by choosing human plays that got few rollouts, if any, it is unlikely that they would match any of the bot's top three choices.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


Last edited by Bill Spight on Tue Jun 02, 2020 7:00 am, edited 3 times in total.
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #55 Posted: Tue Jun 02, 2020 6:57 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Whatever method the bot detector is using, I think the bot detector should be tested thoroughly (similar to clinical testing) to determine the amount of false positives and false negatives it gives in different scenarios (multiple players who train a lot with AI; no cheating, cheating only for some important moves, cheating for many moves per game, activily trying to outsmart the detector while cheating, actively trying to play very AI-like without cheating, ...).

From such testing we may determine if we can get sufficiently close to Blackstone's ratio.


This post by gennan was liked by: Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #56 Posted: Tue Jun 02, 2020 7:14 am 
Dies in gote

Posts: 28
Liked others: 0
Was liked: 2
Rank: 1 kyu
Quote:
The point of the diagrams is telling the difference between human plays and bot plays.

The ones reading this forum are humans, not bots or bot detectors. And it does not matter what humans think looking at those diagrams because very subtle value differences between reasonable plays is certainly NOT what a human investigator is looking for. What a human investigator is looking for is mostly plays that are highly unusual for the rank of the player (or for any human players). And other stuff which I will not detail here in case cheaters are reading. But certainly not if the first move is 4X4 or 3X3.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #57 Posted: Tue Jun 02, 2020 7:19 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 5

Click Here To Show Diagram Code
[go]$$Wc Move 66
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . X O O . . . |
$$ | . . . X X O . . . . O . X X X O O O . |
$$ | . . X , O . O . . , . . . . O X X O . |
$$ | . . . . . . . . . . . . . . O . . X . |
$$ | . . X . . . . . . . . . . X . X X . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . . . X . . . . X O . . . |
$$ | . . O . . . . a X O X X . . . O . . . |
$$ | . . . O . X b X O O O . . X O , . . . |
$$ | . . O X X X O . O . . . . X O X O . . |
$$ | . . O . . X O . . . . . X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]


Who played the atari, who pushed in between?

Winrate difference: 0.4%. Elf's play got 111k rollouts, the human's play got 475.

Elf pushed in between.

Elf's main variation.

Click Here To Show Diagram Code
[go]$$Wcm66 Mainline
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . X O O . . . |
$$ | . . . X X O . . . . O . X X X O O O . |
$$ | . . X , O . O . . , . . . . O X X O . |
$$ | . . . . . . . . . . . . . . O . . X . |
$$ | . . X . . . . . . . . . . X . X X . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . . 4 X . . . . X O . . . |
$$ | . . O . . . 2 . X O X X . . . O . . . |
$$ | . . . O . X 1 X O O O . . X O , . . . |
$$ | . . O X X X O 3 O . . . . X O X O . . |
$$ | . . O . . X O . . . 5 . X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #58 Posted: Tue Jun 02, 2020 7:27 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Adin wrote:
Quote:
The point of the diagrams is telling the difference between human plays and bot plays.

The ones reading this forum are humans, not bots or bot detectors. And it does not matter what humans think looking at those diagrams because very subtle value differences between reasonable plays is certainly NOT what a human investigator is looking for. What a human investigator is looking for is mostly plays that are highly unusual for the rank of the player (or for any human players). And other stuff which I will not detail here in case cheaters are reading. But certainly not if the first move is 4X4 or 3X3.
(Emphasis mine.)

If you are looking for unusual plays for a human, then you should be able to see one or two in these comparisons, shouldn't you?

As for including the example of move 2, as a scientist I decided upon my criteria first, without considering which moves might meet them. The human play gets few rollouts, so that it is unlikely to match a zero bot's plays, and it is a good play, since it is necessary to control for the quality of play. Detecting cheating by the quality of play is rather different from detecting cheating by how unusual a play is for humans.

Edit: I could have omitted the move 2 example, but I did not want to omit any data. That would have been poor practice.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


This post by Bill Spight was liked by: Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #59 Posted: Tue Jun 02, 2020 7:50 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 6.

Click Here To Show Diagram Code
[go]$$Bc Move 93
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . O . . X X X . X O O . . . |
$$ | . . . X X O . X X O O O X X X O O O . |
$$ | . . X , O . O . O X . . . . O X X O . |
$$ | . . X O . . . . O . O a . . O . . X . |
$$ | . . X . . X O . . . . . . X . X X . . |
$$ | . . . . . . X . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . X X X . . . . X O . . . |
$$ | . . O . O X X O X O X X . . . O . . . |
$$ | . . . O . X O . O O O . . X O , . . . |
$$ | . . O X X X O O O . b . . X O X O . . |
$$ | . . O . . X O . . . O . X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]

Who played that attachment, who played the wedge?

The winrate difference is 0.1% (in favor of the human's play). Elf's play got 30k rollouts, the human's play got 395.

Elf played the wedge.

Elf's main variation is rather interesting. Bots can do local reading if they want to, it seems. :)

Click Here To Show Diagram Code
[go]$$Bcm93 Mainline
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . O . . X X X . X O O . . . |
$$ | . . . X X O . X X O O O X X X O O O . |
$$ | . . X , O . O . O X . . . . O X X O . |
$$ | . . X O . . . . O . O . . . O . . X . |
$$ | . . X . . X O . . . . . . X . X X . . |
$$ | . . . . . . X . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . X X X . . . . X O . . . |
$$ | . . O . O X X O X O X X . . . O . . . |
$$ | . . . O . X O . O O O 8 . X O , . . . |
$$ | . . O X X X O O O 4 1 2 . X O X O . . |
$$ | . . O . . X O . 7 9 O 3 X . X O . . . |
$$ | . . . . . . . . a 6 5 0 . X . . . . . |
$$ ---------------------------------------[/go]

(B 103 at a, for a ko.)
:w96: could have played the sagari at 97. The wedge may be considered as a probe.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #60 Posted: Tue Jun 02, 2020 8:13 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 7.

Click Here To Show Diagram Code
[go]$$B Move 101
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . O . . X X X . X O O . . . |
$$ | . . . X X O . X X O O O X X X O O O . |
$$ | . . X , O . O . O X . . . . O X X O . |
$$ | . . X O . . a . O O O X . . O . . X . |
$$ | . . X . . X O . . . . . . X . X X . . |
$$ | . . . . . . X . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . X O X . . . . . . . . . . . . O . . |
$$ | . . X O O . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . X X X . . . . X O . . . |
$$ | . . O . O X X O X O X X . . . O . . . |
$$ | . . . O . X O . O O O . . X O , . . . |
$$ | . . O X X X O O O . . . . X O X O . . |
$$ | . . O . . X O . . . O b X . X O . . . |
$$ | . . . . . . . . . . . . . X . . . . . |
$$ ---------------------------------------[/go]

Who played the atari, who played the butt?

Winrate difference: 0.8%. Elf's play got 20k rollouts, the human's play got 125.

Elf played the butt. No ko this time.

Click Here To Show Diagram Code
[go]$$Bcm1 Mainline variation
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . O . . X X X . X O O . . . |
$$ | . . . X X O . X X O O O X X X O O O . |
$$ | . . X , O . O . O X . . . . O X X O . |
$$ | . . X O . . . . O O O X . . O . . X . |
$$ | . . X . 9 X O . . . . . . X . X X . . |
$$ | . . . . . 8 X . . . . . . . . . . . . |
$$ | . . 7 . . . . . . . . . . . . . . . . |
$$ | . X O X . . . . . . . . . . . . O . . |
$$ | . . X O O . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . X X X . . . . X O . . . |
$$ | . . O . O X X O X O X X . . . O . . . |
$$ | . . . O . X O . O O O . . X O , . . . |
$$ | . . O X X X O O O 6 3 5 . X O X O . . |
$$ | . . O . . X O . 4 . O 1 X . X O . . . |
$$ | . . . . . . . . . . 2 . . X . . . . . |
$$ ---------------------------------------[/go]

Black plays sente against White's bottom side group, and then takes the ponnuki on the left side.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 147 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 8  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group