It is currently Wed Apr 25, 2018 10:43 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 191 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 10  Next
Author Message
Online
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #41 Posted: Mon Mar 26, 2018 5:54 am 
Lives with ko

Posts: 198
Liked others: 20
Was liked: 132
Rank: maybe 2d
jlt wrote:
ez4u wrote:
If you take 89% (Uberdude's 4d opponent), you get 1.8% or about one time in 54. However, if you plug in 80% (upper figure for what was observed for the player under discussion) you get a very different result = 0.0000065% or one time in 15 million. And if you take 70% (the lower figure), you get 5.0E-16 or about one time in 2 quadrillion.


No you don't. If p=0.8 then npn-1(1-p) is about 1/5600. If p=0.7 then npn-1(1-p) is about 1/(2.5 million).


There's a reasonable chance the actual probability is nearer the larger side of these estimates. Because if there is sometimes any correlation at all between a match on one move to a match on subsequent moves, then the moves within a game are not independent, and we should expect comparable to the larger side. (e.g. if there are kinds of games/fights/sequences where each move is 85% and kinds where each move is 75%, and each kind of game is equally likely, then the overall probability for 98% is much higher than if every move in every game was uniformly 80%).

HermanHiddema wrote:
IMO, for the appeal, they should analyse a large sample of PGETC games to see how much of an outlier 98% is.


This.


This post by lightvector was liked by 2 people: hyperpape, wurfmaul2
Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #42 Posted: Mon Mar 26, 2018 5:58 am 
Tengen

Posts: 4643
Location: Cambridge, UK
Liked others: 208
Was liked: 2307
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
HermanHiddema wrote:
IMO, for the appeal, they should analyse a large sample of PGETC games to see how much of an outlier 98% is.


Even if that shows 98% is a significant enough outlier that's not enough to convict IMO (of course if it's not an outlier then that's easy, case closed, less work, unless everyone is using Leela!). To do this well you need to analyse a decent number of games of people who have studied a lot with Leela and see how much of an outlier 98% is to that. It may be hard to find enough such people/games around 4d level. Carlo's offline games are a good start (if recent enough), but I want to know how many were analysed, not just a plural and result is 70-80%.

Update: Rather than going for a walk in the park at lunch I finished analysing moves 50-149 of my PGETC game :) . I scored 80% similarity and my opp* scored 90%. So we have the report at least 2 of Carlo's offline games are in the 70-80% range, and my data of [80,90]. 98 is looking less conclusive. I attach the spreadsheet I used for interest/verification. I note the sequence from move 93 to 136 would have all counted as copies under this metric were it not for 2 timesujis I played in the middle (most people don't get into byo yomi as earlier as I do). Next I might look at the stricter "top 1" instead of "top 3" metric.

* for ez4u, now ranked 3d, but a former 5d.


Attachments:
Leela.xlsx [11.61 KiB]
Downloaded 15 times
Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #43 Posted: Mon Mar 26, 2018 6:09 am 
Gosei
User avatar

Posts: 1985
Location: Tokyo, Japan
Liked others: 1763
Was liked: 1101
Rank: Jp 6 dan
KGS: ez4u
jlt wrote:
ez4u wrote:
If you take 89% (Uberdude's 4d opponent), you get 1.8% or about one time in 54. However, if you plug in 80% (upper figure for what was observed for the player under discussion) you get a very different result = 0.0000065% or one time in 15 million. And if you take 70% (the lower figure), you get 5.0E-16 or about one time in 2 quadrillion.


No you don't. If p=0.8 then npn-1(1-p) is about 1/5600. If p=0.7 then npn-1(1-p) is about 1/(2.5 million).

The orders of magnitude are certainly very different, but I purposely picked p=0.89 to be on the safe side, i.e. I would prefer a few cheaters to go unpunished rather than too many punished innocents (together with the whole team).

And also, p=0.89 may not be unrealistic for that particular game if you assume that most moves were very "ordinary" (dixit Robert Jasiek).

You are right, a spreadsheet error on my part! :sad:

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #44 Posted: Mon Mar 26, 2018 6:14 am 
Beginner

Posts: 5
Liked others: 0
Was liked: 0
Rank: EGF 3d
KGS: AhSayuni
I wonder if we'll get a comment from Metta himself anytime soon...

It's quite a complicated issue...

I really wonder why was only this game from the PGETC checked.
It seems a bit suspicious, considering it was perhaps the least impressive of his wins (he lost only to Csaba Mero and won against several 6d's and 5d's). Perhaps the reason is that it was the only game that someone appealed against (surprisingly, as it didn't change the outcome of the match, unlike other cases).

From Carlo's side, assuming the accusation is correct, I can understand why someone might choose to cheat when playing in a team. He's not playing just for his own sake and he knows that winning can have a great effect on his friends.. Staying or going down a league affects the whole next year.
(Still, no doubt, it's the wrong thing to do)...

Somehow I feel no problem with him being a referee in the EGC.
He'll be doing service to the EGF, and he'll be more obliged to prove himself...

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #45 Posted: Mon Mar 26, 2018 6:15 am 
Dies in gote

Posts: 67
Liked others: 13
Was liked: 10
Rank: 2d
jlt wrote:
Let's pick the highest percentage, i.e. 89%. Suppose for simplicity that for each move, the probability to find Leela's move is p=0.89. Then for n=50 moves, the probability to find correctly exactly 49 moves is npn-1(1-p)
How do you arrive at this?

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #46 Posted: Mon Mar 26, 2018 6:24 am 
Beginner

Posts: 8
Liked others: 1
Was liked: 2
@bernds:

The probability to play like Leela on moves 1,...,49 is p49.
The probability to play differently from Leela on move 50 is 1-p.

So the probability to play like Leela on moves 1,...,49 and differently on move 50 is p49(1-p).

In the same way, for each k, the probability to play differently from Leela on move k and like Leela on other moves is p49(1-p).

Now, there are 50 possible choices for k (k=1,2,...,50), so multiply p49(1-p) by 50 to get the probability to play exactly 49 moves "right" and 1 move "wrong".


This post by jlt was liked by: pnprog
Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #47 Posted: Mon Mar 26, 2018 6:49 am 
Dies in gote

Posts: 67
Liked others: 13
Was liked: 10
Rank: 2d
jlt wrote:
@bernds:

The probability to play like Leela on moves 1,...,49 is p49.
The probability to play differently from Leela on move 50 is 1-p.

So the probability to play like Leela on moves 1,...,49 and differently on move 50 is p49(1-p).

In the same way, for each k, the probability to play differently from Leela on move k and like Leela on other moves is p49(1-p).

Now, there are 50 possible choices for k (k=1,2,...,50), so multiply p49(1-p) by 50 to get the probability to play exactly 49 moves "right" and 1 move "wrong".

Hmm. I have mostly escaped statistics in my education, and your explanation sounds plausible, but I still don't quite understand why we don't treat it as follows: if p is the probability that a player does not make an original move (0.89) and we assume independence, then the probability of the player not making an original move 50 times in a row is p50 (which would come out to only 0.3%.

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #48 Posted: Mon Mar 26, 2018 6:55 am 
Lives in sente

Posts: 1009
Liked others: 447
Was liked: 250
Rank: AGA 4k KGS 4k
GD Posts: 61
KGS: dfan
bernds wrote:
Hmm. I have mostly escaped statistics in my education, and your explanation sounds plausible, but I still don't quite understand why we don't treat it as follows: if p is the probability that a player does not make an original move (0.89) and we assume independence, then the probability of the player not making an original move 50 times in a row is p50 (which would come out to only 0.3%.

You are calculating the probability of making any nonzero number of original moves (including, for example, 13 original moves and 37 unoriginal moves). jlt is calculating the probability of making exactly one original move.

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #49 Posted: Mon Mar 26, 2018 7:02 am 
Beginner

Posts: 8
Liked others: 1
Was liked: 2
@bernds: That's also true, and doesn't contradict my post.

The probability to make all 50 moves like Leela is pn which is about 0.3 %.

The probability to make exactly 49 out of 50 moves like Leela is npn-1(1-p) which is about 1.8 %.

So the probability to make at least 49 out of 50 moves like Leela is pn+npn-1(1-p) which is about 2.1 %.

In my calculation I only used the term npn-1(1-p) because the other one, pn, is much smaller and can be neglected, since our calculation is very rough anyway.


Last edited by jlt on Mon Mar 26, 2018 7:03 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #50 Posted: Mon Mar 26, 2018 7:03 am 
Lives with ko

Posts: 156
Liked others: 32
Was liked: 51
KGS: lepore
First, thanks to Bill for the extremely thoughtful post on Bayesianism. I enjoyed it very much.

Standard hypothesis testing (a la Neyman-Pearson) can lead to trouble when placing too much importance on p-values. A small p-value indicates the data you observe is unlikely to be generated when your null hypothesis it true. This leads people to reject the null hypothesis. However, even with a low p value, your null hypothesis may actually be the most likely of a bunch of unlikely hypotheses that could have generated that data.

Let's consider three hypotheses:

1) The alleged did indeed use the computer for assistance during his tournament game.
2) It is a complete random coincidence that most/all of the alleged's moves mirror the highest value moves from Leela.
3) A less nefarious middle ground between 1 and 2: the user trained himself on the software, and has acquired the ability to mimic successful play by the computer, even if he doesn't know why all of these moves are good.


1 seems possible, given that the game was played by people on a computer in remote locations.

2 doesn't seem likely, for all of the reasons people are describing with probability calculations.

3 could be possible. A nice way to test this would be to observed how the alleged plays over the board. If he can sustain the same proximity of play to Leela in an in-person tournament where he doesn't have access to a computer, then that would lend weight to this hypothesis.


Finally... to be falsely accused of something is a terrible thing. I would be in favor of voiding the game given the evidence, but not banishing the person from Go existence.

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #51 Posted: Mon Mar 26, 2018 7:40 am 
Tengen

Posts: 4102
Location: North Carolina
Liked others: 419
Was liked: 672
Rank: AGA 3k
GD Posts: 65
OGS: Hyperpape 4k
Another thing that occurred to me: while I'm not sure how much difference it makes, the probability of "this game contains a 100 move sequence with X similarity to Leela" is greater than "this particular 100 move sequence from the game have X similarity to Leela". A matching sequence could be found at 40-140, 50-150, 60-160, etc, and while those aren't independent, the chance that one of three will match is greater than the chance that a specific 100 move sequence will match.

I did not read anything that made me think the arbitrators went fishing, but it's something else that we need to be careful about, and a question that would be nice to have answered: how were moves 50-150 selected?

_________________
Occupy Babel!

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #52 Posted: Mon Mar 26, 2018 7:52 am 
Lives with ko

Posts: 156
Liked others: 32
Was liked: 51
KGS: lepore
hyperpape wrote:
I did not read anything that made me think the arbitrators went fishing, but it's something else that we need to be careful about, and a question that would be nice to have answered: how were moves 50-150 selected?


While arbitrary, selecting this stretch of moves doesn't seem unreasonable.

The very beginnings and endings are likely to be replicatable by many players.

By move 50, you are into the middle game, where "book learning" stops, and you would not expect play to be the same between a human and a computer. (or between any two separate entities, really)

Approaching move 150 is likely macro-endgame. Here too you are likely to see differences between a human and a computer (which is likely to play sub-optimally if it increases its win
probability).

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #53 Posted: Mon Mar 26, 2018 8:29 am 
Lives with ko

Posts: 158
Liked others: 20
Was liked: 69
Rank: OGS 7 kyu
Alessandro Boh Pace wrote:
in the last 2 years he didn't play online and played instead only against Leela also studing and reviewing with it.

I am not sure, but with this sort of training, won't the player learn to answer with "Leela's type moves" only to Leela herself? And when faced with an opponent with a different style than Leela, won't the similarity of his moves with Leela's move going to drop drastically?

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!


This post by pnprog was liked by: Gomoto
Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #54 Posted: Mon Mar 26, 2018 8:37 am 
Lives in gote
User avatar

Posts: 420
Location: Earth
Liked others: 135
Was liked: 62
@pnprog,

This is very similar playing exclusivly tygem for 2 years and then change the server or going to a real life tournament. :shock:


This post by Gomoto was liked by: pnprog
Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #55 Posted: Mon Mar 26, 2018 8:47 am 
Tengen

Posts: 4643
Location: Cambridge, UK
Liked others: 208
Was liked: 2307
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
pnprog wrote:
Alessandro Boh Pace wrote:
in the last 2 years he didn't play online and played instead only against Leela also studing and reviewing with it.

I am not sure, but with this sort of training, won't the player learn to answer with "Leela's type moves" only to Leela herself? And when faced with an opponent with a different style than Leela, won't the similarity of his moves with Leela's move going to drop drastically?

Seeing as I got 80% similarity in my last league game and I've never studied with Leela (and my opponent got 90%), it suggests a mid-dan playing a sensible game can easily get a high similarity. I wonder if the strongest players in league A have a lower similarity (Lukas Podpera 7d said on FB he is usually 70-80%) because they are stronger than Leela; also it would be interesting to check styles, e.g. I expect calm territory Artem is more similar to Leela than crazy fighter Ilya is. The game in question was played on 28th November 2017, Carlo played in the 2017 EGC in July and 3 Italian tournaments since. In the EGC he entered as 3d and won 10/15 across the main and weekend, gaining 50 GoR, a pretty good result but not as spectacular as you might expect from a 6d beater. The Italian tournaments suffer from being against mostly weaker players: I expect his opponents' Leela similarity would be lower as they play worse moves, and would his responses also be less similar or could he keep playing Leela honte style to answer their mistakes? There were just 2 of 15 >= 4d, he beat 4d, lost to 5d. So if you want to analyse his Leela similarity in offline tournaments within a few months of the Israel game against similar strength opponents then there's only 2 games to use. Not enough for quality stats.


This post by Uberdude was liked by: pnprog
Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #56 Posted: Mon Mar 26, 2018 9:13 am 
Lives with ko

Posts: 158
Liked others: 20
Was liked: 69
Rank: OGS 7 kyu
bernds wrote:
jlt wrote:
Let's pick the highest percentage, i.e. 89%. Suppose for simplicity that for each move, the probability to find Leela's move is p=0.89. Then for n=50 moves, the probability to find correctly exactly 49 moves is npn-1(1-p)
How do you arrive at this?

For the record, I had made a small python script to estimate those probabilities, and the result is consistent with jlt method of calculation.

Here it is:
Code:
from random import random

p=0.80 #frequency of similar move
tm=50 #number of moves to consider
m=49 #number of similar moves among the "tm" moves

s=0
nbp=0
while 1:
   nb=0
   for move in range(tm):
      x=random()
      if x<=p:
         nb+=1
   if nb>=m:
      nbp+=1
   s+=1
   
   if s%100000==0:
      print
      print nbp,"/",s
      print ("%.6f")%(float(nbp)/float(s)*100.)+"%"

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #57 Posted: Mon Mar 26, 2018 12:00 pm 
Tengen

Posts: 4102
Location: North Carolina
Liked others: 419
Was liked: 672
Rank: AGA 3k
GD Posts: 65
OGS: Hyperpape 4k
mhlepore wrote:
While arbitrary, selecting this stretch of moves doesn't seem unreasonable.
I agree that it's a reasonable range. The question is: was it the only range they considered? Because if they considered more than one range, that somewhat raises the chances of a significant similarity being found.

_________________
Occupy Babel!

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #58 Posted: Mon Mar 26, 2018 3:56 pm 
Dies in gote

Posts: 46
Liked others: 6
Was liked: 1
Rank: Over 8000K
For a kyu player to play moves that match 98% (or whatever the exact no. is) of a high dan bot that has beaten a pro already? That's way too unbelievable now isn't it?! Even pros have trouble matching move for move or even playing the same top few moves for that high a percentage of their moves.

Also, you guys might be mistaking very obvious real life situations that the judges and players there were able to witness, e.g. the accused kept leaving the table way too often. Because there's no way the guy would arouse suspicion had he just sat there the whole game.

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #59 Posted: Mon Mar 26, 2018 4:30 pm 
Lives in sente

Posts: 1009
Liked others: 447
Was liked: 250
Rank: AGA 4k KGS 4k
GD Posts: 61
KGS: dfan
shoryuu wrote:
For a kyu player to play moves that match 98% (or whatever the exact no. is) of a high dan bot that has beaten a pro already? That's way too unbelievable now isn't it?! Even pros have trouble matching move for move or even playing the same top few moves for that high a percentage of their moves.

The player is 4d EGF.

Quote:
Also, you guys might be mistaking very obvious real life situations that the judges and players there were able to witness, e.g. the accused kept leaving the table way too often. Because there's no way the guy would arouse suspicion had he just sat there the whole game.

The game was played online.

Top
 Profile  
 
Offline
 Post subject: Re: “Decision: case of using computer assistance in League A
Post #60 Posted: Mon Mar 26, 2018 7:59 pm 
Lives with ko

Posts: 158
Liked others: 20
Was liked: 69
Rank: OGS 7 kyu
Gomoto wrote:
pnprog wrote:
Alessandro Boh Pace wrote:
in the last 2 years he didn't play online and played instead only against Leela also studing and reviewing with it.

I am not sure, but with this sort of training, won't the player learn to answer with "Leela's type moves" only to Leela herself? And when faced with an opponent with a different style than Leela, won't the similarity of his moves with Leela's move going to drop drastically?

@pnprog,

This is very similar playing exclusivly tygem for 2 years and then change the server or going to a real life tournament. :shock:

No that much similar in fact, because when playing exclusively on tygem for 2 years, you won't be playing exclusively the same opponent like he did. At least you will play against various styles and various levels.

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 191 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 10  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Majestic-12 [Bot] and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group