“Decision: case of using computer assistance in League A”

[jlt]

@bernds:

The probability to play like Leela on moves 1,...,49 is p⁴⁹.
The probability to play differently from Leela on move 50 is 1-p.

So the probability to play like Leela on moves 1,...,49 and differently on move 50 is p⁴⁹(1-p).

In the same way, for each k, the probability to play differently from Leela on move k and like Leela on other moves is p⁴⁹(1-p).

Now, there are 50 possible choices for k (k=1,2,...,50), so multiply p⁴⁹(1-p) by 50 to get the probability to play exactly 49 moves "right" and 1 move "wrong".

[bernds]

@bernds:

The probability to play like Leela on moves 1,...,49 is p⁴⁹.
The probability to play differently from Leela on move 50 is 1-p.

So the probability to play like Leela on moves 1,...,49 and differently on move 50 is p⁴⁹(1-p).

In the same way, for each k, the probability to play differently from Leela on move k and like Leela on other moves is p⁴⁹(1-p).

Now, there are 50 possible choices for k (k=1,2,...,50), so multiply p⁴⁹(1-p) by 50 to get the probability to play exactly 49 moves "right" and 1 move "wrong".

Hmm. I have mostly escaped statistics in my education, and your explanation sounds plausible, but I still don't quite understand why we don't treat it as follows: if p is the probability that a player does not make an original move (0.89) and we assume independence, then the probability of the player not making an original move 50 times in a row is p⁵⁰ (which would come out to only 0.3%.

[dfan]

Hmm. I have mostly escaped statistics in my education, and your explanation sounds plausible, but I still don't quite understand why we don't treat it as follows: if p is the probability that a player does not make an original move (0.89) and we assume independence, then the probability of the player not making an original move 50 times in a row is p⁵⁰ (which would come out to only 0.3%.

You are calculating the probability of making any nonzero number of original moves (including, for example, 13 original moves and 37 unoriginal moves). jlt is calculating the probability of making exactly one original move.

[jlt]

@bernds: That's also true, and doesn't contradict my post.

The probability to make all 50 moves like Leela is pⁿ which is about 0.3 %.

The probability to make exactly 49 out of 50 moves like Leela is np^n-1(1-p) which is about 1.8 %.

So the probability to make at least 49 out of 50 moves like Leela is pⁿ+np^n-1(1-p) which is about 2.1 %.

In my calculation I only used the term np^n-1(1-p) because the other one, pⁿ, is much smaller and can be neglected, since our calculation is very rough anyway.

[mhlepore]

First, thanks to Bill for the extremely thoughtful post on Bayesianism. I enjoyed it very much.

Standard hypothesis testing (a la Neyman-Pearson) can lead to trouble when placing too much importance on p-values. A small p-value indicates the data you observe is unlikely to be generated when your null hypothesis it true. This leads people to reject the null hypothesis. However, even with a low p value, your null hypothesis may actually be the most likely of a bunch of unlikely hypotheses that could have generated that data.

Let's consider three hypotheses:

1) The alleged did indeed use the computer for assistance during his tournament game.
2) It is a complete random coincidence that most/all of the alleged's moves mirror the highest value moves from Leela.
3) A less nefarious middle ground between 1 and 2: the user trained himself on the software, and has acquired the ability to mimic successful play by the computer, even if he doesn't know why all of these moves are good.


1 seems possible, given that the game was played by people on a computer in remote locations.

2 doesn't seem likely, for all of the reasons people are describing with probability calculations.

3 could be possible. A nice way to test this would be to observed how the alleged plays over the board. If he can sustain the same proximity of play to Leela in an in-person tournament where he doesn't have access to a computer, then that would lend weight to this hypothesis.


Finally... to be falsely accused of something is a terrible thing. I would be in favor of voiding the game given the evidence, but not banishing the person from Go existence.

[hyperpape]

Another thing that occurred to me: while I'm not sure how much difference it makes, the probability of "this game contains a 100 move sequence with X similarity to Leela" is greater than "this particular 100 move sequence from the game have X similarity to Leela". A matching sequence could be found at 40-140, 50-150, 60-160, etc, and while those aren't independent, the chance that one of three will match is greater than the chance that a specific 100 move sequence will match.

I did not read anything that made me think the arbitrators went fishing, but it's something else that we need to be careful about, and a question that would be nice to have answered: how were moves 50-150 selected?

[mhlepore]

I did not read anything that made me think the arbitrators went fishing, but it's something else that we need to be careful about, and a question that would be nice to have answered: how were moves 50-150 selected?

While arbitrary, selecting this stretch of moves doesn't seem unreasonable.

The very beginnings and endings are likely to be replicatable by many players.

By move 50, you are into the middle game, where "book learning" stops, and you would not expect play to be the same between a human and a computer. (or between any two separate entities, really)

Approaching move 150 is likely macro-endgame. Here too you are likely to see differences between a human and a computer (which is likely to play sub-optimally if it increases its win
probability).

[pnprog]

in the last 2 years he didn't play online and played instead only against Leela also studing and reviewing with it.

I am not sure, but with this sort of training, won't the player learn to answer with "Leela's type moves" only to Leela herself? And when faced with an opponent with a different style than Leela, won't the similarity of his moves with Leela's move going to drop drastically?

[Gomoto]

@pnprog,

This is very similar playing exclusivly tygem for 2 years and then change the server or going to a real life tournament.

[Uberdude]

in the last 2 years he didn't play online and played instead only against Leela also studing and reviewing with it.

I am not sure, but with this sort of training, won't the player learn to answer with "Leela's type moves" only to Leela herself? And when faced with an opponent with a different style than Leela, won't the similarity of his moves with Leela's move going to drop drastically?

Seeing as I got 80% similarity in my last league game and I've never studied with Leela (and my opponent got 90%), it suggests a mid-dan playing a sensible game can easily get a high similarity. I wonder if the strongest players in league A have a lower similarity (Lukas Podpera 7d said on FB he is usually 70-80%) because they are stronger than Leela; also it would be interesting to check styles, e.g. I expect calm territory Artem is more similar to Leela than crazy fighter Ilya is. The game in question was played on 28th November 2017, Carlo played in the 2017 EGC in July and 3 Italian tournaments since. In the EGC he entered as 3d and won 10/15 across the main and weekend, gaining 50 GoR, a pretty good result but not as spectacular as you might expect from a 6d beater. The Italian tournaments suffer from being against mostly weaker players: I expect his opponents' Leela similarity would be lower as they play worse moves, and would his responses also be less similar or could he keep playing Leela honte style to answer their mistakes? There were just 2 of 15 >= 4d, he beat 4d, lost to 5d. So if you want to analyse his Leela similarity in offline tournaments within a few months of the Israel game against similar strength opponents then there's only 2 games to use. Not enough for quality stats.

[pnprog]

Let's pick the highest percentage, i.e. 89%. Suppose for simplicity that for each move, the probability to find Leela's move is p=0.89. Then for n=50 moves, the probability to find correctly exactly 49 moves is np^n-1(1-p)

How do you arrive at this?

For the record, I had made a small python script to estimate those probabilities, and the result is consistent with jlt method of calculation.

Here it is:

Code: Select all

from random import random

p=0.80 #frequency of similar move
tm=50 #number of moves to consider
m=49 #number of similar moves among the "tm" moves

s=0
nbp=0
while 1:
   nb=0
   for move in range(tm):
      x=random()
      if x<=p:
         nb+=1
   if nb>=m:
      nbp+=1
   s+=1
   
   if s%100000==0:
      print
      print nbp,"/",s
      print ("%.6f")%(float(nbp)/float(s)*100.)+"%"

[hyperpape]

While arbitrary, selecting this stretch of moves doesn't seem unreasonable.

I agree that it's a reasonable range. The question is: was it the only range they considered? Because if they considered more than one range, that somewhat raises the chances of a significant similarity being found.

[shoryuu]

For a kyu player to play moves that match 98% (or whatever the exact no. is) of a high dan bot that has beaten a pro already? That's way too unbelievable now isn't it?! Even pros have trouble matching move for move or even playing the same top few moves for that high a percentage of their moves.

Also, you guys might be mistaking very obvious real life situations that the judges and players there were able to witness, e.g. the accused kept leaving the table way too often. Because there's no way the guy would arouse suspicion had he just sat there the whole game.

[dfan]

For a kyu player to play moves that match 98% (or whatever the exact no. is) of a high dan bot that has beaten a pro already? That's way too unbelievable now isn't it?! Even pros have trouble matching move for move or even playing the same top few moves for that high a percentage of their moves.

The player is 4d EGF.

Also, you guys might be mistaking very obvious real life situations that the judges and players there were able to witness, e.g. the accused kept leaving the table way too often. Because there's no way the guy would arouse suspicion had he just sat there the whole game.

The game was played online.

[pnprog]

in the last 2 years he didn't play online and played instead only against Leela also studing and reviewing with it.

I am not sure, but with this sort of training, won't the player learn to answer with "Leela's type moves" only to Leela herself? And when faced with an opponent with a different style than Leela, won't the similarity of his moves with Leela's move going to drop drastically?

@pnprog,

This is very similar playing exclusivly tygem for 2 years and then change the server or going to a real life tournament.

No that much similar in fact, because when playing exclusively on tygem for 2 years, you won't be playing exclusively the same opponent like he did. At least you will play against various styles and various levels.