I am not that good at statistics, but if we consider that a player, in average, plays 70-80% of moves that are similar to Leela offline (let's take 80% moves), then what is the probability that he plays 49 or 50 similar moves out of 50 online?

In my simulation, the probability is below 0.02%

Of course, one have to check carefully that the measurement is consistent of both cases, and backed by enough data for the offline measure, and we make the assumption that offline performance can be translated in online performance, and that it is consistent across different opponents, and so on, and so on, and so on...

Beside, this is mildly related to the topic, but sometime ago, I stumbled upon this article https://www.chess.com/article/view/better-than-ratings-chess-com-s-new-caps-system#comments that is considering a replacement for ELO. In my understanding, this measure a similarity of play between chess players and chess bots as a measure of the player's strength. I was considering adding that sort of calculation into GoReviewPartner, but just for fun

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

First, let me say something about being a Bayesian. Before the 20th century, everybody was a Bayesian, but by then were aware of its problems. To be a Bayesian is to believe that hypotheses have, or may have, probabilities. Frequentists believe that only events have probabilities. Frequentists won the field in the early 20th century, as ideas such as hypothesis testing were developed without assigning probabilities to hypotheses. In the mid 20th century there were still a few famous Bayesians, such as L. J. Savage and I. J. Good. I was fortunate enough to spend an afternoon with Savage, who showed me that I was a closet Bayesian. Later in the 20th century Bayesianism had a rebirth, perhaps because of computer programmers who were able to write programs that made sense in Bayesian terms, and those programs worked.

Bayesiansism when I came across it, in the mid 20th century, was largely subjectivist, because how you alter your probabilities in the face of evidence depends upon your prior beliefs. This subjective property, OC, is a major problem from a scientific point of view. I. J. Good talked about how to interrogate yourself to find out what your probabilistic beliefs were. The question you ask is a good example. What would be convincing statistical evidence in itself? How about aceing all four par threes on the course?



IIRC, Uberdude observed 89% in his opponent, who was of comparable strength but had not undergone Carlos's training regime. Anyway, as a Bayesian, you have to have a non-zero belief in any probability that you may possibly come to assign to a hypothesis, or you will never assign that probability. IOW, your beliefs have a distribution over a range of probabilities. So you can't just focus on, say, 75% or 80% in itself.

I know in a way I am dodging the question, but let me say that first, I would want to know more about the games and what kind of moves he made which were like Leela's and what kind were not, and so on. Second, like just about everybody else, I think that there is enough evidence to think that in this one game he played unusually like Leela. But there could be any number of reasons for that. Cheating is not the only hypothesis to consider. It may be the one with the highest prior degree of belief, and the one that is most bolstered by the evidence, but that still may not give it a high posterior degree of belief. It may be the most supported hypothesis, but that is a different question.

Another question that Baysians must ask is how do we know what we know? An example from the early scientific literature has to do with the probability that the sun will rise tomorrow. Suppose we start by assigning equal weight to each non-zero probability that the sun would rise the next day, starting from day 1, which in those days was supposed to be some 6,000 years in the past. As we update our beliefs with the sun rising every day for 6,000 years, the probability that it would rise tomorrow becomes very, very, close to 1. (Yay!) But someone, I forget who and where, published a paper pointing out that on the same evidence the probability that the sun would rise 5,000 years from today was only ⅔. That result was not appealing. That result illustrates the problematic nature of Bayesianism for science. Speaking for myself, my belief about the sun rising has to do with the rotation of the earth. Given that knowledge, whether the sun rose yesterday is irrelevant to whether it will rise tomorrow.

In the parable of the golfers Regan states that the chance of a scratch golfer making a hole in one on a par three is about 1/5000. If we gathered 10,000 such golfers and had them each hit a ball from the tee on a par three hole, we would expect 2 of them to make a hole in one. He goes on to say that suppose that we slipped a piece of paper with a black dot on it into the pocket of 10 of those 10,000, and one of them made a hole in one. The probability that (at least) one of them would do so is around 1/500, i.e., (1 - (4999/5000)^10). The statistical evidence is very strong that this is not just a chance event, that there is a reason for it. Good enough, Regan suggests, to win a civil case in court.

Something that he does not go on to say, but I think that he should have said, is that once we know about the black dots, we should chalk up that hole in one to chance. Having a black dot in your pocket is irrelevant to your golfing ability. But in the parable he has the black dot stand for "physical or observational evidence of cheating". I.e., given such evidence, then the hole in one (statistically very rare occurrence) may offer support. In this case, all we have is the statistical evidence. IMO, this evidence supports looking for physical or observational evidence of cheating. For example, by monitoring Carlos's play in the future.

Frequentists agree, BTW. I still remember the prof announcing, "Statistics proves nothing."

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Humans can learn from programs and apply what they have learnt. The result can be very similar games. Therefore, probability is not evidence. What we do need in important online games is referees locally supervising the players to prevent almost all cheating of all forms.

The game in question has 166 moves, the allegation is presumably that after move 4 black starts using Leela. In other words he turned on Leela after opting for a double 3-3 fuseki. After that every single move chosen is a 'top' Leela move. The game doesn't look that remarkable to me. The mathematics behind the decision has, especially in the case of an appeal, to be the subject of some debate if online games are still to be taken into account.

Nobody has answered my question yet though, is there any appeal of the decision?

_________________
North Lecale

On the issue of the (mis)use of statistics in (in)justice:
- https://en.wikipedia.org/wiki/Roy_Meadow
- https://en.wikipedia.org/wiki/Lucia_de_Berk
- https://en.wikipedia.org/wiki/Birmingham_Six (many other problems here, but part of the evidence was an expert witness saying he was 99% sure a chemical test proved the accused had been handling explosives, but it turned out handling playing cards could produce a false positive (from their similar to nitroglycerine coating), and they had indeed been playing cards)

I wasn't aware that the Birmingham Six was an example of mis-use of statistics; wasn't it mostly the simple fabrication of evidence? Mind you the TOS probably forbids me discussing that here, so here is the game from November instead.

]

_________________
North Lecale

Is there a reason why the other games played in this year's league did not form part of the investigation? They would presumably provide useful data for determining whether the game against Israel was out of the ordinary for this player. Or whether all online games show a different pattern to all offline games etc.

Some people have here mentioned 98% out of 100 moves, but I suppose it is actually 50 moves of the player, within moves 50-150 of the game.

I think it is very important why exactly moves 50-150 were studied. Was this decided beforehand? Why are not numbers given for the whole game?

This guy Metta is the chief referee of the coming EGF Congress in Pisa. What happens then, will the EGF allow him to referee?

That's a good point.

This is what they did


Leela move 4 - does that correspond to move 7 of the game, or does it mean move 57? I guess we will see a technical report soon to clear up this confusion. In any EGF event you have 3 stages of appeal. #1.Appeal to the Referee, #2.Appeal to the Tournament Appeals Committee, #3.Appeal to the EGF version of #2. Given that this happened in November and has only become known now, which steps were taken?

_________________
North Lecale

The game almost only has ordinary moves. Already this lets it be possible that two different players / programs find most of the same moves, especially if they use a similar playing style. That the player is said to have studied with the program for 2 years makes this all the more likely that same moves are not coincidence and not cheating but a direct consequence of adapting a playing style and "knowledge" / "experience" from training with the program during the years before the game. I do hope very sincerely that the judgement is overturned by higher instances.

Leela move 4 means the 4th best move in that position according to Leela's analysis.

Of the 50 moves considered, 49 were in the top 3 best moves according to Leela's analysis (with the additional constraint that the move should not be more than 5% worse than Leela's top choice).

This is where the 98% number comes from, it is 49/50.

The only move out of 50 not in Leela's top 3 was Leela's 4th choice in that position, and was less than 1% worse than Leela's top choice.

Ah, you are right, that seems like the best way to interpret that statement. So did they then only look at a section of the game or at the whole game? For me it would be kind of strange if they didn't look at the whole game. I suppose that a script already exists to show the comparison data per ply for the game.

_________________
North Lecale

Let's pick the highest percentage, i.e. 89%. Suppose for simplicity that for each move, the probability to find Leela's move is p=0.89. Then for n=50 moves, the probability to find correctly exactly 49 moves is np^n-1(1-p) which is about 2%. During rounds 1--3 of the Pandanet EGC, 60 games were played, so you would expect at least one false positive.

jlt, a good point to illustrate, but even that is placing too much significance on this test. That 2% is the chance of a randomly chosen game being at 98% Leela (and you should include 100% too). But when the game you choose to investigate is selected because someone else noticed it was similar to Leela it's like putting the black spot in the golf analogy on the player who hit the hole-in-one after and because he did so. It's not independent so the simple probabilities are not appropriate.

Finally I saw a confirmation, an appeal is planned, so the decision is not final. Judging by the last appeal I saw, it could take a year to figure this out.

Russia

CzechRepublic

Romania

Hungary

Serbia

Ukraine

_________________
North Lecale

This calculation was why I was trying to get something more 'common sense' from Bill.
If you take 89% (Uberdude's 4d opponent), you get 1.8% or about one time in 54. However, if you plug in 80% (upper figure for what was observed for the player under discussion) you get a very different result = [edit --> wrong! 0.0000065% or one time in 15 million] 0.018% or one time in about 5,600. And if you take 70% (the lower figure), you get [edit --> wrong!! 5.0E-16 or about one time in 2 quadrillion] 3.8E-7 or about one time in 2.5 million.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

No you don't. If p=0.8 then np^n-1(1-p) is about 1/5600. If p=0.7 then np^n-1(1-p) is about 1/(2.5 million).

The orders of magnitude are certainly very different, but I purposely picked p=0.89 to be on the safe side, i.e. I would prefer a few cheaters to go unpunished rather than too many punished innocents (together with the whole team).

And also, p=0.89 may not be unrealistic for that particular game if you assume that most moves were very "ordinary" (dixit Robert Jasiek).

IMO, for the appeal, they should analyse a large sample of PGETC games to see how much of an outlier 98% is.