Will pay for real ai programmer/dev analysis for cheating

[weiqiasia]

Hi, our official state association has monthly tournament, this is mainly for selecting players represent our state to compete in national competition.

There's a beginner beat all the high dans and a few of the players suspected he cheated using ai. we need a solid proof before disqualifying him.

We are hiring ai developer to analyze his games, about 7 sgf

Please contact me for more info, and let me know how much you charge.
Will pay with Paypal.

[gennan]

It sounds to me that you already know that the player cheated in these 7 games. So I don't really understand what you expect from this developer. Are you looking for someone to come up with some numbers to support your suspicion?
As you are even paying for this, there is a very high risk of confirmation bias, I would say.

[jlt]

You may want to contact Antti Törmänen 1p, he has some experience with cheating detection and if I am not mistaken, he has a program based on this paper:

https://arxiv.org/pdf/2009.01606.pdf

[gennan]

It is not only important to successfully detect cheating (true positives), but is also important (perhaps even more) that the detection method gives very few false positives (leading to unjustful disqualifications and public discrediting of innocent players). You want to avoid things like the Prosecutor's Fallacy and Confirmation Bias.

I think what's actually needed for proper automatic cheating detection, is this procedure (which is quite common in machine learning and software classifier competitions):

[1] prepare a large (the larger, the better) collection of games where you know for certain whether there was cheating or not, because it includes games of various volunteers who cheated on purpose in various ways that they saw fit. This collection needs to have games from all levels of play and many different playing styles.
[2] with this annotated collection of games, developers can create and test classifiers (by machine learning or some other method).
[3] you can objectively compare the quality of various classifiers by (for example) their Matthews correlation coefficient.
[4] you could even create a competition between classifiers from different developers by using a (perhaps undisclosed) representative subcollection of the annotated games that were not used in the creation and testing of those classifiers.

I think that step [1] will be a lot of work, requiring a coordinated effort to create a high quality dataset to use in the next steps. Also, I think step [1] is typically a task for an organisation and not a task for the developers of classifiers.

But once you have this data set, a public competition with prize money could be quite a cheap way to get a very good classifier (see for example the CASP14 protein folding competition that was won by Deepmind's AlphaFold in 2020).

[RobertJasiek]

"we need a solid proof"

Then do not rely on programs' analysis to infer guilt. You might use programs to infer likely innocence. If programs do not suggest likely innocence, then establish evidence on objective facts, such as eye-witnesses.

If you have already held a tournament for which objective facts cannot be established, then the mistake is one of the organisers, which should have held the tournament in a manner that objective facts can be established.

Compromise solution: Suppose players in the tournament are expected to play at least amateur X dan real world level. Let the alleged cheater prove his innocence in the tournament game(s) by letting him play test games against (X-3) dans (3 ranks weaker dans). If he wins more than 50% of the test games played so that objective facts are established, assume his innocence in the earlier tournament game.

[Javaness2]

and then the sun rose

[John Fairbairn]

and then the sun rose

And then, Ian, we could all see the tournament organisers floating up the Lagan in bubbles?

[Bill Spight]

We are hiring ai developer to analyze his games, about 7 sgf

Pardon me, but why an AI developer? (A rhetorical question.) A forensic statistician has a more relevant skill set. To catch Jack the Ripper you don't ask someone who makes knives. Although they may have relevant information.

[CDavis7M]

"we need a solid proof"

Then do not rely on programs' analysis to infer guilt. You might use programs to infer likely innocence. If programs do not suggest likely innocence, then establish evidence on objective facts, such as eye-witnesses.

If "solid proof" is needed then surely eye-witnesses are out -- too unreliable.

----------

My suggestion would be to hold a supervised playoff match between the suspected cheater and the runner up.

[Javaness2]

and then the sun rose

And then, Ian, we could all see the tournament organisers floating up the Lagan in bubbles?

That phrase was rather quaint when I was a child, what wonders TV holds, but rest assured that if you lived near Slieve Croob you really wouldn't want to take that trip.

[jlt]

P.S. A few months ago I read an email which suggests that the organizers of this competition https://m.facebook.com/2020WYAWT/about

have a cheating detection program, so you may want to ask them.

[RobertJasiek]

If "solid proof" is needed then surely eye-witnesses are out -- too unreliable.

It depends on what they saw, how they report it and how the court judges that.

An eye-witness contradicting himself in central aspects cannot be trusted.

Several eye-withnesses reporting the same independently from each other can be trusted.

[Bill Spight]

Here is a post on statistician Andrew Gelman's site about an accusation of cheating on chess.com.

https://statmodeling.stat.columbia.edu/ ... meone-off/

And here are a couple of links to the accused cheater's site.

https://lostontime.blogspot.com/2019/10 ... ed_67.html

https://lostontime.blogspot.com/2019/10 ... ed_89.html

[John Fairbairn]

Thanks, Bill.

One of those links contained a point I think has been generally overlooked, and I certainly totally overlooked it.

I gather from the complaint, though, that it’s the online chess equivalent of having one of your publications publicly flagged by an algorithm as reporting fake data.

I have forgotten the details, and even some of the bigger points, as it was many years ago and I wasn't directly involved, but as I recall there was a case of a British organisation being affected with a denial of service by a flood of e-mails by Stop the War protesters. The response was to treat 'stop the war' as a junk phrase at server level and so e-mails containing it were stopped getting through. That caused major problems as it blocked all sorts of legitimate and valuable e-mails from other people such as politicians (e.g. "should we actually stop the war?").

In that case, both sides forgot the maxim that actions have consequences, and further forgot that in most cases the consequences are the ones you least expect.

Seems like chess and go are following the same dangerous path. The aspect that would worry me most is that the anti-cheaters are unwittingly creating an atmosphere that leads to their beloved game being regarded as a vice. It's happened before. When I was young, there were many parents who saw cards as the devil's playthings and so banned their children from playing bridge or whist.

[RobertJasiek]

I do not think that with "anti-cheaters" you mean everybody not cheating. You also do not even mean all organisers, referees, server admins investigating. Apparently, you only mean those officials being rash with identifying alleged cheating.