daal wrote:
Polama wrote:
I'd posit there's two types of cheaters: people who just run the bot and destroy their opponent, who want to play the best move every move. And people who want to get away with cheating, who spot check for blunders or the occasional tricky move, but who are savvy enough to not blindly play clearly bot sequences.
I really don't think you'll catch the latter group with anything short of monitoring.
Isn't the whole idea of neural networks that AI can learn to recognize patterns that humans might not be aware of? Feed it a bunch of games where savvy players cheat and let it make the call. For a server like KGS, the combination of automated cheater detection and an automatic temporary ban would certainly serve as a deterrent without ruining anyone's reputation. It doesn't have to always be right, it just has to be the authority.
First, I'm not sure there is a subtle pattern here. There's just so much data the machine doesn't have. Which algorithm might the cheater be using? During which of there games were they hungry, or drinking, or are playing after a great night sleep? Which tsumego have they done lately? Which AI games have they reviewed that they might have picked up a move from? What were there goals for this game? I tend to open strong and then blunder away the middle game. But sometimes I set out to work on playing simply enough to not blunder bad. Those games look very different from my normal games. Sometimes I want to play an unfamiliar opening. Sometimes I miss a really obvious move, but 20 moves later find a clever way to escape my fate. Was that a blunder-recover pair, or professional level reading depth?
I think this is a common misconception with AI: if there's relevant data it doesn't have access to, that's a hard limit on what it can achieve. Add in the number of variations of AI networks the cheater could be using, and just the huge variation in styles and relative strengths among humans, plus the reality that everybody is after the same goal, and there's only so many plausible moves in most situations: I strongly suspect any signal from savvy cheating is just swamped by the rest of the variance in the world.
And even if there is a signal, subtle signals require lots of training data. AlphaGo Zero self-played ~5 million games. ImageNet (detecting images) is 14 million labels. On the one hand, binary "are they cheating" classification is an easier space than "choose every move in a Go game". On the other, I still think this is a very subtle thing that will require nuanced understanding of human styles. Maybe it's only 50,000 cheating players we need to record? Just 1% of learning to play? I'd still think that's low. I think it'd tend to learn subtle spurious correlations among the players who were tested. In particular, I'm sure it'd be very suspicious of anybody stronger than most of the 50,000 players we sampled. If you review too many games with bots, maybe you can't play anymore. But maybe I'm wrong and that's enough.
Of course, we'll need to resample those players every couple years as our understanding of Go improves...