Life In 19x19

Knotwilg wrote:Today you will see more "confirmation bias" when comparing pro moves with AI. Older games will probably show a bigger deviation hence will contain more cases where the pro was right and the AI turns out to be wrong. Maybe AI should go back to study the classics one day ...

Something that I have been looking at lately is the revision of the bot's evaluation of a human's play that was off the bot's radar or got relatively few rollouts. In reviewing pro games I was surprised how often the bot, given a fairer comparison, gave a higher winrate to the pro's choice over its own. The difference was typically less than 2%, but sometimes substantial. We are still talking about fewer than 10% of the plays, but it's not a rare occurrence.

I'm not sure I agree with the framing of this example. If you told me an airplane had a 99% chance of not crashing, I would not get on that plane. But if you told me 99% of bots say this is the best move, I am going to believe them.

Well, I'd probably say the same, but at the same time I'd say the framing here is suspect: apples and pears. I wouldn't get on the aeroplane because crashing would kill me. But I wouldn't care about the percentages on best move because it's just a game and has no effect on me. The bot 99% part also waves a red flag for me - the result of reading books like Freakonomics. I can't say whether it's suspect, but I wouldn't be surprised if a guy like Levitt shot it down.

But more important to me is the relative evaluation percentages. When 80% of the AIs prefer Move A to Move B, do they think Move B is fatally bad, or is it just a tenth of a percentage point worse? Learning to avoid huge mistakes is where AI shines, in my opinion.

I agree this is valuable (especially with Bill's various cautions), and I think it's how pros are mainly trying to use it. But, just in passing, I think a lot of pro plays that are being counted as pros having learnt something from AI may stem from a quite different impulse: experimentation. When Shin Fuseki became all the rage, very many pros tried it out simply to get a handle on it. We know this because many of them contributed essays, letters and whatnot to go magazines explaining what they were doing and (eventually) explaining why they did or did not agree with Shin Fuseki principles (I think the fairest simple summary, which might apply today with AI, is "Too hard"). That was an era of pros working hard to get in touch with fans. We are not seeing the same level of openness now with AI, in an age when fans are more or less taken for granted by pros. But there are some strong indications of experimenting (internet accounts among other things) and I'd be rather surprised if experimentation didn't actually account for most of the examples we see.

Bill Spight wrote:As Knotwilg says, the global setup matters.

My preliminary investigation.

Using Waltheri I looked for real world examples. To reduce the effect of the rest of the board I chose only those before move 20 and looked them up in the Elf commentaries. Of the 10 results so far (and I may not do any more. ):

In 9 the ladder was Elf's top choice;
In 1 the hane at d was Elf's top choice;
In 0 was c on Elf's radar.

----
Discussion:

In all 10 the ladder was bolstered by a stone in the diagonally opposite corner. IOW, the pro did not choose this variation so early in the game without that support for the ladder. If the diagonally opposite corner were empty, for instance, the opponent would have a ladder breaker just by occupying it. So even for humans I guess we have to consider this position to be a situational play.

That said, in no case was the 5-5 connection preferred by Elf over the 5-3 connection. So the ladder seems to be a poor choice in general, and to be avoided, anyway. This may become a Noseki.

10 examples seemed a poor sample size to me and I have a tool for this.
I analysed 91 positions at 30k playouts with elf v2 (which is terrible at ladders):
Re: trust - I don't have very much confidence at all that bots or pros are giving us 'correct' moves for more ambiguous opening positions but I'm not particularly concerned. I think in go we all spend a lot of time learning things that aren't quite correct but if we're lucky they are correct enough to take us a little bit further.
I agree that bots are limited but can show us ideas to explore and develop our own individual understandings of. They're also great for spotting blunders!

Knotwilg wrote:Bizarre. Repeating the exercise shows a different result. Katago, like conventional wisdom, wants to play the ladder.
...Well, it's been a good exercise, even if conventional wisdom was finally confirmed by AI. I can't reconstruct how I came to a different insight in the first place.

I would like to borrow the position below (i.e. just before Knotwilg's analysis) to show that "conventional wisdom" and "AI" don't necessarily belong together (or maybe even on the same planet).

I happened to notice along the way to the original position that katago (in my case the 20-block net) does not necessarily want to cut at "a" below. Serious attention is given to the hane at "b" instead. Faithful readers may remember that in my case deeper analysis requires turning things on and then going somewhere else and doing something else for a while. So I did and I did. The question is, what was the resolution that I found when I returned?

Click Here To Show Diagram Code

[go]$$Bc What is the next move?
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O O b . . . . . . . . . . . . . |
$$ | . . X O X . . . . , . . . . . O . . . |
$$ | . . O X X . . . . . . . . . . . . . . |
$$ | . . a O . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]