Why isn't playing a move you want a bot to analyze obvious?

[lightvector]

(quoted from another thread)

This latter phenomenon (rating unconsidered moves highly) is too common - troubling even - to ignore. It needs a name so we can talk about it more. I think it was Bill Spight who first noticed it, and is certainly pointing it out most often, so I propose we call it Spight Analysis, or something like Spight Retrospective Analysis. When we use this tool, human pros can usually be shown to be performing very often only a whisker away from AI-bot level.

Setting aside the phenomenon of bots ignoring moves that are almost as good, and focusing just on the fact that if you want a bot to analyze a specific move much more accurately you can play that move on to the board - I'm wondering why this isn't already intuitively obvious to most people. It has seemed completely obvious to me ever since MCTS bots existed - a bot isn't analyzing enough a move that you want to know about? Well play it on the board and now 100% of the analyzing will be for that move.

This isn't special to Go or MCTS even either. It's also what I see tons of people do all the time intuitively with chess bots, online and offline. Chess bot suggests move A puts you up a pawn's worth. What about move B instead? Well, play it on the board and have the chess bot tell you now whether you're still up a pawn or whether actually now you're even or behind.

Maybe subtle differences in Go UI and Chess UI are what result in this? Or maybe just the Go community is still just really inexperienced with what the Chess community has known by now for decades?

[dfan]

Setting aside the phenomenon of bots ignoring moves that are almost as good, and focusing just on the fact that if you want a bot to analyze a specific move much more accurately you can play that move on to the board - I'm wondering why this isn't already intuitively obvious to most people. It has seemed completely obvious to me ever since MCTS bots existed - a bot isn't analyzing enough a move that you want to know about? Well play it on the board and now 100% of the analyzing will be for that move.

This isn't special to Go or MCTS even either. It's also what I see tons of people do all the time intuitively with chess bots, online and offline. Chess bot suggests move A puts you up a pawn's worth. What about move B instead? Well, play it on the board and have the chess bot tell you now whether you're still up a pawn or whether actually now you're even or behind.

Maybe subtle differences in Go UI and Chess UI are what result in this? Or maybe just the Go community is still just really inexperienced with what the Chess community has known by now for decades?

This has surprised me a bit as well. I think part of it is that Go players are still learning to use and interpret these sorts of tools, but another aspect is that a fair amount of current Go AI analysis is not performed interactively in practice; you feed your game to GoReviewPartner or OGS or what have you and get back a report. So it's not as natural to a lot of people (yet) to use these tools to explore interactively rather than perform a static analysis. I don't think this was ever such an issue with chess engines, although comments such as "Kasparov's 23rd move was not considered by any engine, and they don't see how good it is until three moves later!" have always been popular.

[hyperpape]

Is it a common blind-spot to not realize you can check that way?

I thought John was not pointing out the technique so much as the consequence. Though the bot didn't consider our move, that doesn't imply that it's not a good move--sometimes it's just that the bot has a very different style.

[Kirby]

Is it a common blind-spot to not realize you can check that way?

I thought John was not pointing out the technique so much as the consequence. Though the bot didn't consider our move, that doesn't imply that it's not a good move--sometimes it's just that the bot has a very different style.

Probably lightvector was pointing to the phrasing: "too common - troubling even". If it's easy to explore other options that might be good, what is there to be troubled about?

[John Fairbairn]

I thought John was not pointing out the technique so much as the consequence. Though the bot didn't consider our move, that doesn't imply that it's not a good move--sometimes it's just that the bot has a very different style.

Precisely.

It is glib to say the technique is easy and obvious. It is debatable to say it is obvious, because people unfamiliar with how these programs work (like me) don't know what to trust. There's been enough argument among "experts" about the meaning and/or reliability win rates on this site, for heaven's sake.

It may be easy (when you know how) but that doesn't mean people do it. It's also easy to take your litter home with you - how many do it?

My strong impression is that this "easy" thing is not done very often here (which is why Bill's efforts stand out to me) and the result is that comments typically of the type "the human played here but the bot played there" - which I find "too common" - leave a residual sense that the human was a duffer, even if he was a pro. That is what I find "troubling."

My own sense of what is going on is that bots beat humans mainly because they are much more consistently accurate. They do not get distracted or tired. They do not make rash moves just because the human has gone to the toilet. For the most part I believe the pros do know what they are talking about. as more of these "easy" checks might show. And of course they can talk about everything far better than the bots.

[Kirby]

My version of Lizzie preferred the shoulder hit. It didn't even list Takemiya's Showa move as a candidate. No surprise there. But what might surprise some is that when this Showa move was forced upon Lizzie, she didn't bat an eyelid and gave the same win rate as before!

You were curious about Showa's move. You tried it out to see what your bot thought, and got an answer. Seems like you figured it out.

For the most part I believe the pros do know what they are talking about. as more of these "easy" checks might show.

Easiness depends on how you are using Lizzie/LZ. If you have a pro game loaded in Lizzie with pondering mode on, as you go through the game, you'll see LZ's analysis of the moves you're exploring in the game tree. From this perspective, it's pretty easy to get a bot's analysis of what the pro played. In addition, you'll also see a handful of alternative moves that LZ may have considered - but it won't show you all possible moves. Nonetheless, the move the pro played in the game you reviewed is there already, because that's the move the pro played.

The example you gave was slightly different in that you wanted to explore a move that wasn't loaded or played. LZ had some possible suggestions, but it's not exhaustive. But then when you played the move, you got the analysis, as expected.

So if you are interested in seeing what LZ thinks of a particular move, I would recommend either loading that move from an SGF or playing that move. LZ won't explore every possible move, especially if it's not one that's part of the game you're reviewing...

Good luck!

[jann]

I think this IS specific to MCTS. Average users may not be aware of the differences between chess-style and go-style analysis - they assume the bot analyzes the position, and that's all there to it.

To an extent this is true: the bot should find a move whose "real" winrate is not far from the objectively best move's one (its search would return garbage otherwise). But this only means unconsidered moves are unlikely to be significantly better (and most of them likely significantly worse).

BTW this is partially the UI's fault as well. LZ for example allows some custom shaping of its search tree, the UI should actively extend analysis to all branches it sees in the SGF game tree, even if the bot would not consider the game move by itself. This will likely be done as UIs mature.

[Bill Spight]

The example you gave was slightly different in that you wanted to explore a move that wasn't loaded or played. LZ had some possible suggestions, but it's not exhaustive. But then when you played the move, you got the analysis, as expected.

So if you are interested in seeing what LZ thinks of a particular move, I would recommend either loading that move from an SGF or playing that move. LZ won't explore every possible move, especially if it's not one that's part of the game you're reviewing...

One advantage of playing the move if it one of the bot's lesser choices is that playing the move will give you more playouts, and therefore a more reliable winrate estimate. In one case I saw the winrate estimate gain 20%, making it the new first choice.

[gennan]

Even when a move is not the AI's choice, the move is fine if the winrate isn't affected much. So when analyzing with an AI, I mostly look at moves where the winrate clearly drops. And even if the winrate drops quite a bit, crosschecking with KataGo may reveal it only loses 1 point. Also in those cases, I wouldn't consider it a reason to deeply investigate alternatives. It's more like an inaccuracy than a clear mistake (at least for my level of mid-dan).

[Gomoto]

Why isn't playing a move you want a bot to analyze obvious?

Analyzing with a bot is a skill obviously

[lightvector]

I thought John was not pointing out the technique so much as the consequence. Though the bot didn't consider our move, that doesn't imply that it's not a good move--sometimes it's just that the bot has a very different style.

Precisely.

It is glib to say the technique is easy and obvious. It is debatable to say it is obvious, because people unfamiliar with how these programs work (like me) don't know what to trust. There's been enough argument among "experts" about the meaning and/or reliability win rates on this site, for heaven's sake.

It may be easy (when you know how) but that doesn't mean people do it. It's also easy to take your litter home with you - how many do it?

My strong impression is that this "easy" thing is not done very often here (which is why Bill's efforts stand out to me) and the result is that comments typically of the type "the human played here but the bot played there" - which I find "too common" - leave a residual sense that the human was a duffer, even if he was a pro. That is what I find "troubling."

My own sense of what is going on is that bots beat humans mainly because they are much more consistently accurate. They do not get distracted or tired. They do not make rash moves just because the human has gone to the toilet. For the most part I believe the pros do know what they are talking about. as more of these "easy" checks might show. And of course they can talk about everything far better than the bots.

Sorry for wording things a bit bluntly - yes I agree very much with everything you mentioned! The thing that I find striking is that is that at least among the Chess players I know, very few if any of them would suggest "the human played here but the bot played there" would imply that the human move was bad, without further analysis. And among strong players I've seen a lot of practical knowledge about the ways bots are and aren't useful for human analysis.

And it's not as if the evaluations given by (pre-AZ) Chess programs are much better than winrates. Ostensibly they're scaled to be units of material (i.e. pawn = 1), but in practice you quickly notice that they often differ significantly from it due to all sorts of positional factors or tactics, sometimes with no obvious or fixed correspondence to the material in a given position. So in practice they too are an opaque value with no useful intrinsic meaning other than "higher is better on some ill-defined scale". But experienced players generally already know that very small differences are not necessarily trustworthy, and have an understanding for what differences actually do mean in practice. And I highly doubt that it's due to most of them having read a statistical error analysis written by Chess bot devs of the flavor that that Bill has wished from Go bot devs.

These observations might not be entirely representative as I haven't actually delved *that* much into Chess. But to the degree they are at least somewhat accurate, I suppose what's going on is that there hasn't actually been a generation of strong Go players who have grown up with the bots from the start. And I guess that there's also not that much overlap between the players of the two games, so all the confusions in the Go community are unravelling only painfully slowly, despite the fact that seemingly right next door in computer Chess thousands upon thousands of people already have the answers.

[EdLee]

there hasn't actually been a generation of strong Go players who have grown up with the bots from the start.

This reality is only a few years away.
Like the generation who never saw a world without global connections at their fingertips.

[Bill Spight]

And it's not as if the evaluations given by (pre-AZ) Chess programs are much better than winrates. Ostensibly they're scaled to be units of material (i.e. pawn = 1), but in practice you quickly notice that they often differ significantly from it due to all sorts of positional factors or tactics, sometimes with no obvious or fixed correspondence to the material in a given position. So in practice they too are an opaque value with no useful intrinsic meaning other than "higher is better on some ill-defined scale". But experienced players generally already know that very small differences are not necessarily trustworthy, and have an understanding for what differences actually do mean in practice. And I highly doubt that it's due to most of them having read a statistical error analysis written by Chess bot devs of the flavor that that Bill has wished from Go bot devs.

I doubt it, too. They are relying on experience. And I'm gaining experience, too, feling my way along as I try to understand the Elf commentaries. That's why I have revised my estimated margin of error for Elf (when not too far from 50%) from 5% to 4%. But chess players are ahead of us by a decade or two in gaining that experience. And it seems to me that win rate estimates are better than centipawn estimates in that they are checkable. It's just that nobody has bothered to check. If I were in academia I might do the research.

[Knotwilg]

I admit I have been guilty of suggesting the human move was inferior because the bot didn't consider it, although the qualification was largely implicit and in the eye of the beholder. As I'm gaining more experience with LZ (my current analysis tool) I have learnt to be cautious, not always to trust the bot and to investigate these unseen moves by feeding them. Although this is a small community, my misguided efforts should not have a large impact.

The title of this thread is a bit quora-trolling-alike. "Why is X" while X is debatable. For those who have installed an analysis tool, it is obvious to play a move you want the bot to analyze. I'm usually a late adopter of technology (and once I have a brand, I stay loyal). I'm a natural conservative, not an early adopter. I think if you are smart enough to play go, you are smart enough to install a bot for analysis purposes. It's just a matter of following instructions.

Now that we're here, what frustrates me when analyzing with LZ is, in order of priority:

1. That White starts out with 55%, suggesting komi is too large, but we don't seem to think that part of her assessment is true.
2. Ladders. They are not rare and can play a big role in a local evaluation. Then you see the ladder forming and facepalm ...
3. When you play out a suggested sequence with winning percentage x% on the first move, on any move (i) in that sequence the winning percentage x_i can vary by a lot.

[Uberdude]

Now that we're here, what frustrates me when analyzing with LZ is, in order of priority:

1. That White starts out with 55%, suggesting komi is too large, but we don't seem to think that part of her assessment is true.

We don't? I think it's true, and I thought most people did too. So long as it's understood in the correct context, i.e. LZ thinks that in games between itself white has a slight advantage with the 7.5 komi. That such an advantage exists in games of a superhuman player doesn't mean a weaker player can make significant use of that advantage.

P.S Title is obvious to me too
P.P.S I think Bill really should install Lizzie.