Can you find big mistakes?

[Knotwilg]

5. where does AI find a major mistake (point of % difference) which didn't even occur to you

(5) is the point getting criticism here, as a heuristic for finding key learning points.

Unless by 'here' you mean L19 in general, I fail to see where this is happening.

OP says: "When discussing games and variations these days it is likely that someone will quote the computer and its points loss or percentages. Such discussions appear to always be underpinned by the point of view, one that I regard as something of a fallacy, that larger points loss means it's a worse mistake."

As I understood the OP, AI is not even central to the debate. Katago was just a convenient way to find a strong player for the experiment. It could just as easily have been a friendly pro you had locked up in your wardrobe.

I didn't understand the OP that way. Because strong players won't usually comment in terms of point or percentage differences.

In the meantime, the experiment has led to an interesting tangent, so I'm willing to let go of what I perceived as the kernel.

[kvasir]

5. where does AI find a major mistake (point of % difference) which didn't even occur to you

(5) is the point getting criticism here, as a heuristic for finding key learning points. I'm not doing (5) exclusively in my reviews and I don't recommend it as an exclusive, all encompassing device either. When reviewing pro games with commentary, I substitute (2) with the commentary. When reviewing pro games without commentary, there's not so much you can do besides (5). You can form your own opinion about the game but chances are low you can second guess the pro decisions. I would say such exercises are probably more entertaining than educational.

Your comments confirm that there is a widespread view that the magnitude of the change in evaluation is of crucial importance. It is in the fifth step of your method that the computer evaluation appears to be taken at face value to identify key learning points. This is not to the exclusion of all other approaches and it doesn't appear to be the primary tool. However, when it comes to professional games without commentaries it is the primary tool, at least this what I understood.

This post wasn't directed at your method of reviewing games or anyone else's. And, of course, there are other widespread views that I also find suspicious (i.e. that every minuscule changes in evaluation is important, that crossing from evaluating that white is winning to black winning is very important,...). However, this experiment is more relevant to how changes in score evaluation are perceived than changes in other form of evaluation.

There is also, like John points out, a broader point. That is a comment about what is important when reviewing and studying from the human and personal perspective. Just as with the example of penalty kicks in women's football, it would similarly be difficult to make much progress in Go unless the player's weaknesses are identified correctly.

The only way to be objective would be to take your role as a player as the starting point. It is after all a game between two human players. In contrast, the computer's evaluation will only tell you what the likely result is if two similar programs play each other from some position. This is often very different from what happens if you were to play another human player. That is, the computer evaluation is not the most objective evaluation. To have a more objective outlook you need to add your own perspective of the game and that of your opponents.

Similarly, if you wish to find your own mistakes it is not objective to conclude that if it is a mistake for someone else then it is a mistake for yourself. Especially, if the someone else is a computer program, which "thinking" you can't even begin to imitate and it is inevitable that you will play poorly in comparison.

There was also the other point. Sometimes we try so hard to fail spectacularly. That is, any move would have been better than our move. One quality of good players is striking a balance between pushing to find good moves and avoiding bad moves. Or more precisely, this is a quality that good human players exhibit.

[Knotwilg]

Your comments confirm that there is a widespread view that the magnitude of the change in evaluation is of crucial importance. It is in the fifth step of your method that the computer evaluation appears to be taken at face value to identify key learning points. This is not to the exclusion of all other approaches and it doesn't appear to be the primary tool. However, when it comes to professional games without commentaries it is the primary tool, at least this what I understood.

You understand correctly and I do stand by that criterium. One reason to treat "mistakes" by order of point difference is that larger differences are falling outside the uncertainty boundaries of the system. Another reason is they are likely to impact the game more. It's almost a circular reasoning trying to explain this. That's probably the reason why it's widespread too ...

This post wasn't directed at your method of reviewing games or anyone else's. And, of course, there are other widespread views that I also find suspicious (i.e. that every minuscule changes in evaluation is important, that crossing from evaluating that white is winning to black winning is very important,...).

Since these minute variations are close to the precision of the system, these are generally disregarded. I don't find those practices so widespread.

However, this experiment is more relevant to how changes in score evaluation are perceived than changes in other form of evaluation.

There is also, like John points out, a broader point. That is a comment about what is important when reviewing and studying from the human and personal perspective. Just as with the example of penalty kicks in women's football, it would similarly be difficult to make much progress in Go unless the player's weaknesses are identified correctly.

Of course. But AI evaluation is a starting point. And since we don't have Sai on our shoulders to tell us on a daily basis what we're doing wrong and why, AI is the best we have. Repeating: we shouldn't just stare at it. We should reason for ourselves and then verify. And with many repetitions of that try finding patterns. We can ask a forum like this for a shortcut too. At 1-2d, I won't get massive reply with high confidence. So I resort to self review with AI.

The only way to be objective would be to take your role as a player as the starting point. It is after all a game between two human players. In contrast, the computer's evaluation will only tell you what the likely result is if two similar programs play each other from some position. This is often very different from what happens if you were to play another human player. That is, the computer evaluation is not the most objective evaluation. To have a more objective outlook you need to add your own perspective of the game and that of your opponents.

I would argue the opposite. Adding your own perspective makes it subjective.

Similarly, if you wish to find your own mistakes it is not objective to conclude that if it is a mistake for someone else then it is a mistake for yourself. Especially, if the someone else is a computer program, which "thinking" you can't even begin to imitate and it is inevitable that you will play poorly in comparison.

I beg to differ. As a father, perhaps, and then still there are proven practices to raise a kid. As a Go player, hardly. I might enjoy the game more when killing a big group than by playing better endgame but that doesn't mean a 10 losing move which is a reckless attack is "good for me".

I dislike analogies with a passion but penalty kicks are proven to statistically score more when kicked high in the corners. If you know the goalkeeper, you may factor in personal differences. And there remains a fair deal of psychological warfare. When you decide to go for the middle because you thought you outsmarted the keeper, the stats will tell you the high corners were better options. When you go for a high corner and you fail, the lesson is to train harder on high corners, rather than going for the middle next time. There are exceptions. My compatriot Eden Hazard was extraordinarily good at waiting until the very last moment, observe the keeper and change direction in the very last fraction. We're talking top professionals now. Top pros are entitled to defy the odds.

[Knotwilg]

I think the moves in the game give a different perspective on bad moves. Many moves in normal games are already worse in terms of evaluation than mistakes made when someone is not really playing Go. I can put it bluntly: when trying to play Go you shouldn't be happy with moves that are no better than if someone is not playing Go.

I wonder. Is a move that tries to achieve something but fails badly, eg. because of a misread, or because it was the start of an "all-or-nothing" abandon sequence, really worse than a random tobi that doesn't even try to do anything meaningful ?
I think I would be happyer with the former.

At least you will learn more from the former: it was played with good faith but turned out to be a mistake with massive impact. Maybe the lesson was very specific, maybe there's a technical pattern in there, maybe the general lesson is to read very hard in a fight and bail out to a conservative move when uncertain.

Deliberately playing badly, there's no learning. The fact that deliberate bad moves may not result in big point differences is not disproving that big point differences are likely to point to major mistakes, either.

But I think the OP's main point is that proper learning can't be correlated to major point swings in AI evaluation. I strongly disagree with that point of view. If learning doesn't reduce the randomness of the game, to bring the swings closer around the equilibrium, you're learning the wrong things.

[xela]

Women's football is fascinating analogy.

Imagine a penalty kick. A premier league player misses one and the game ends in a loss instead of a draw. That's a big as well an important mistake. The pro and his coaches (the equivalent of using AI in go) will spend a lot of time analysing ways to make sure that doesn't happen again. This analysis will include advanced things such as the goalkeeper's size, his preference for one side or the other, but also other arcana such as the state of the pitch, or the game, or the weather, and of course the temperaments of the various possible penalty takers.

But in women's football we used to see penalty kicks being missed simply because the penalty taker could not kick the ball hard enough or accurately enough. The coaches in women's football therefore concentrated on improving fitness and strength and intuition, and simply practised kicking in general. We have seen the results for ourselves - the women's game is now blessed great skill and excitement, at least at the highest levels.

Amateur dan players in go are still at the level of the early female penalty takers. Remember that Go Seigen famously said of a player "he's very weak - he's only a 4-dan pro." As proof of his words, he beat Yasunaga Hajinme 4-dan down to four stones. So, I for one, certainly think that amateur dans playing with AI, except as an experimental device, are being wrongly obsessed with bling and "Go faster" decals.

What they need, instead, is go's equivalent of kicking or heading a football reliably every time...

I think it's a terrible analogy, but an instructive one :-)

If you kick a ball and it doesn't go where you intended or as fast as you intended, then it's immediately obvious what happened. Likewise if you're running towards the ball but an opponent gets there first because they're faster, or you have to slow down because you're tired... I think amateur footballers are well aware of the gaps in their technique and fitness.

In go, if you make a bad shape, or a bad choice about direction of play, the consequences of the mistake can be tens of moves later. And it's dependent on the opponent knowing how to punish the mistakes: in amateur games you often won't see any consequence at all. Without a teacher beside you, you can't see clearly. "Play more consistently" is good advice, in the sense that if you do it, you'll get better results. But it's not useful advice unless it's accompanied by a lot more instruction on what typical small mistakes look like and how to avoid them.

When you sit down to review a game without a teacher in the room, but with access to an AI, what are you going to do? Look at the numbers, or not look at the numbers? The numbers may not be perfect, but they're better than getting no help at all. We've already had many conversations about how the numbers can sometimes be misleading, and how to interrogate the machine, compare alternative moves, get its evaluations on different lines of play, get more information out of it than a single number... You need to work a bit to filter out the signal from the noise, and you won't get it right every time. (You won't get accurate, clear and useful advice from a human teacher every time either.) But that's not a reason to give up. Don't throw the baby out with the bathwater!

Practically speaking, if the AI review says you made three mistakes of 10 points or more, and 30 mistakes of around half a point, where are you going to start? Some of the half-point mistakes might be highly significant psychologically, and thinking about them might help you on the road to more consistent play. But many of them will lead you down paths that are not going to help at all. And if you can't see clearly, you don't know which is which. You can spend a lot of time lost in the wilderness. If you look at the 10-point mistakes, you won't strike gold every time, but I think you've got a lot more chance of finding something useful.

[xela]

\I decided I'd only play next to my previous stones or one space jumps (sometimes I'd jump over the white stones)...

So I thought: could I edit the KataGo source code to make it play according to this policy? Given its highly trained judgement, together with this odd type of handicap, would it still be superhuman, or would it be easy to beat?

It's more effort than I'm up for this week. But it turns out someone has done something very similar, with knight's moves instead of one space jumps, and put it on OGS as a bot. And it's ranked 5k there. See https://online-go.com/player/796120/KeimaBot

[kvasir]

So I thought: could I edit the KataGo source code to make it play according to this policy? Given its highly trained judgement, together with this odd type of handicap, would it still be superhuman, or would it be easy to beat?

Could be fiendishly hard to beat in handicap games. If we assume that it is allowed to play some other moves. Unless moves like blocking, hane, nobi, atari and capture are allowed I can't see how it could work. I think the keima bot is playing exclusively knight moves, this wouldn't work at all with one space jumps. The difference is that knights move can reach all near by intersections in just a few moves but one space jumps can never play some intersections.

[kvasir]

You understand correctly and I do stand by that criterium. One reason to treat "mistakes" by order of point difference is that larger differences are falling outside the uncertainty boundaries of the system. Another reason is they are likely to impact the game more. It's almost a circular reasoning trying to explain this. That's probably the reason why it's widespread too ...

It does seem like circular reasoning to me. For one thing you appear to justify your view about points differences by referring to the same points differences.

I just don't see why the important thing is the magnitude change in computer evaluation rather than any other criteria.

I would suggest that it is more important to learn something, anything, than it is to somehow identify the right thing to learn. If someone could always learn one thing from every game that they played then that would be greater capacity to improve in this game than that of pretty much everyone else. Even if only by small steps the impact is huge with enough iterations.

Since humans, and animals for that matter, learn really fast from only a few positive examples I'd also suggest the interesting question isn't how to identify mistakes but how to identify lessons that can be learned easily. For example that the one space jump is never bad.

I would argue the opposite. Adding your own perspective makes it subjective.

Perspective is to me something intrinsic that can't be eliminated. Maybe you can change perspectives (but this seems hard) the way you could change your standpoint. You can try to offer more than one perspective if one doesn't do. The history of perspective in art is really very interesting, go back about 1000 years and no one appears to be able get it right. There is something called Byzantine perspective or reverse perspective, which projects images onto canvas with no regard for the viewer. Leaving the one that is to experience the art, or the game review, out of the equation isn't the accepted way to offer an experience of something objective.