In any case, our aim should be to extract a principle. Without a principle, we don’t need an SL page - just look at LZ for analysis.

_________________
be immersed

Me too. Even in John Fairbairn's example... or rather, suppose we found a similar case where mostly all the bots agreed, yet was similarly incomprehensible at first sight for humans. What do you do?

You play with the bot in Lizzie or a similar GUI, and try different openings, vary the placement of certain stones, but repeat the same shape locally, and see when wants the that variation and when not. You play out with the bot and see how it responds to attempts to refute it (using the bot itself to aid you so that you don't just get trounced by superhuman-level play unrelated to that situation). You discuss with other people about it in a serious effort to understand. You over time collect a dozen more distinct examples with the same kind of move suggested.

And maybe the move stays incomprehensible still, maybe even most of the time. But I suspect a nontrivial fraction of time, the seemingly "worse than useless" play by the bot will actually become understandable. You may learn that "ah, right, this is indeed a probe, and it looks like it takes a local loss, but if in this case we can expect to get sente due to XYZ, so we can come back and it is not a loss" or "ah, it looks like this or that formation is strong, but much later in the game, bot suggests this move which suddenly reveals a defect that we had not anticipated", or "ah, it looks like this is a good trade for us, but if we play out dozen more moves, it's actually kind of hard to see where we have enough territory now, we have to re-think our judgment".

The above takes a lot of work, so maybe it's not practical in most cases. But I don't think it's out of reach of human ability fundamentally.

Or... maybe you demonstrate that the bots are mistaken, as they often still are.

As far as what to do on an SL page in the meantime - yeah, in that case some practical considerations are needed. I also would lean towards not just deleting pages or "overturning things", but treating bot opinions just like any other part of discussion. Even if you don't fully trust them or think that you would be able to follow up their moves well enough to be useful, what they say is still meaningful data.

SL provides a lot of functions. For some it is an encyclopedia, for some it is a tutorial, for others it is a filthy mess.
I think humans need their heuristics for narrowing down the choice of their moves, it's the reason that we have pages like Take the last big point in fuseki. When I read analysis on SL, I've never assumed that it is 100% accurate, but I would tend to trust most of the arguments which are presented to me. If the argument is presented really unclearly, I probably wouldn't listen to it. Presenting something like LZ says A is 2% better than B in this position would probably cause me to engage ignore_mode. SL has to retain a human presentation, but that presentation can certainly be enhanced by LZ and other AI.

_________________
North Lecale

Not that this is what you imply but in general I think the power of bot analysis is underestimated. With Bill's series as one of the best learning experiences I have ever had in Go, I would say we are way beyond the state where bots make incomprehensible or even weird moves which we can only understand if we find that one tesuji 30 moves deep. That would even be a misunderstanding of how bots work: probabilistic evaluations are essentially not about deep reads but about wide distributions, so they lend themselves to building intuition rather than developing tesuji.

Of course, articulating that intuition is the bigger challenge. We still communicate our ideas through language, rather than through sequences, although I have argued before that at the expert level, at least in my perception, verbal language is abandoned in favor of go language, i.e. pros show sequences to each other without too many spoken words.

Hence, the articulation problem is not exactly a new one. I believe Charles Matthews coined that term long before AI came about. We do have pro wisdom, sometimes by themselves, often by ghost writers and we may expect that quality is better than what we make of AI sequences on an internet forum. Still, further interpretation of that wisdom has proven non-trivial, especially when many of those books have been written in Japanese or other oriental languages. Our endless discussions on thickness are a witness of that.

If traditional human go wisdom scores better in articulation, today's AI do better in evaluation. If we enhance that with our own, be it fallible, articulation, the result in learning may not be worse.

I was not able to reproduce John's observed behaviour of LZ wanting to run out the ladder and expecting black to atari from on top using a recent LZ 247. I was however able to see something similar with the year-and-a-half old LZ 157 that comes with Lizzie 0.6 on low playouts. So I don't see anything mysterious here, just the well known problem of LZ with ladders on low playouts which are lessened in stronger more recent networks and ameliorated by giving more playouts.

To illustrate, here is 157's initial thoughts on ~300 total playouts, it wants to run out, and you can see mainline variation is for black to atari from on top, white takes a stone in gote and black reinforces the lower right and then white cuts and there's a fight. But bear in mind the first move only had 105 playouts so once you get several moves down the variation there are even fewer so these are mostly just playing on policy network instinct rather than much reading.


Let LZ think a bit more (3k playouts) and now run out is not favoured, it wants to 3-3 invade at lower right, and is also looking at the attach I mentioned before as a promising blue circle move (lower playouts than #1 choice at 3-3, but higher winrate).


What does it now think would happen if white runs out? Mouse over that move and we see that with only 209 playouts it now expects black to atari underneath and then white to tenuki and 3-3.


Does this mean it has realised the ladder works for black? That's easy to test, just play out the escape, atari under, escape, and what does it expect black to do? Answer is atari again, so yes it now has enough playouts to realise the ladder works for black. And we can see with each extra stone white is adding to the captured ladder black's winrate is increasing. This all makes sense.


Give another thousand or two playouts and now that attachment is the blue move, note that more recent LZ engine as I'm using here no longer always picks the move with highest playouts as the first choice move, it's now a combination of playouts and winrate and this attachment already counts to LZ as the best move despite fewer playouts than 3-3.


Is there a point between my first and second diagram where LZ would want to run out, expects black to atari on top and then white would 3-3 like John saw? Maybe, but my computer is too fast for me to find it easily. But if it does happen, you don't need to just scratch your head and retreat into a fog of confusion. Did LZ think extend once and then tenuki to 3-3 was actually a good exchange/probe of testing if black wants to ladder and the take the 3-3, or was it just LZ not reading enough and deciding "oh well, take the stone in gote is bad, I'd better 3-3 now?" Play it out to test!

Extend once, if black atari on top and the 3-3 then with 15k playouts we see 157 thinks black is at 38.7%.


If directly 3-3, then black is 45%, ie this is better for black than before, so it WAS good for white to run out if black will atari on top. This doesn't seem particularly mysterious to me, it makes sense as a probe so white has, in sente, created the option of saving the 2 stones which is a big move for territory and also leaves the cut afterwards.


But of course black may not atari on top, he could atari under as the ladder works. The downside of this is white gets a ladder breaker later. (Or if black answers it then white can run out the stones, which isn't an immediate disaster for black, but if that's going to happen he probably prefered to atari on top instead). We can ask LZ about this too. It says black is 47.5%, so this is the best for black, i.e. with enough playouts black would have atarid from below for ladder if white pulled out, because it doesn't want white to have the option of taking the 1 stone on edge.


I think the bookish theory would often not like pulling out the stone once and then tenuki, because if black spends the next move in the area he makes a tortoise-shell capture which is better for him than his 1 gote if white didn't pull out. However, to spend a move in that area is slow, so this is a case of thinking about local analysis leading you astray globally. We can see that LZ agrees it's a bad exchange for white if black answers correctly underneath (47% for black > 45), but it's a good exchange if white tricks black into playing the wrong answer of top atari ( 39% for black < 45). And because LZ 157 is bad at ladders at low playouts it falls for the trick initially. LZ 247 is better at ladders so it would not atari on top but atari under (or if it would atari on top it is for a much smaller window of low playouts which I didn't catch it in).

QED

The state of affairs is rather that some claims about whole board positions are 50% accurate, as is shown through bot review which shows a line that is 10% (not 2%) better than both the "good" and the "bad" diagram which are roughly equal themselves according to the bot.

I am definitely not going to question moves that are in the 2% range of bot analysis, although if a 3-3 invasion is consistently 2% better than an approach by bot standards, it may deserve mentioning.

Great post, Uberdude!

I have linked to it from the topic, How to use AI for review.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

I've had some further thoughts, and it seems to me that uberdude's analysis with newer LZ's than mine could be used to show two things:

1. The power of AI analysis is overestimated and the power human analysis is underestimated.

Arguments:

(a) The pro kakari move is not yet the best move but seems to have moved up significantly (it was not even on the radar in my investigation) and to be now well within the margin of error Bill postulates.

(b) The later bots are telling us the earlier bots are WRONG (even if they are both stronger than humans), so whatever "principles" the earlier bots were using are also WRONG according to the later bots - which in turn may be shown in due course also to be WRONG. Wrong does not mean worse than humans, but still does not mean RIGHT. On top of that, this is not some slow-moving evolution like the demolition of the flat-earth theory. Changes are happening almost daily depending on which bot or which version of a bot you use. It seems almost pointless trying to make sense of such rapid changes. Just chasing shadows. Shin Fuseki likewise showed that, and that was still on a much slower timescale.

2. The difference between bots and human pros is in danger of being much exaggerated.

That follows partly from (a) above, but much of the discussion on AI, here and elsewhere, seems to me to fall into two camps. One is the "AI is the best thing since sliced bread" syndrome, with some members of that set teetering into a cargo cult mindset. The other "whoa, hold your horses" set mostly urges caution, but in some extreme cases that caution can descend into denial and flat-earthism.

Unfortunately, when humans choose to fall into a camp, they often have a tendency to automatically rubbish the other camp's views. That may be happening here.

Yet, looking at the figures, very many pro moves seem to garner approval from bots, and while humans may be struggling to win on 2 stones against them, constant defeats by a soulless machine able to replicate its behaviour every time is not the same as constant defeats by a distractable human. I'd be very happy to say I was only two stones weaker than a human pro, which would make me one of the best amateurs ever. A pro might not be so happy to say he is two stones weaker than a bot, but I think that's because he has skin in the game - otherwise I suspect he'd be quite chuffed.

If we set aside the camp psychology, could we not argue that pros and bots are not really so far apart, and if that's the case, surely the pros' views of the best moves remain valid even for themselves - and a fortiori for amateurs?

Of course, we can try - misleadingly, I think - to switch tack completely and start arguing from the basis that bots sometimes show a human move is not just different but vastly different (e.g. 30% drop in win rate). I have strong reservations about the use of such examples. I am not disputing that such a move is likely to bad. But in go one bad apple does not spoil the whole barrel. If a pro makes one such horrendous move against a bot, of course he will lose. But what about all the other 250 moves where he kept pace with the bot? Until someone shows us that a pro is consistently - almost every move - falling behind significantly in win rate, it is unfair to rubbish him and his opinions because of a mistake.

To me, the interesting area is the one where even pros don't have opinions, or have differing opinions among themselves. I think I was the first to consistently point out, when I started doing my Go Seigen books, just how often and radically pros can differ - ranging from "brilliant" to "awful" for the same move, with some pros not even deeming the same move worthy of a comment.

Pros actually do acknowledge that they often don't know whether a move is good or bad. The uncertainty may be couched in euphemisms ("I would have played A instead" or "B is also possible") but it is still a strong undercurrent. That is precisely why they are studying bot play themselves. They clearly hadn't found a principle they can reliably apply in some aspects of the game before, but they did acknowledge that even before bots came along. Therefore, when they do feel confident enough to express an opinion about a move or a principle, I think we have to pay heed to it as, at the very least, an honest opinion that has worked for themselves. And which, to repeat myself, has worked well enough for them to get them within a couple of stones of the bots.

I don't want to be in a camp, but if I need to be in one, my camp is "yes, we should try to use AI evaluation to critically assess conventional wisdom, whether it has been spread by pros directly, by their ghost writers, or by amateurs further interpreting it on the Internet"

My original post was not so much "let's replace all pro wisdom with AI wisdom whatever the margin of error" but "I've seen claims on whole board positions on SL, some of which are quoted from pros but others may be assigned to amateurs, which don't pass the test of LZ (>10% difference)".

LZ can misread ladders, more so than KataGo, as I've shown in reply. While that inspires caution, it shouldn't make us despair and say "whatever bots say, future bots will overrule it, so let's stick to conventional wisdom". Sure, 3-3 invasion may be a fad, but so was 4-4 when Go & Kitani came up with it and so was a centre oriented strategy (among amateurs) when Takemiya was in his prime.

Anyway, what I take from this discussion is some caution, especially when
- overruling or criticizing conventional wisdom by actual professionals
- the difference of probabilities is low
and a request to keep traces of the original discussion (which is hard for me who likes synthesis)

John, I don't know what version of LZ and how many playouts you used on your example, but I think before we worked out you were using #157 and only hundreds of playouts. So I didn't use a newer bot to show the older bot was wrong, I used the same bot but just gave it long enough to read the ladder (which was <30 seconds on my PC with a GPU, but may be much more if you have a slow computer). Do not assume LZ #157 with 100 playouts is superhuman, it may well be just mid-dan amateur strength overall (opening positional judgement much stronger, but ladder strength much weaker), e.g. here I (4 dan) beat LZ #145 in an even game when it had about 1000 playouts per move with a ladder, but I wasn't getting massacred before that either. If you want to analyse pro games/opinions with LZ 157 I recommend 10k playouts minimum, and more in situations with ladders and semeais.

Andrew: I have upgraded LZ since you kindly gave advice earlier on, but I admit to not bothering too much about number of playouts. I'm working on the principle that policy moves are more likely to be based on principles, and principle are what - in principle! - we are trying to find. If a bot spends ages and comes up with a move that tactically overrides an earlier principle, that just seems to be the exception that proves the rule. We are all used to that sort of behaviour with go proverbs. I'm quite prepared to be shown I've got my head on backwards, of course.

As to acknowledging that bots make mistakes, I'm well aware of that for the simple reason that I have won quite a few games against them (ladders/tsumego blips), but again I haven't the patience or the machine to go for very high playouts.

Based on my experience working with bots and MCTS, I think using this as a justification to put credence on low-playout search results is dangerous and you simply should not be doing so. It's the precisely other way around - more playouts will usually clarify any "principles" to be found, and less playouts will usually obscure them.

Bots learn things in a different way than humans do (as should be at least partly suggested by their very different style). And one result of this is that the bot's raw policy while being superhuman in some intuitive aspects is often uncertain and mistaken in ways that professional players would hardly ever be uncertain in or mistaken. Search is necessary to correct these flaws, and what you are seeing is exactly that.

So often, it's less that a bot spends ages and ages to come up with some clever tactic, and more a bot spends ages and ages and realizes its policy is being really stupid in several ways at all different points down this or that variation and affecting its judgment in ways that it really shouldn't.

Additionally, despite bot holistic judgment being "superhumanly" strong on average, the individual raw value net evaluations are often are fantastically noisy sometimes. The superhuman part emerges in the average, not in any single evaluation.

So often, it's less that a bot spends ages and ages to come up with some clever tactic, and more a bot spends ages and ages to accumulate enough playouts so that the noise averages out and you are actually left with a reasonable guide to the good moves.

Use more playouts.

Addendum: You can somewhat reduce the number of playouts needed by using a bot *interactively* though, to explore variations dynamically and see what it says rather than sitting and waiting for long enough for the bot to correct its own flaws (if you're on weaker hardware).

Uberdude's post above is excellent and is essentially what I would do. If you haven't already internalized the kind of way of using it that he walks through in that post, giving that post a second read and trying to apply the same kind of thing in your own usage could be worthwhile.

Let me give another example how I use bots for learning.

When I started playing, the "Catenaccio joseki" was very popular, as a response against the equally popular high pincer. Later we were told that it wasn't good for the player who had two stones on the second line, and the pattern went out of fashion.

I tested it in LZ/KataGo, early enough and in an opening where the play would make sense from both sides. First of all, we stay well below the 10% threshold, of course, even below 5%, or below 1,5 points as per KataGo. Secondly, it's not the moves which directly result in low stones that are considered "bad". I know this is not what pros/teachers said at the time but I could have thought that. The "bad moves" are the pincer and the jump:

Click Here To Show Diagram Code

[go]$$W Catenaccio joseki
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . O . . . . . , . . . . . , X . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . 2 . 7 . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . 3 . 1 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . , . . . 4 . X . 5 . |
$$ | . . . . . . . . . . . . a b . . 6 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

When W1 approaches with the idea of breaking up Black's side framework, KataGo (like other bots) puts White at 55%, slightly more than 1 point ahead. The preferred answer for Black is to back off at ''a'' or ''b''.

Black's [pincer] B2 is slightly inferior. White's chances grow to 57% and with half a point. Next, White should invade the corner at B6. However, the [jump] of W3 is the big thing in this pattern evaluation. It loses more than a point and drops White back to 54%.

All the remaining moves don't change much about how KataGo evaluates this pattern. It likes the 3-3 invasion still better than the slide of W5, but marginally so. And it hardly objects against White's [slide] at W7, even if it's on the second line.

After the pattern, KataGo still finds White ahead by 53% and half a point.

My interpretation is that, given the fact that bots like early corner invasions, a pincer makes the corner invasion even more attractive and so loses points over extending the other side of the corner, and the jump makes it much less appealing to jump into the corner, because White has already commmitted to the approach stone, so it loses points over the direct invasion. Once that commitment has been made, sliding to make a [base] is not bad per se.

I tested it in a 4 star point opening as well, where the results were similar.

Edit: The bold part says why I think bot analysis is great for learning: even if knowledge was passed on to me for the right reasons, I may have remembered it for the wrong reasons. I may have learned that playing on the 2nd line in the opening is bad per se. Autonomous analysis with a strong player telling you where (but not why) it really goes wrong, can help unlearning wrong insights.

https://senseis.xmp.net/?CatenaccioJoseki

Here is a remarkable game — remarkable in the number and size of its blunders — that pertains to the question of large points (oba) in the opening, all of which had been played by move 11. It is a game between two pro 2 dans trying out the New Fuseki in 1934. That explains the number of blunders, as they were not very strong and were in unfamiliar territory. By comparison with a play in a corner, all of the oba lost 10% in winrate or more, by Elf's reckoning in its commentary. I wonder if the oba plays would have been regarded as so questionable before the AI era. I have amended the Elf commentary file to include winrate losses by comparison with Elf's top choices in the opening. Tesuji means that the play was Elf's top choice.

To meet the size requirements I have deleted several secondary variations.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Is your "oba" = playing in the largest space available between two stones on the board?

This online go dictionary, http://www.godictionary.net/term/ooba.html , indicates that it is a wide extension of high value. According to that definition a splitting play is not an oba, but since it prevents a wide extension of high value, I have always regarded it as such.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

This is a fascinating thread and a fascinating topic. Center stage is a very real debate about what to do with vast swaths of received "go wisdom," and both sides have very compelling arguments.

I won't pretend to have anything new to add as to the substance of those arguments, but I'd like to urge some caution, as the chosen course of action here has implications for the entire English-speaking go community. If you (or anyone) decides that conventional wisdom must all be tested for purity by the furnace of AI--and proceed to purify the online resources compiling those analyses and teachings--you had better be sure that the AI analyses are actually instructive. Of course some of them are (the "correct" 3-3 invasion sequences being nearly everyone's go-to examples, or the general caution against being pincer-happy), but the body of go wisdom (the tradition, we might say) has the advantage of being means tested for the sake of instruction. Sometimes our search for heuristic guides and human-language-based explanations of go strategy can oversimplify, but that's the nature of human thought at work. The advantage of the body of such thought that's been handed down is in many cases (though of course not all) it's been filtered for its utility in guiding amateurs.

The hard work of converting AI insights into teachable lessons for amateurs is happening here on these forums; I believe this activity has to continue before we simply discard what's been handed down. As noted--baby, bathwater, etc.

Thanks for the caution. Two points:
- I'm more inclined to clean up/correct SL if the diagrams are provided by amateurs like myself than when copied from original pro teachings
- if the material taught is significantly different than the course of action observed, the teaching material will be discarded anyway in due time; in the 90s with abundance of 4-4 joseki, the interest in Ishida's dictionary was declining too

It seems that many people fear that we will say the approach is wrong and the 3-3 invasion is correct, or that an advice is wrong because the bots don't consider it. No, we may say that the 3-3 invasion is more popular by bots (and pros) and try explaining why that may be. And we may force the alternative upon the bot and see how it evaluates the move and try understanding why it wasn't part of its policy. And if a diagram sends out a message that one play is better than another, I believe being skeptical about such firm truth if the bots say both are worse than something else by 10%, that's just healthy critical thinking.

I asked for opinion and I get advice for caution, so I'll take that to heart.

For simplicity, let's assume that, given , the remaining sequence given here is optimal for both sides (questionable due to the stated loss with the pincer of , but that loss is somewhat small). You've indicated that the slides don't result in significant loss.

Then isn't it possible that this sequence, which optimally results in slides on the second line, is inferior to white's 3-3 invasion, because the 3-3 invasion here is better than this sequence which results in white's two plays on the second line? I get that LZ prefers the 3-3 invasion. And I get that the percentage dip happened on the jump and not the slide. But if playing optimally from the jump results in two slides, those two slides could be considered as part of the expected result, and could be considered in the reasoning to say that this sequence is bad.

Imagine:
LZ (explaining): "No, don't play the jump here; that'll just end in two slides on the second line. Play the 3-3 instead, because that's better than playing on the second line here."
You play the jump anyway.
LZ (*sigh*): "Ok, that's a 10% loss, because this sequence ends in that second line play shape, which is worse than the 3-3"

Looking just at the percentage dip, you can get that the jump was a bad idea. But you don't have enough information to say *why* it's a bad idea. Is it just because the jump itself is bad in and of itself? Or is it because the expected sequence (that doesn't lose additional points) results in something inferior to the expected sequence resulting from 3-3? Or perhaps these are equivalent!

---

Think of playing out a ladder as another example: You could say that playing a move to escape from a ladder in futile is bad, because the ladder isn't working for you. Or you could explain it and say that, "the only way that move will work out is if you can escape from the atari. and if you play all that out, the resulting sequence is you get captured. so playing here is bad, because getting captured is bad." The move that dips the % is probably the first one that tries to escape from the ladder. But if you're looking for a *reason*, maybe it's the resulting position, sometimes.

---

My point here is that, if the second line slides "go with" the optimal sequence after the jump, then it's not necessarily unreasonable to say that the slides are bad, if that's what you expect to result from that sequence... ¯\_(ツ)_/¯

_________________
be immersed