https://mp.weixin.qq.com/s/03YjCodvYpw8yRJywrXl1g

By Cui Can 5P

I just read a paper using Kata to analyze ancient games which was quite interesting
It compared the "big 3" Ancient Chinese players (Huang Longshi, Fan Xiping, and Shi Xiangxia), Dosaku, Jowa, Shusaku against modern top professionals and top amateurs

General conclusion: no significant difference in the first 180 moves between Huang, Fang, Shi, Jowa, and Shusaku vs top professionals, while Dosaku is a bit lower.

All were significantly "more accurate" compared to top amateurs.

With regards to opening, all had significant differences compared to top pros, but Huang/Fang/Shi are not significantly different from Jowa/Shusaku.

As a side note, even top amateurs had "more accurate" opening than the ancient players, indicating that there have been advancements in opening theory.

Some other interesting findings:

1) Even though Dosaku era openings look a lot more reasonable by modern standards, they actually were significantly worse than the Huang/Fang/Shi openings which look a lot more weird, probably because modern opening theory is derived from Dosaku's lineage.
2: Shusaku was incredibly consistent.
3: AI did not think very highly of Lee Changho compared to several other top pros, probably partly because the analysis took only up to 180 moves so his endgame was not fully considered. Another possible reason is that AI might not think too highly of playing "safer than usual" moves to ensure win.

I have noticed that style plays a huge role in the accuracy of the first 50 moves.

So I am not surprised that Lee Changho does not score high, big part of his style was literally to walk a thin line of almost underplaying in the opening in order to direct the game into situations that played to his strengths.

I have been running some analysis on games from Go Seigen's books "black opening" and "white opening", that I think is based on serialized articles in Go World from the 60s and 70s. It is actually striking how low the accuracy is for the first 50 moves. I think it comes down to style, a clash of styles and possibly recording errors in a couple of games. Interestingly there are many games that seem to be "corrected" versions of actual title matches, these corrections are definitely not "computer approved" since they are sometimes worse than the actual games.

Then I have also checked practice games by pros that I have observed online and the accuracy in the opening is actually really astonishing, even though the style is not very AI driven. These seem to be practice games (I haven't tried to ask what they are up to), so they may be more focused on being good training partners than anything else.

I have also started checking my games, and really concluded that accuracy alone is not really a good measurement of quality. Usually the first 50 moves are pretty good in my online games, judging by accuracy, percentage AI move and top-5 AI moves, but my experience or feeling is that it is not closely related to how good the effort was. Sometimes it seems like the flow of AI moves is with one or both of the players, especially if the game is quiet, but other times it is like every move is terribly wrong every time according to the computer even though the game seems completely reasonable given the level of players, but when this is the case it is usually not until the middle game

I'd be interested if time limits were taken into account, it is obviously a huge factor. I wouldn't be surprised if longer time meant much higher accuracy, but then I'd also think that very fast games could have better openings than some longer games simply because the players fall back to something less provocative that just works. So that would actually be very interesting.

Thanks for thr tip-off. The results don't surprise me, in the sense that they seem consistent with several other recent opinions. But the methodology here is way more impressive, of course.

A couple of remarks:



In one sense, Huang, Fan and Shi may be more impressive in that they appear to have played faster than their Japanese counterparts.



I suspect Dosaku himself may have come out worse because almost all his best games were giving handicaps. As to Chinese openings looking weird, that is a consequence of group tax, but Katago has already given the thumbs up to the old Chinese style with that rule applied. I am munching contentedly from a pack of dried figs on which the advertising blurb says "Naturally ugly but delicious". I think something similar can apply to old Chinese openings.



Is this a fair comparison, given that old no-komi go was a kind of asymmetric warfare, with Black able to adopt a different style of play from White? Shusaku's consistency ("I had Black") seems relevant to that.

The komi can also be changed in Katago's settings.

I know, and have said so. But if it's set at 0 Black can play safe and White has to gamble - asymmetric.

(And it might be worth noting Shusaku thought komi should be about 5.)

Thanks for the good post Illuck.

There are a few statements that in my mind don't make logical sense, so please clarify:

- Dosaku's opening is comparatively weaker than the ancient Chinese
- today's top amateurs have stronger opening than the ancient Chinese, indicating an advancement in opening theory
- opening theory has been derived from Dosaku more so than from ancient Chinese

How does that fit together?

Thanks!

Does Katago play the opening in an asymmetric way when the komi is set at 0 ?

I don't actually know but it's not really relevant when you're a human just trying to win the game and not produce perfect play. Remember the days when Monte Carlo bots won every game against you by half a point? You would probably never say they were just half a point better than you.

One thing that comes up in AI analysis of Shusaku's play is sheer consistency. I assume that's reflecting his play-safe attitude. At least it makes him stand put from other players, I gather.

It does, it is actually very practical when you use katago to analyze no komi and reverse komi games. For example the variations look a lot more "normal" in the first 50 moves for black in reverse komi games than in komi games. There is also a lot more slack in the point estimates, comparing the top few moves. This is for reasonable amount of playouts, basically with komi it will search lot of fantastic variations early but without komi or reverse komi more normal ideas seem to have priority.

Of course this is subjective but I'd say "definitely!", and I am not talking about in the limit with excessive playouts only the practical experience.

A really good point about accuracy and quality - it does seem like over enough games samples there is a clear correlation, but definitely valid (for example, someone at the Baidu Tieba noted that the "accuracy" difference between Lee Sedol and Lee Changho was greater than between Lee Changho and top amateurs).

Definitely a good point as well. My personal suspicion is that Dosaku's "accuracy" was lower partly due to how much stronger than his opponents he was (so he probably didn't have to be 100% focused) and partly due to him taking white in a lot of games where a certain degree of overplay was probably needed.

For a bit of an additional context, below is a table somewhat demonstrating "consistency"

I think Shusaku's games also included games where he took white (though probably smaller proportion compared to Dosaku?) Would be interesting to see if there is a big difference in "accuracy" between games as black vs games as white.

Edit: Forgot to mention that the table indicates average estimated winrate decreases from the "worst" N moves in each game.

Just my two cents:

Partly this is due to Group Tax which changes opening slightly (for example, Kata under Group Tax do not favor early 3-3 invasions), so there is some inherent difference there (Kata with Group Tax was used to analyze the ancient Chinese games).

Another part is due to the fact that a lot of opening choices are more or less driven by fashion - there are many moves that are close to optimal in the opening, and the ancient Chinese players had a preference for one set (e.g. 9-3 loose pincer to respond to keima approach) which, while looking quite strange in today's eyes, doesn't actually lose much compared to the more "meta" responses (and under Group Tax settings has been played by Kata).

A reason the author proposed is that might be due to the four "seated stones" at the star points at the beginning of the game. That may also just give fewer chances for mistakes compared to empty corners, but how much that impact is I have no clue.

Edit: Just re-read what you wrote and realized that the above doesn't directly answer your question. If we take those three premises as true (which it might not be), one interpretation is as follows:

Dosaku (earlier era than Jowa/Shusaku and Fan/Shi, around same time as Huang), the oldest "ancient Japan" style opening in the context of the players analyzed - call it J1.

Jowa/Shusaku, slightly more recent "ancient Japanese" opening - call it J2.

Huang/Fan/Shi - I think study didn't really differentiate too much (though it might have and I just forgot), assume they are similar, "ancient Chinese" style opening (has Group Tax) - call it C1.

Modern Amateurs/Professionals - sourced mostly from Japanese teachings (though obviously with later developments by all three countries) as they were the strongest players in the late 1800's/early 1900's. Call it J3.

The paper's interpretation of the analysis results is that J1 < C1 = J2 < J3

Does that answer your question?

kvasir (or anyone): when looking at old Chinese go, does Katago make any allowance for the facts that points count in a seki and there may be an extra point for last dame. I have in mind Ohashi Hirofumi's assertion that in a half-point game with 90% win rate, a one-point loss (or presumably point counted in a seki) can reduce the win rate to 10%.

I think you are in luck, John Fairbairn, because I saw a post somewhere that someone did this experiment of training weights for KataGo to use group tax rules. There were some examples of how the large enclosure for a hoshi was now preferred. I'll need to look for this post.


Edit:
https://lifein19x19.com/viewtopic.php?f=18&t=18031&p=263305#p263305
You actually replied to this thread, so maybe this is not new to you. Anyway, if you use KaTrain you can actually set the rules to "Ancient Chinese".

Edit 2:
You can actually check the exact rules available:
https://lightvector.github.io/KataGo/rules.html

As a side note, my personal take on the results is that while the methodology appear to make sense and source sgf/analysis are published, it is quite surprising. I'm of the camp that the modern pros are likely stronger due to a much more structured (and likely earlier and more intense) learning process in addition to having multiple very strong opponents to motivate each other. The two pieces that I am wondering about relate to whether "accuracy" is a good enough indicator of strength (given the fairly limited number of the ancient games) and whether "accuracy" between komi and no komi games can be compared directly.

Nonetheless, I have to say that my opinion of the historical players has definitely been changed by the paper (at the very least, the results indicate somewhat comparable strength).

Over the years I have seen many flattering comments about the skill of old players. I tended to brush them off as either polite remarks or as nationalistic jingoism, despite the fact that I think it's a no-brainer that Leonardo da Vinci was a better painter than, say, David Hockney (whom I admire BTW).

But when I was starting my Museum of Go Theory project, I took a closer look at these comments, and invariably found that they had real substance to them. At around the same time, various people had started running AI tests on old players, and although it would be rash to call them scientific, they too have tended to corroborate the early opinions (of people, like Go Seigen, I might add!).

There are special conditions to do with conditions for go in China in the early Qing (i.e. the Huang/Fan/Shi era), such as the wealth of Yangzhou and its go sponsors), and the peripatetic nature of go masters, spreading the game nationwide in a way that did not happen to the same extent in Japan. There was also the sheer number of players, who did not rig the results of games, as Shusaku did with his Castle Games - most Castle Games were controlled in some way (e.g. "too ill to play, take my pupil instead"). In China the competition was much more honest and, I suggest, forced up the skill level. No-one has yet come up with a convincing explanation of why the level of go in China dipped after Fan and Shi, and it is a shame that too many people (even in China) judge the overall level of old Chinese go by its late Qing standards. This paper seems to confirm that we should be basing our judgements on early Qing.

Go is a zero sum game. All that matters is the win or loss. Playing an opening that an AI likes may have some correlation to winning the game, but it's by no means a pure indicator.

So an analysis like this is cute, and shows how well some pro openings compare to AIs, but it doesn't do much to indicate "who was stronger". The only sure way to do that is to have direct competition between the pros in question.

You can have a 10k that memorizes a bunch of pro openings, and that 10k can play like a pro for awhile. But that 10k isn't a pro. Have him play an even game against a pro, and that'll become clear very quickly.

_________________
be immersed

Something else to consider, best practical play vs best theoretical play. By which I mean taking into account who is ahead, who is behind, how much time does each player have to work with, etc.

If you are behind, contemplating two alternative plays, A and B, where A is slightly better than B but is simple, few ways for the opponent to go wrong, B may be the better practical play. On the other hand, if ahead, a slightly inferior but simple A might be the better practical choice than a slightly better but very complex line. How much time left relates to this. If one player has lots of time but the other is in time trouble, the better practical choice for the first player might be a very complicated line even if slightly inferior to a theoretically better simple line.

I do not think even our best AIs are yet able to take these factors into account but are returning best theoretical play assuming everything else is equal. But typically NOT equal. Usually one player or the other is ahead and one has more time than the other. Since the ancient pros would be taking these things into account and the AI not, comparing might be misleading.

Or am I wrong? Do we have an AI returning the best PRACTICAL move?

I think this is generally where the very top players excelled. Though I think that meaningful positional judgements tend to occur later in the game (maybe median around move 120). As mentioned Lee Changho and Shusaku might be counter examples who played a bit closer to underplay even early in the game. Many of the other tops pros (think Lee Sedol, Cho Hunhyun) were more known for finding more practical ways to shake a losing game up.



Kata does a mix. It can avoid capturing races in won games and I believe that Kata's more risk taking play in reverse komi and handicap games is evidence of a practical approach as well. Time control is probably irrelevant to classical players. Definitely a factor in modern Go though.