Life In 19x19

Dear all,

I'm happy to share the result of 11-months of training a new KataGo version to (probably) solve Igo Hatsuyoron 120 problem!!!

This is probably the most difficult Go problem ever created, see wikipedia for the historical context: https://en.wikipedia.org/wiki/Igo_Hatsuy%C5%8Dron

This was done with the incredible help from Thomas (Cassandra here, one of the very best world experts of this problem) and David (lightvector), creator of KataGo:
- Thomas helped a lot by providing many new positions to submit to KataGo to train it. This was an interactive and crucial activity: I regularly sent him new and better versions of KataGo, he identified weaknesses and misunderstanding, provided new positions around these issues, KataGo selfplayed these positions and eventually understood them! Until Thomas found new weaknesses .
- David not only created KataGo, but he also helped me a lot to setup the run, provide the initial panel of positions, adjust the parameters, debug some difficult technical issues and gave me precious advice all along
I used 2 RTX GPUs (3080+2070) during most of these 11-months of selfplay + training KataGo.

The best 40b network is derived from 40b-s5G09 (best net of g170's run, June 2020): this net was already very strong (only ~250 Elo weaker than current best kata1 40b network) and able to play many rule sets: Japanese, Chinese, Tromp-Taylor, New-Zealand...
You will find several details on this run in the attached pdf: process, intermediate results, final results, impact of komi and rule sets, etc.

The latest 40b network is linked in the file, and the direct link is: https://magentacloud.de/s/8i9BmTSKjf2ipCr
Igo Hatsuyoron 120's initial position is available as an sgf included in the pdf (in Annex A).

I hope you'll enjoy this marvelous problem, best regards!
Friday9i

PS: I just updated the file to correct a small detail (the first 50 downloads contain an incorrect date, no big deal)

This looks completely amazing, I just spent some time going over some of the SGF's in this thread and the problem has so many wonderful features, it's incredible. I hadn't seen this problem before, and it is (obviously) a thing of beauty indeed.

I would really like to try running this with the network you found! However, I seem to miss how I can use it in KataGo. I'm currently using Lizzie 0.7.2 with KataGo 1.2. Tt needs a katanetwork.gz file, which does not seem to work with the model.bin that you provide in the link. Is there a way to install it?

Thank you so much and thank you for sharing all your work into this wonderful problem!

To use a recent KataGo network, Lizzie is good but too old I'm afraid, but there is an alternative & up-to-date version called Lizzie Yzy. I don't use it but heard it is quite good, you can try it: https://github.com/yzyray/lizzieyzy/releases
Otherwise, Sabaki + KataGo works well, that's what I'm using generally (but it necessitates some installation).

gaius wrote:I would really like to try running this with the network you found! However, I seem to miss how I can use it in KataGo. I'm currently using Lizzie 0.7.2 with KataGo 1.2. Tt needs a katanetwork.gz file, which does not seem to work with the model.bin that you provide in the link. Is there a way to install it?

This link

https://magentacloud.de/s/oBPLmFf3i7L7pXg

will lead you to a PACKED version ("40bmodel.bin.gz") of Friday9i's latest 40-block IH120 network ("model.bin").

If Lizzie needs a "*.gz" file for the network, this one will probably help.
For no special reason, I had unpacked Friday9i's *.gz-file, used the original *.bin-file, and deleted the *.gz-file after unpacking.
I use KataGo with Sabaki, and for Sabaki a *.bin-file is also fine.

The 20-block network is a *.gz-file.

You will have to check the correspondence of the file names with the one used by Lizzie, and if necessary, you will have to edit one of the file-names.

+ + + + + + + + + + + + +

Alternatively, you could pack your already downloaded "model.bin" into a "model.bin.gz" file e.g. with using 7-zip (with format = "gzip").

gaius wrote:I would really like to try running this with the network you found! However, I seem to miss how I can use it in KataGo. I'm currently using Lizzie 0.7.2 with KataGo 1.2. Tt needs a katanetwork.gz file, which does not seem to work with the model.bin that you provide in the link. Is there a way to install it?

It's not Lizzie 0.7.2 problem. It's KataGo 1.2 problem. You should update KataGo.
My Lizzie 0.7.2 + KataGo 1.8 work fine with this net

is the solution of the problem Igo hatsuyoron # 120 described in this branch? I don't want to read in vain

Amigo wrote:is the solution of the problem Igo hatsuyoron # 120 described in this branch? I don't want to read in vain

If you are interested in a commented version of the 2022 solution to IH120, as created via Karl's KataGo 40b project, you may refer to

https://igohatsuyoron120.de/2015/2022MainLine.htm

Folks,

This is absolutely fantastic work, but given the many twists and turns it's gone through, it's very difficult to understand the very simple basics of this problem. Would you consider adding to https://igohatsuyoron120.de/2015/2022.htm a simple summary of the problem and it's latest state. The article ends with a baffling comparision to the previous best result, but does not clearly state the outcome.

This is my takeaway, but I'm very unclear if I've correctly understood:
- The problem is black to play
- The starting board position is reachable via alternating plays but only if white plays first. It is not possible to reach - this board position with black playing first and white capturing one stone (are we sure?)
- Assuming white played first, the final outcome is that black cannot win (white wins by +1) according to the latest main sequence. An unfortunately conclusion is that it's not a perfectly posed problem given the traditional framing of black to play and win. Is this different if we assume black played first and lost a stone along the way?

leftoftengen wrote:Folks,

This is absolutely fantastic work, but given the many twists and turns it's gone through, it's very difficult to understand the very simple basics of this problem. Would you consider adding to https://igohatsuyoron120.de/2015/2022.htm a simple summary of the problem and it's latest state. The article ends with a baffling comparision to the previous best result, but does not clearly state the outcome.

There is ABSOLUTELY NOTHING "simple" about this problem.
Even things that seem to be crystal clear are NOT. And I dare to doubt that we have identified ALL internal connections and dependencies so far.
(( KataGo 60b is quite busy digging up a few more right now. ))

The outcome, however, is clear: Assuming a captured Black stone during the setup of the problem, White wins by two points.

This is my takeaway, but I'm very unclear if I've correctly understood:
- The problem is black to play

Yes.

- The starting board position is reachable via alternating plays but only if white plays first. It is not possible to reach - this board position with black playing first and white capturing one stone (are we sure?)

Both ways are possible.

However, we think that the "ideal" solution to the problem ends with the smallest margin possible, and this is "White + 1".
In classical times (in China), starting the game with White was quite common (at least not unusual). So either there is a reference to the origin of the game included, or this issue was simply of no importance in Japan.

- Assuming white played first, the final outcome is that black cannot win (white wins by +1) according to the latest main sequence. An unfortunately conclusion is that it's not a perfectly posed problem given the traditional framing of black to play and win. Is this different if we assume black played first and lost a stone along the way?

Inoue Dôsetsu Inseki knew the thought processes of his professional colleagues down to the very last ramification (and also was aware of those of an AI appearing in over 300 years in the future).
In his epilogue, Dôsetsu explains that one can only become a true master of the game of Go if one goes beyond the limits of one's previous knowledge.

But wasn't the Go community extremely satisfied with "Black goes first and wins" in his masterpiece for several decades?
(( Fujisawa Hideyuki's study group did not choose the best order in occupying Black's liberties on the right in the Main Semeai, and incidentally negated Joachim's Oki after the capture of the Hanezeki's Tail. All issues that would result in a White win (leaving all the other vital spots aside). Can we be sure that this was NOT done on purpose? At that time, utilising neither my Guzumi nor Harry's Hasami-Tsuke nor my Second Throw-in in the Hanezeki (which were all identified by KataGo 60b quite early in the training process) were mandatory to succeed with Black. ))

There is ABSOLUTELY NOTHING "simple" about this problem.
Even things that seem to be crystal clear are NOT. And I dare to doubt that we have identified ALL internal connections and dependencies so far.

Thomas: I agree. And I feel sure you are the best person in the whole world to answer this question: How much of this problem did Dosetsu himself really understand?

I happen to believe that, as regards specifics, Dosetsu probably understood very little, and it would not surprise me in the least if even you understood more.

Inoue Dôsetsu Inseki knew the thought processes of his professional colleagues down to the very last ramification

However, I also believe in something else, which is similar to what you say there, but I would put it rather differently.

I start from the premise that Dosetsu was not specially interested in making problems that were hard or interesting for their own sake. He was not an entertainer but an educator. At least I think that's a fair inference from the very little of his work that survives, and it also explains Dosaku's faith in him to nurture Dochi.

I further infer (with a lot of confidence in this case) that he created problems to teach thought processes about tactics. That's really what the title of the book is about, after all.

When it came to CXX, my feeling is that he probably created a problem for study by his pupils which deliberately had something hard/instructive/teasing or even unfathomable at various junctures. At each of those junctures, I therefore suspect that even he did not know what the local outcome was - all he knew and cared about, I suggest, is that it was instructive. I could also easily imagine that he and his pupils pored over CXX in the same way you and your colleagues do, and that "B/W+1" or whatever was simply their best guess so far. Either that or he was an alien!

What's your view on all that?

The walk with HONTE was a little shorter than usual because we got into the 8% chance of rain
So I enjoy a cigar in parallel on the loggia ...

John Fairbairn wrote:

Inoue Dôsetsu Inseki knew the thought processes of his professional colleagues down to the very last ramification.

However, I also believe in something else, which is similar to what you say there, but I would put it rather differently.

I start from the premise that Dosetsu was not specially interested in making problems that were hard or interesting for their own sake. He was not an entertainer but an educator. At least I think that's a fair inference from the very little of his work that survives, and it also explains Dosaku's faith in him to nurture Dochi.

I further infer (with a lot of confidence in this case) that he created problems to teach thought processes about tactics. That's really what the title of the book is about, after all.

Dear John, I think we both mean the same thing about Dôsetsu, in principle.

If his Yôshin Teiki had not only survived the centuries in small parts, but had been preserved in its entirety, our speculations today would be based on a much more reliable foundation.
From the fragments that have survived, he seems to have been a fan of a systematic elaboration of a theme, which in my estimation is not typical of an East Asian mindset (but I have too little experience in this regard, you are the expert).

He must have been aware that this kind of systematic approach would not really help him with his social ambitions. It was probably even a hindrance (since it was "unusual" in Japan).
In relation to the game of Go itself, however, he perfected this mindset because he knew it was the right one.

Yes indeed, Dôsetsu must have been a fantastic educator, because he very obviously understood a lot about pedagogy. And in addition he knew all the strengths, weaknesses and blind spots of both the other professionals and his students.
This is what I meant above. Not the "though processes" themselves (taken literally), but their results.

In my opinion, his "never-be-sure-there-would-be-a-solution" problem XVII also fits well into the overall picture. For pedagogical reasons, one would demonstrate something like this on a relatively simple problem (and not on an extremely complex one), which is what he did with his "Black moves and dies" (which I do not think was included in his collection of problems by him in error or maliciously by others).

When it came to CXX, my feeling is that he probably created a problem for study by his pupils which deliberately had something hard/instructive/teasing or even unfathomable at various junctures.

Yes, I totally agree with you.

At each of those junctures, I therefore suspect that even he did not know what the local outcome was - all he knew and cared about, I suggest, is that it was instructive.

Due to my many years of intensive involvement with the problem, I unfortunately have to partially disagree with you here.

He knew that CXX was very instructive and not only knew the local results, but was also very aware of the global connections and interdependencies. There is no other explanation for the numerous variations that are decided by a single point or a single move only.
It may well be that he did not play through ALL (especially "narrow") variations that we know today down to the last detail. But then he must have known the laws / principles of all the individual forms and their global interplay, which he took with him to the grave. For all this cannot have been put together "purely by chance".

There is ABSOLUTELY NOTHING "simple" about this problem.
Even things that seem to be crystal clear are NOT. And I dare to doubt that we have identified ALL internal connections and dependencies so far.

Thomas: I agree. And I feel sure you are the best person in the whole world to answer this question: How much of this problem did Dosetsu himself really understand?

100,00%.

I happen to believe that, as regards specifics, Dosetsu probably understood very little, and it would not surprise me in the least if even you understood more.

This is too much of an honour, John.
It seems rather that, especially after drawing diagrams, a lot of (detailed) knowledge is erased from current memory or shifted to deeper regions. And I also need the intensive help of Karl's KataGo 40b-version to find out where my KataGo 60b-version in training goes wrong.

But in fact I have documented much more about the solution than Dôsetsu has.

Small addition:

In the old copy of Igo Hatsuyôron discovered by Araki Naomi in 1982, said to contain only the problems of the original version, there were TWO problems that were "unknown" until then, CXX and LXI.

If we take the number of pages of the commentary in Cheng Xiaoliu's 2010 book on Igo Hatsuyôron as a yardstick, LXI is one of the few extremely complex problems in the collection. The front-runner is, of course, CXX.

This leads me to believe that later generations after Dôsetsu basically expected the problems to have a solution, but found none for either LXI or CXX. And so these TWO problems were simply suppressed during further copying. Instead, problems from other historical problem collections were included.