It is currently Thu Jun 21, 2018 10:00 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next
Author Message
Offline
 Post subject: Patterns across various versions of LeelaZero training
Post #1 Posted: Sun Mar 11, 2018 12:54 pm 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
I was curious to see how an AlphaZero-like system discovers various Go patterns while training from self-play, from scratch.
So I did a little data-mining starting from the large SGF collection of self-play games from https://sjeng.org/zero/ (the all_sgf.xz file).
The version I did this analysis is about 3 weeks old now - the latest LZ version included in my analysis is af9ce63c from Feb 21st.
I skipped the first 1.5M games from that SGF collection to eliminate the initial random play, so the first version analyzed is 39fe0ce3 from Dec 28th, 2017.

I parsed the SGFs and extracted all the corner, 9x9 sized patterns containing exactly 10 stones (5 for each of white and black), and which appeared in at least 3 different LZ versions. Symmetries and color-swaps are taken into account, too.

I generated a report with top patterns from each version (versions are listed in chronological order). One can find the version IDs on the LZ project page here: http://zero.sjeng.org/

For each pattern, the plot next to it shows its frequency across the training versions. The x-axis numbers correspond to version index (from 0 to 45, since there are 46 versions between 39fe0ce3 and af9ce63c), while the y-axis numbers are the frequency of the pattern relative to other patterns in the same version, for instance 0.003 means this pattern appeared 3 times in each 1000 patterns in the games of a LZ version). Taller ticks mean that the pattern's relative frequency within a particular version is higher.

Each pattern is only shown in the earliest LZ version it occurred in - looking at the plot next to it should give an idea about popularity trend: if ticks for later versions are higher, the pattern became more popular in later versions, while if it is lower its popularity faded away.

I generated a report for the top one pattern in each version: http://361points.com/media/top1patterns/patterns.html
The page is very slow to load the patterns, it will show first some "....xxxooo..." text before showing image patterns.
Interestingly, we can see first the old-style 3x3 invasion against the opponent's 4x4 stone, with hane-tsugi in the end; LZ did this in older versions, but we see from the frequency plot how it abandoned it eventually.
Image

Second pattern is the AlphaGo style of crawling one more time on 2nd line instead, to get sente; LZ discovered this later, but it seem to have abandoned it too; I bet it will show again in the most recent versions (after the switch to the 10-blocks, on March 5th), I will redo the analysis to incorporate more recent versions.
Image

Third pattern is the double-hand and capture, which we see from the plot that it became super-popular in the 2nd half of the timeline. I expect that as LZ gets even stronger, it will abandon this in favor of the 2nd pattern, above.
Image

For people interested in more data: I also generated a report for the top-10 patterns in each version: http://361points.com/media/top10patterns/patterns.html
This larger analysis may take several minutes to load - sorry about that, I hope I can figure out a better way.

_________________
Sorin - 361points.com


This post by sorin was liked by 12 people: Bill Spight, Caesura, Calvin Clark, gamesorry, Gomoto, gustav, jeromie, johnsmith, Kap, Marcel Grünauer, Uberdude, Waylon
Top
 Profile  
 
Offline
 Post subject:
Post #2 Posted: Sun Mar 11, 2018 4:08 pm 
Judan
User avatar

Posts: 7968
Location: Santa Barbara, CA
Liked others: 302
Was liked: 1795
GD Posts: 312
Hi Sorin,

Thanks.

Which software did you use to generate the board image ?

Top
 Profile  
 
Offline
 Post subject: Re:
Post #3 Posted: Sun Mar 11, 2018 6:26 pm 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
EdLee wrote:
Hi Sorin,

Thanks.

Which software did you use to generate the board image ?


Hi Ed,

I used jgoboard 3.0 from here:
https://github.com/jokkebk/jgoboard

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #4 Posted: Sun Mar 11, 2018 11:55 pm 
Tengen

Posts: 4846
Location: Cambridge, UK
Liked others: 241
Was liked: 2451
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Very interesting, thanks for your efforts in making this. We can see it discovering quite a lot of AlphaGo style moves, e.g. in the top 10 view near the end I spied the attach inside big high shimari after ignored knight's approach.

One suggestion: can you include a rank estimate or Elo rating along with network name? Doesn't need to be super accurate and of course depends on hardware, but would be nice to have an idea if the network half way through is nearer 8 kyu or 1 dan.

I expect the folks over at reddit/cbaduk would like this: would you like to post it there or can I? Is the image rendering server or client side, wouldn't want to overload your web server.


This post by Uberdude was liked by: sorin
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #5 Posted: Mon Mar 12, 2018 8:18 am 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
Uberdude wrote:
Very interesting, thanks for your efforts in making this. We can see it discovering quite a lot of AlphaGo style moves, e.g. in the top 10 view near the end I spied the attach inside big high shimari after ignored knight's approach.

One suggestion: can you include a rank estimate or Elo rating along with network name? Doesn't need to be super accurate and of course depends on hardware, but would be nice to have an idea if the network half way through is nearer 8 kyu or 1 dan.

I expect the folks over at reddit/cbaduk would like this: would you like to post it there or can I? Is the image rendering server or client side, wouldn't want to overload your web server.


I was also very surprised to see that it follows pretty much all the peculiar patterns of AG. I wonder now if these are "fundamental truths" in Go (that humans just missed to discover), or just artifacts of the particular training algorithm used.

Good idea about attaching Elo to versions, I will do that.

No concern with reposting, rendering is client-side, thanks for your concern!

_________________
Sorin - 361points.com

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #6 Posted: Mon Mar 12, 2018 9:01 am 
Judan

Posts: 7166
Liked others: 1805
Was liked: 2603
sorin wrote:
I was also very surprised to see that it follows pretty much all the peculiar patterns of AG. I wonder now if these are "fundamental truths" in Go (that humans just missed to discover), or just artifacts of the particular training algorithm used.


Oh, I expect that there is path dependency, both in bots' training regimes and in human baduk history. For instance, if unfinished games early on are scored by proximity scoring, the bot might start off playing an influence oriented game. Scoring unfinished games by number of prisoners might produce more aggressive play, quien sabe?

For the 3-3 invasion patterns that you show in your first note, humans overlooked the crawl on the second line (second pattern) instead of the hane (first pattern). As a result of that, perhaps, we felt that the 3-3 invasion was somewhat dubious. Now, AlphaGo, in all varieties, likes it.

But what do we make of Black's two step hane, preventing White from crawling? That pattern seems to be superseding the second pattern. Also, it is a human joseki. Could it be that that pattern is so good for Black that in later development Leela Zero will reject the early 3-3 invasion? Nobody knows. But it looks like Leela Zero took a detour before returning to a human joseki, while humans never quite rejected the first joseki.

Anyway, a great study! :D Many thanks.

Edit: Corrected to refer to the right bot. :oops:

_________________
The Adkins Principle:

At some point, doesn't thinking have to go on?

— Winona Adkins


Last edited by Bill Spight on Mon Mar 12, 2018 2:23 pm, edited 1 time in total.

This post by Bill Spight was liked by: sorin
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #7 Posted: Mon Mar 12, 2018 11:49 am 
Beginner

Posts: 4
Liked others: 0
Was liked: 1
Rank: Eternal KGS 4d
KGS: redreoicy
Online playing schedule: T-f
I'm nearly certain the second pattern disappeared because it was too good for the 3-3 side, and won't be coming back. At least, it's not likely to become more popular than the third pattern.


This post by redreoicy was liked by: sorin
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #8 Posted: Mon Mar 12, 2018 1:18 pm 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
redreoicy wrote:
I'm nearly certain the second pattern disappeared because it was too good for the 3-3 side, and won't be coming back. At least, it's not likely to become more popular than the third pattern.


Looking at AG games, it very rarely plays pattern 3 (for the 4x4 player). By that measure, I think pattern 3 may fade away as LZ gets stronger, and pattern 2 re-emerge... Time will tell (soon :-) ).

_________________
Sorin - 361points.com

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #9 Posted: Mon Mar 12, 2018 8:21 pm 
Beginner

Posts: 4
Liked others: 0
Was liked: 1
Rank: Eternal KGS 4d
KGS: redreoicy
Online playing schedule: T-f
sorin wrote:
redreoicy wrote:
I'm nearly certain the second pattern disappeared because it was too good for the 3-3 side, and won't be coming back. At least, it's not likely to become more popular than the third pattern.


Looking at AG games, it very rarely plays pattern 3 (for the 4x4 player). By that measure, I think pattern 3 may fade away as LZ gets stronger, and pattern 2 re-emerge... Time will tell (soon :-) ).

Ok, but I didn't find a single instance of pattern 2 in all the released games by alphagozero.

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #10 Posted: Mon Mar 12, 2018 11:12 pm 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
redreoicy wrote:
sorin wrote:
redreoicy wrote:
I'm nearly certain the second pattern disappeared because it was too good for the 3-3 side, and won't be coming back. At least, it's not likely to become more popular than the third pattern.


Looking at AG games, it very rarely plays pattern 3 (for the 4x4 player). By that measure, I think pattern 3 may fade away as LZ gets stronger, and pattern 2 re-emerge... Time will tell (soon :-) ).

Ok, but I didn't find a single instance of pattern 2 in all the released games by alphagozero.


I see, you are right!! I did a quick check here http://www.alphago-games.com/#agzero_vs_agmaster where one can see this pattern in many of the game pics, but then I realized that it is always AG Master playing that shape. Very interesting, I see - I wonder if it is possible that LZ has already discovered something that AG Master doesn't know!

_________________
Sorin - 361points.com

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #11 Posted: Sun Apr 01, 2018 12:36 pm 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
I did some more analysis of LeelaZero patterns, using the latest available collection of training SGFs (up to version 1ccb7342 which is a little more recent and stronger than the version that defeated Hajin Lee 4p).

Also, this time I made the analysis more focused and hopefully more interesting, by picking relatively few patterns to analyze and compare.
Here is what I have so far:


See below for some patterns not included in the above analysis, but nevertheless interesting, from analyzing up to version 1ccb7342.
Timeline graphs show relative frequency of the pattern in various training versions (older versions on the left, newer versions on the right). The red line highlights where the currently analyzed version is in the timeline.

Timeline for what I call "Go Seigen pattern" (possibly misnomer) - we see that this became more popular with LeelaZero relatively recently (but it is still a rare pattern overall, appears less than twice in ten thousand patterns on the average)
Attachment:
go_seigen_joseki.PNG
go_seigen_joseki.PNG [ 135.98 KiB | Viewed 2290 times ]


AlphaGo-like two-space extension by black (maybe to prevent the peep by white, if black plays just one-space jump, like AlphaGo played against Lee Sedol in game 2)
Attachment:
alphago_joseki2.PNG
alphago_joseki2.PNG [ 137.81 KiB | Viewed 2290 times ]


Timeline for another AlphaGo move, the double-hane at S6 (instead of the human move at Q4):
Attachment:
alphago_joseki3.PNG
alphago_joseki3.PNG [ 136.92 KiB | Viewed 2290 times ]

_________________
Sorin - 361points.com


This post by sorin was liked by 6 people: Bill Spight, Gomoto, gustav, jeromie, Uberdude, Waylon
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #12 Posted: Sun Apr 01, 2018 2:01 pm 
Judan

Posts: 7166
Liked others: 1805
Was liked: 2603
Wonderful stuff! Many thanks. :D

BTW, I think that the two space high approach originated with Honinbo Shuei. In his 21st Century Go set, Go Seigen likes it as :w2: if Black starts with a 3-4 point. :)

_________________
The Adkins Principle:

At some point, doesn't thinking have to go on?

— Winona Adkins


This post by Bill Spight was liked by: sorin
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #13 Posted: Mon Apr 02, 2018 3:30 am 
Oza

Posts: 2052
Liked others: 12
Was liked: 3004
Quote:
BTW, I think that the two space high approach originated with Honinbo Shuei. In his 21st Century Go set, Go Seigen likes it as :w2: if Black starts with a 3-4 point.


This has long been the opening that most fascinates me, for various reasons, some of which seem applicable to discussion of AI.



It appeared before Shuei, but in more of a middle-game context than pure fuseki, but what was noticeable about its adherents was not so much that Shuei was among them (he played it when very young and immature then abandoned it for a long while) but that it was taken up by Shuho and Shusai in a big way. It was also associated with Honinbo pupils. It seems therefore it must have been heavily discussed among the Honinbo family. I am not aware of any record of what they might have said, regrettably, but we can probably infer some things from their games, and in particular how they responded to it. For example, the common defence at A. Even today, whenever I see it, I marvel at this move being played by pros. It seems so counter-intuitive to me. It is down to Honinbo Shugen.

The analogous B (rather than the usual C) took a long time coming, and remains hyper-rare, but Takahashi Shigeyuki (Shusai's son-in-law) tried it in the 1930s, including in this amusing opening:



Another aspect of my fascination are the continuations at D and E. E had a try-out in the 1930s Shin Fuseki period and was promptly dropped (perhaps it was seen as helping White play towards the centre?) but D has had a reasonable run and has thrown up several unusual josekis. This looks very Shin Fuseki-ish but was introduced by Onoda Chiyotaro in the 1920s. However, as I have pointed out in "Old Fuseki vs New Fuseki" Shin Fuseki was part of an organic process starting in the Meiji period (and so this high approach was part of that). Shin Fuseki had an explosive hype moment in 1933 but was not really new (shin).

This opening also has some fascinating pincer josekis. The extra gap changes the dynamics of the pincer in several instructive ways.

The merits of the high approach are such that several major players have tried it not as an approach but as first move in the corner (or even as first move in the game). Sekiyama Riichi tried it first in the 1930s and was soon followed by the likes of Kitani and Go Seigen. It hasn't quite disappeared and I was amused to see Sekiyama Toshimichi (Riichi's grandson) play it again last year (2017) in a couple of games. The Sekiyama link is actually even stronger than it looks, as the first recorded instance of the high approach is by Sekiyama Sendayu in 1824.

At a more strategic level, the high approach has been discussed by Go Seigen, of course, but even more extensively by O Rissei - in both cases as part of the 21st Century Go movement.


PS: The following quote from the original post source needs correcting:

Quote:
The official Japanese name is "nikken-takagakari" because it is a two-point jump (nikken), it is high ("taka(i)") (4th line instead of 3rd), and is an approach ("kakari" - turns into "gakari" just because it sounds better in the context).


There is nothing official about it, and there's rather more to lenition than just sounding better (see rendaku and Lyman's Law if you're interested), but the word nikken does not exist. It is niken, and it means two-space not two-space jump (three-space is sangen BTW, and one-space does indeed have -kk-: ikken). While we are on this topic, I'll mention that kakaru doesn't mean approach, although I think it's acceptable. But the usage in Japan is not quite agreed either. Some explain it as attack since e.g. teki ni kakaru means attack (fall on) the enemy. I think this is the commonest explanation, but others say it means getting a handhold (i.e. relating it to the word tegakari). But fuzzyists like me think it has both connotations.

PPS: By chance the first game I started transcribing after writing the above featured the two-space approach, with an interesting comment by Masubuchi Tatsuko (Black):



Masubuchi said she ignored White 4 rather than answering as usual at Q14 because that would mean falling in with White's plans (so maybe a feature of the high-approach is that it is more highly strategy-oriented than other approaches?) and played 5 as a kind of probe (yes, a probe this early - not that rare, actually). However, she seemed to be a little dissatisfied with the result as she said that White had a achieved an efficient position after 10.


This post by John Fairbairn was liked by 4 people: Charlie, Gomoto, gowan, sorin
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #14 Posted: Mon Apr 02, 2018 7:46 pm 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
John Fairbairn wrote:
This has long been the opening that most fascinates me, for various reasons, some of which seem applicable to discussion of AI.



It appeared before Shuei, but in more of a middle-game context than pure fuseki, but what was noticeable about its adherents was not so much that Shuei was among them (he played it when very young and immature then abandoned it for a long while) but that it was taken up by Shuho and Shusai in a big way. It was also associated with Honinbo pupils. It seems therefore it must have been heavily discussed among the Honinbo family. I am not aware of any record of what they might have said, regrettably, but we can probably infer some things from their games, and in particular how they responded to it. For example, the common defence at A. Even today, whenever I see it, I marvel at this move being played by pros. It seems so counter-intuitive to me. It is down to Honinbo Shugen.


Thank you for your detailed post John, as usual lots of very interesting Go culture information!

I asked LeelaZero's opinion on Takahashi Shigeyuki's game after white's "niken takagakari" (thank you for the spelling correction also!) in the lower-left, and its top local choices are: first E3, second D3, third hasami at H3 - so the counter-intuitive looking kosumi (A in your quoted diagram) looks natural to AI for some reason!

_________________
Sorin - 361points.com


This post by sorin was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #15 Posted: Thu Apr 05, 2018 3:26 pm 
Tengen

Posts: 4846
Location: Cambridge, UK
Liked others: 241
Was liked: 2451
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Insightful analysis again Sorin, thanks!

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #16 Posted: Fri Apr 27, 2018 6:47 am 
Dies in gote

Posts: 34
Liked others: 0
Was liked: 10
Quote:
The last example in this quick overview is the pattern that is #1000 in global popularity: invasion at 1x1 against the opponent's 4x4! Interestingly, this still comes up in relatively recent versions, every now and then. That happens very rarely (about once in 250K patterns, and is mostly likely due to the artificial randomization for the fist 30 moves in self-play training games) but it is not "dead" yet in recent versions - unlike the pattern above - so it looks like LeelaZero still has some hopes for this variation.


This has been remarked several times, but maybe it's worth repeating here: the first line contains more go board squares than any other, so there's proportionally more chance of playing a first line move if a move is picked nearly randomly. That's why this is still appearing in training (but no match) games, and also why it has not died out, unlike some other patterns, whose "luck" odds just worse, i.e. it is the most-symmetries gathering first-line response to the most popular pattern.

Excellent work BTW. Can't wait to see the next update.


This post by Garf was liked by: sorin
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #17 Posted: Sun Apr 29, 2018 9:37 am 
Lives with ko

Posts: 127
Liked others: 95
Was liked: 79
Garf wrote:
This has been remarked several times, but maybe it's worth repeating here: the first line contains more go board squares than any other, so there's proportionally more chance of playing a first line move if a move is picked nearly randomly. That's why this is still appearing in training (but no match) games, and also why it has not died out, unlike some other patterns, whose "luck" odds just worse, i.e. it is the most-symmetries gathering first-line response to the most popular pattern.
Excellent work BTW. Can't wait to see the next update.


Your explanation makes sense indeed, I didn't think about this argument before!

I re-did my analysis, but now I realize that I don't know exactly what to publish.
Basically I have the following data:
  • "pattern view": top-1000 most popular patterns (across all versions), each with a popularity timeline graph
  • "version view": version-by-version (in chronological order), top-10 (or so) patterns that are most popular in that given version and have not been top-10 in any older version (to prevent duplicates)

I am looking at this data but it just doesn't seem to be very interesting to publish as an update to the previous ones, it seems to be "too much data" and too little differences from previous data.
Any suggestions for what would be interesting to show, better than the two "tons of patterns and timelines" sets I described above?

_________________
Sorin - 361points.com

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #18 Posted: Sun Apr 29, 2018 9:02 pm 
Gosei
User avatar

Posts: 2013
Location: Tokyo, Japan
Liked others: 1793
Was liked: 1115
Rank: Jp 6 dan
KGS: ez4u
Garf wrote:
...

Excellent work BTW. Can't wait to see the next update.

I second both statements here! :clap: :bow:

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #19 Posted: Mon Apr 30, 2018 7:36 am 
Dies in gote

Posts: 34
Liked others: 0
Was liked: 10
For me the most interesting to see is how the bot likes (or dislikes) the normal human moves over time, and the same for the moves from the Alpha Go Zero games. Where has it started rediscovering theory, and where is it inventing new things...


This post by Garf was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: Patterns across various versions of LeelaZero training
Post #20 Posted: Tue May 22, 2018 7:19 am 
Tengen

Posts: 4846
Location: Cambridge, UK
Liked others: 241
Was liked: 2451
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
sorin, someone has noticed that LZ seems to have gone off the early 3-3 invasions somewhat: https://www.reddit.com/r/cbaduk/comment ... _in_leela/.

Give it a week or two for random fluctuations to even out and then re-running your analysis on 3-3 invasion and related patterns would be interesting.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot] and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group