It is currently Wed Jun 19, 2019 9:52 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 17 posts ] 
Author Message
Offline
 Post subject: How strong is Leela Zero
Post #1 Posted: Thu Jan 03, 2019 8:35 am 
Dies in gote

Posts: 23
Liked others: 7
Was liked: 3
A question - in the current Elo scale used in leela zero Strength Graph, where would human professional be? Can someone give a rough range? How many dan rank is LZ stronger than human now?

Top
 Profile  
 
Offline
 Post subject:
Post #2 Posted: Thu Jan 03, 2019 2:45 pm 
Honinbo
User avatar

Posts: 8511
Location: Santa Barbara, CA
Liked others: 322
Was liked: 1964
GD Posts: 312
Hi chut,

Suppose we have anecdotal evidence that the current best engine (which may not be LZ) is roughly 2 or 3 stones better than top human pros, how does that affect your life ? ( What would you do about it ? )
Same question: for ~5 stones.
Curious about the human feelings behind the question.
How much faster is the current fastest car than the fastest running human ?
For some, it really matters a car reaches speed X in N seconds.

Top
 Profile  
 
Offline
 Post subject: Re:
Post #3 Posted: Thu Jan 03, 2019 9:34 pm 
Dies in gote

Posts: 23
Liked others: 7
Was liked: 3
EdLee wrote:
Hi chut,

Suppose we have anecdotal evidence that the current best engine (which may not be LZ) is roughly 2 or 3 stones better than top human pros, how does that affect your life ? ( What would you do about it ? )
Same question: for ~5 stones.
Curious about the human feelings behind the question.
How much faster is the current fastest car than the fastest running human ?
For some, it really matters a car reaches speed X in N seconds.

I am primarily a computer scientist, and being a go player is only for casual amusements. My interest would be - to use the car analogy - what else can be done to make the car go faster, and more fuel efficient.

I would like to see the different engines competing against each other, to see analysis of how different strategies lead to better/worst performance. And having some milestone markers - human, Alphago Fan, Alphago Lee etc - will make comprehension a lot easier for my feeble mind.

I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly. Is that due to a non-linear scaling used in AlphaGo curve?

Top
 Profile  
 
Offline
 Post subject: Re: Re:
Post #4 Posted: Thu Jan 03, 2019 10:21 pm 
Lives in gote

Posts: 330
Liked others: 271
Was liked: 168
chut wrote:
I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly.


My guess is that LZ is still far-far away from AGZ. In terms of steepness of the progress curve, I guess it depends how you plot it, but I think the reason why LZ may still improve relatively fast is because it has a lot more room to grow, being weaker.

I am also fascinated by your original question, by the way, I would like very much to see more pros playing experimental games with handicap against LZ to measure the gap.

_________________
Sorin - 361points.com

Top
 Profile  
 
Online
 Post subject: Re: Re:
Post #5 Posted: Thu Jan 03, 2019 11:04 pm 
Honinbo

Posts: 8492
Liked others: 2483
Was liked: 2946
sorin wrote:
chut wrote:
I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly.


My guess is that LZ is still far-far away from AGZ. In terms of steepness of the progress curve, I guess it depends how you plot it, but I think the reason why LZ may still improve relatively fast is because it has a lot more room to grow, being weaker.


I agree. I think with AlphaGo Zero we are seeing the law of diminishing returns, while Leela Zero isn't yet strong enough to see that.

Quote:
I am also fascinated by your original question, by the way, I would like very much to see more pros playing experimental games with handicap against LZ to measure the gap.


Me, too.

_________________
There is one human race.
----------------------------------------------------

The Adkins Principle:

At some point, doesn't thinking have to go on?

— Winona Adkins

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #6 Posted: Fri Jan 04, 2019 2:29 am 
Lives in gote
User avatar

Posts: 429
Liked others: 39
Was liked: 135
chut wrote:
A question - in the current Elo scale used in leela zero Strength Graph, where would human professional be? Can someone give a rough range? How many dan rank is LZ stronger than human now?


The strength depends on the hardware and on thinking time, but from the thread "LeelaZero adventures on Fox", and from petgo3's rank on KGS, I guess that on medium hardware, and with relatively fast time settings, LZ is about professional strength but not top pro.

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #7 Posted: Fri Jan 04, 2019 2:58 am 
Judan

Posts: 5880
Location: Cambridge, UK
Liked others: 335
Was liked: 3140
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
The only game I played as LZ against a known pro on Fox was here. It was a low dan Chinese pro Li Jianyu who I think is this guy: https://www.goratings.org/en/players/1730.html, goratings #441 with 3079 elo (very top pros being 3600+). LZ 191 with a few thousand playouts (probably a 20 or 30 second byoyomi with my GeForce 1060 GPU) beat him with ease, the winrate was a one-way and rapid decline. How much better would a top pro do?

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #8 Posted: Sun Jan 06, 2019 5:14 am 
Judan

Posts: 5880
Location: Cambridge, UK
Liked others: 335
Was liked: 3140
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Here's a game from WBaduk, an account called leelazero (black) vs Ichiriki Ryo 8p (goratings #43). I don't know the hardware / time settings, but LZ pretty much crushed him in a practically one-way victory much like my LZ did against a lower grade pro (when it didn't mess up a ladder!).



Winrate grpah from my running LZ #198

Attachment:
LZ Ichirki winrate 198.PNG
LZ Ichirki winrate 198.PNG [ 161.26 KiB | Viewed 1926 times ]


Attachments:
LZ vs Ichirki.sgf [11.62 KiB]
Downloaded 1066 times
Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #9 Posted: Mon Jan 07, 2019 4:21 am 
Gosei
User avatar

Posts: 2125
Location: Tokyo, Japan
Liked others: 1949
Was liked: 1202
Rank: Jp 6 dan
KGS: ez4u
You might be interested in this link: https://github.com/breakwa11/GoAIRatings#go-ai-ratings

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21


This post by ez4u was liked by: chut
Top
 Profile  
 
Offline
 Post subject: Re: LZ's Progress
Post #10 Posted: Mon Jan 07, 2019 8:24 pm 
Gosei
User avatar

Posts: 2125
Location: Tokyo, Japan
Liked others: 1949
Was liked: 1202
Rank: Jp 6 dan
KGS: ez4u
chut wrote:
...
I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly. Is that due to a non-linear scaling used in AlphaGo curve?

I was thinking about this comment and so was playing a little with the LZ historical data. Below are two graphs that I made just to emphasize the change in net size over the life of the LZ project to date.

LZ Elo ratings versus cumulative games (in millions), showing the different net sizes. LZ is still far away from the 30 million or so games used by AG. In addition, the project has only recently moved to the 40x256 block size that is (or at least was originally intended as) the final target.
Attachment:
LZ Elo vs Self-Play Games by Net Size.jpg
LZ Elo vs Self-Play Games by Net Size.jpg [ 81.29 KiB | Viewed 1710 times ]

LZ Elo ratings development over time, showing the different net sizes. Comparing this to the first graph gives us some impression of the effect of the LZ project's strategy of starting with small nets for the early learning and then moving to larger nets as things went along.
Attachment:
LZ Elo Over Time by Net Size.jpg
LZ Elo Over Time by Net Size.jpg [ 77.74 KiB | Viewed 1710 times ]

So far I can't see anything profound here but as long-time readers must realize by now, I like pretty pictures! As always, YMMV.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21


This post by ez4u was liked by 2 people: chut, Waylon
Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #11 Posted: Tue Jan 08, 2019 3:32 am 
Judan

Posts: 5880
Location: Cambridge, UK
Liked others: 335
Was liked: 3140
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
ez4u, I think it would be a good idea to add the rating of LZElf (v0 and/or v1) as calculated in matches versus LZ to those graphs. If the LZ self-play ratings were non inflationary this would be a flat line as it's the same fixed-strength player. However, it actually increases so gives some idea of the background expansion of the inflationary Leelo (Leela Elo) universe against which LZ's progress is measured. (The test matches are at fixed playouts so Elf will stay the same strength, but as LZ moves to more blocks it will get more time than Elf so you expect it to do better, I think time parity would be fairer so this method will under-report the inflation).
viewtopic.php?p=235675#p235675

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #12 Posted: Tue Jan 08, 2019 3:58 am 
Gosei
User avatar

Posts: 2125
Location: Tokyo, Japan
Liked others: 1949
Was liked: 1202
Rank: Jp 6 dan
KGS: ez4u
Uberdude wrote:
ez4u, I think it would be a good idea to add the rating of LZElf (v0 and/or v1) as calculated in matches versus LZ to those graphs. If the LZ self-play ratings were non inflationary this would be a flat line as it's the same fixed-strength player. However, it actually increases so gives some idea of the background expansion of the inflationary Leelo (Leela Elo) universe against which LZ's progress is measured. (The test matches are at fixed playouts so Elf will stay the same strength, but as LZ moves to more blocks it will get more time than Elf so you expect it to do better, I think time parity would be fairer so this method will under-report the inflation).
https://www.lifein19x19.com/viewtopic.p ... 75#p235675

My graphs are just the info in the 'best net' table on the LZ homepage. I did nothing but break out the net size with different colors. Feel free to improve on them.

On time parity, somewhere in the discussions on github people commented on the fact that different hardware will yield time parity for different levels of visits. So how do we decide what time parity is? Personally I don't know what you mean by 'fair' when you are looking at different generations of the same program. Aren't you going a little bit too far into anthropomorphism? Is 157 going to have its feelings hurt if we keep claiming that 199 is 'better'? ;-)

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #13 Posted: Mon Jan 14, 2019 3:41 am 
Dies in gote

Posts: 23
Liked others: 7
Was liked: 3
It is good to know that LZ is 15 Dan presently. It is very impressive.
What elo number would corresponds to human 9 Dan strength?
I am curious at who point LZ left human behind, and the number of self-played games to get to that point.

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #14 Posted: Mon Jan 14, 2019 4:38 am 
Judan

Posts: 5880
Location: Cambridge, UK
Liked others: 335
Was liked: 3140
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Saying LZ is 15 dan is pretty meaningless IMO, especially if you aren't clear what 1 dan difference means (is it give 1 stone handicap or a winrate difference?). What I can say is watching LZ play Japanese pros on wbaduk LZ always wins, sometimes forcing a resign before move 100, and that's not just against lower strength pros but also against Shibano Toramaru (who is goratings #68, though maybe he is fooling around a bit in these practice games). But then my LZ messed up a ladder on Fox and lost to a 3p down at goratings #600 (fastish game on moderate hardware but it made the same mistake even with 30k+ playouts).

chut wrote:
I am curious at who point LZ left human behind, and the number of self-played games to get to that point.

In the match with Haylee back around May last year LZ network 136 or thereabouts (7.2 million self-play games) games beat her with ease on even and 2 stones but lost on 3. Haylee is a lot weaker than a top pro like Ke Jie and I'm not aware of any serious matches between players of his calibre and LZ to test when LZ passed top humans. You could trawl through the game records of a player called something like LeelaZer15 on Fox who I believe uses a 15-block network (157 I guess, maybe switched to bijxo super-trained) on a 1080 GPU (I briefly chatted with him) to see if he's played any top pros by cross-referencing with Professional Player Usernames on Fox Go Server. That account usually plays 2 handi plus komi games against Fox 9ds (a wide range of strength) and mostly wins.

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #15 Posted: Mon Jan 14, 2019 6:14 am 
Dies in gote

Posts: 23
Liked others: 7
Was liked: 3
Uberdude wrote:
Saying LZ is 15 dan is pretty meaningless IMO, especially if you aren't clear what 1 dan difference means (is it give 1 stone handicap or a winrate difference?). What I can say is watching LZ play Japanese pros on wbaduk LZ always wins, sometimes forcing a resign before move 100, and that's not just against lower strength pros but also against Shibano Toramaru (who is goratings #68, though maybe he is fooling around a bit in these practice games). But then my LZ messed up a ladder on Fox and lost to a 3p down at goratings #600 (fastish game on moderate hardware but it made the same mistake even with 30k+ playouts).

I stone handicap per rank difference only applies to amateur ranking. The strength difference for professional is much closer, if I am not mistaken the difference between 1p and 9p is at most two stones handicap.

In some systems 1 rank difference = 100 points elo difference. But different ranking systems has different scaling and different winning probability between the ranks.

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #16 Posted: Mon Jan 14, 2019 6:24 am 
Lives in gote
User avatar

Posts: 429
Liked others: 39
Was liked: 135
Uberdude wrote:
watching LZ play Japanese pros on wbaduk LZ always wins


Do you have an estimate of the number of visits at each move?

Top
 Profile  
 
Offline
 Post subject: Re: How strong is Leela Zero
Post #17 Posted: Mon Jan 14, 2019 6:35 am 
Judan

Posts: 5880
Location: Cambridge, UK
Liked others: 335
Was liked: 3140
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
jlt wrote:
Uberdude wrote:
watching LZ play Japanese pros on wbaduk LZ always wins


Do you have an estimate of the number of visits at each move?


No idea. Maybe it would be possible to estimate this by guessing a network you think they are using (e.g. latest best 40 block) and then seeing if the game LZ ends up playing moves which only rise to become the first choice of your locally run LZ after some number of playouts e.g. I run LZ and it wants to play move A until 20k but then after 20k move B is number 1 (and let's assume run for a further 100k and B stays number 1), then if game LZ played B you have some evidence visits was over 20k but no ceiling. If after 150k local testing the number 1 move switched to C then you have evidence game playouts was also <150k. Just based on random enthusiasts having a 1080 Ti or two I'd expect the Wbaduk LZ to be at least something like that so probably several 10s of thousands playouts a move, but that's just a guess. Note I think these games are actually played on the Japanese 'Yugen No Ma' server and wbaduk has some relay thingy.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group