Life In 19x19
http://lifein19x19.com/

How strong is Leela Zero
http://lifein19x19.com/viewtopic.php?f=18&t=16343
Page 1 of 1

Author:  chut [ Thu Jan 03, 2019 8:35 am ]
Post subject:  How strong is Leela Zero

A question - in the current Elo scale used in leela zero Strength Graph, where would human professional be? Can someone give a rough range? How many dan rank is LZ stronger than human now?

Author:  EdLee [ Thu Jan 03, 2019 2:45 pm ]
Post subject: 

Hi chut,

Suppose we have anecdotal evidence that the current best engine (which may not be LZ) is roughly 2 or 3 stones better than top human pros, how does that affect your life ? ( What would you do about it ? )
Same question: for ~5 stones.
Curious about the human feelings behind the question.
How much faster is the current fastest car than the fastest running human ?
For some, it really matters a car reaches speed X in N seconds.

Author:  chut [ Thu Jan 03, 2019 9:34 pm ]
Post subject:  Re:

EdLee wrote:
Hi chut,

Suppose we have anecdotal evidence that the current best engine (which may not be LZ) is roughly 2 or 3 stones better than top human pros, how does that affect your life ? ( What would you do about it ? )
Same question: for ~5 stones.
Curious about the human feelings behind the question.
How much faster is the current fastest car than the fastest running human ?
For some, it really matters a car reaches speed X in N seconds.

I am primarily a computer scientist, and being a go player is only for casual amusements. My interest would be - to use the car analogy - what else can be done to make the car go faster, and more fuel efficient.

I would like to see the different engines competing against each other, to see analysis of how different strategies lead to better/worst performance. And having some milestone markers - human, Alphago Fan, Alphago Lee etc - will make comprehension a lot easier for my feeble mind.

I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly. Is that due to a non-linear scaling used in AlphaGo curve?

Author:  sorin [ Thu Jan 03, 2019 10:21 pm ]
Post subject:  Re: Re:

chut wrote:
I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly.


My guess is that LZ is still far-far away from AGZ. In terms of steepness of the progress curve, I guess it depends how you plot it, but I think the reason why LZ may still improve relatively fast is because it has a lot more room to grow, being weaker.

I am also fascinated by your original question, by the way, I would like very much to see more pros playing experimental games with handicap against LZ to measure the gap.

Author:  Bill Spight [ Thu Jan 03, 2019 11:04 pm ]
Post subject:  Re: Re:

sorin wrote:
chut wrote:
I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly.


My guess is that LZ is still far-far away from AGZ. In terms of steepness of the progress curve, I guess it depends how you plot it, but I think the reason why LZ may still improve relatively fast is because it has a lot more room to grow, being weaker.


I agree. I think with AlphaGo Zero we are seeing the law of diminishing returns, while Leela Zero isn't yet strong enough to see that.

Quote:
I am also fascinated by your original question, by the way, I would like very much to see more pros playing experimental games with handicap against LZ to measure the gap.


Me, too.

Author:  jlt [ Fri Jan 04, 2019 2:29 am ]
Post subject:  Re: How strong is Leela Zero

chut wrote:
A question - in the current Elo scale used in leela zero Strength Graph, where would human professional be? Can someone give a rough range? How many dan rank is LZ stronger than human now?


The strength depends on the hardware and on thinking time, but from the thread "LeelaZero adventures on Fox", and from petgo3's rank on KGS, I guess that on medium hardware, and with relatively fast time settings, LZ is about professional strength but not top pro.

Author:  Uberdude [ Fri Jan 04, 2019 2:58 am ]
Post subject:  Re: How strong is Leela Zero

The only game I played as LZ against a known pro on Fox was here. It was a low dan Chinese pro Li Jianyu who I think is this guy: https://www.goratings.org/en/players/1730.html, goratings #441 with 3079 elo (very top pros being 3600+). LZ 191 with a few thousand playouts (probably a 20 or 30 second byoyomi with my GeForce 1060 GPU) beat him with ease, the winrate was a one-way and rapid decline. How much better would a top pro do?

Author:  Uberdude [ Sun Jan 06, 2019 5:14 am ]
Post subject:  Re: How strong is Leela Zero

Here's a game from WBaduk, an account called leelazero (black) vs Ichiriki Ryo 8p (goratings #43). I don't know the hardware / time settings, but LZ pretty much crushed him in a practically one-way victory much like my LZ did against a lower grade pro (when it didn't mess up a ladder!).



Winrate grpah from my running LZ #198

Attachment:
LZ Ichirki winrate 198.PNG
LZ Ichirki winrate 198.PNG [ 161.26 KiB | Viewed 13845 times ]


Attachments:
LZ vs Ichirki.sgf [11.62 KiB]
Downloaded 4453 times

Author:  ez4u [ Mon Jan 07, 2019 4:21 am ]
Post subject:  Re: How strong is Leela Zero

You might be interested in this link: https://github.com/breakwa11/GoAIRatings#go-ai-ratings

Author:  ez4u [ Mon Jan 07, 2019 8:24 pm ]
Post subject:  Re: LZ's Progress

chut wrote:
...
I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly. Is that due to a non-linear scaling used in AlphaGo curve?

I was thinking about this comment and so was playing a little with the LZ historical data. Below are two graphs that I made just to emphasize the change in net size over the life of the LZ project to date.

LZ Elo ratings versus cumulative games (in millions), showing the different net sizes. LZ is still far away from the 30 million or so games used by AG. In addition, the project has only recently moved to the 40x256 block size that is (or at least was originally intended as) the final target.
Attachment:
LZ Elo vs Self-Play Games by Net Size.jpg
LZ Elo vs Self-Play Games by Net Size.jpg [ 81.29 KiB | Viewed 13629 times ]

LZ Elo ratings development over time, showing the different net sizes. Comparing this to the first graph gives us some impression of the effect of the LZ project's strategy of starting with small nets for the early learning and then moving to larger nets as things went along.
Attachment:
LZ Elo Over Time by Net Size.jpg
LZ Elo Over Time by Net Size.jpg [ 77.74 KiB | Viewed 13629 times ]

So far I can't see anything profound here but as long-time readers must realize by now, I like pretty pictures! As always, YMMV.

Author:  Uberdude [ Tue Jan 08, 2019 3:32 am ]
Post subject:  Re: How strong is Leela Zero

ez4u, I think it would be a good idea to add the rating of LZElf (v0 and/or v1) as calculated in matches versus LZ to those graphs. If the LZ self-play ratings were non inflationary this would be a flat line as it's the same fixed-strength player. However, it actually increases so gives some idea of the background expansion of the inflationary Leelo (Leela Elo) universe against which LZ's progress is measured. (The test matches are at fixed playouts so Elf will stay the same strength, but as LZ moves to more blocks it will get more time than Elf so you expect it to do better, I think time parity would be fairer so this method will under-report the inflation).
viewtopic.php?p=235675#p235675

Author:  ez4u [ Tue Jan 08, 2019 3:58 am ]
Post subject:  Re: How strong is Leela Zero

Uberdude wrote:
ez4u, I think it would be a good idea to add the rating of LZElf (v0 and/or v1) as calculated in matches versus LZ to those graphs. If the LZ self-play ratings were non inflationary this would be a flat line as it's the same fixed-strength player. However, it actually increases so gives some idea of the background expansion of the inflationary Leelo (Leela Elo) universe against which LZ's progress is measured. (The test matches are at fixed playouts so Elf will stay the same strength, but as LZ moves to more blocks it will get more time than Elf so you expect it to do better, I think time parity would be fairer so this method will under-report the inflation).
https://www.lifein19x19.com/viewtopic.p ... 75#p235675

My graphs are just the info in the 'best net' table on the LZ homepage. I did nothing but break out the net size with different colors. Feel free to improve on them.

On time parity, somewhere in the discussions on github people commented on the fact that different hardware will yield time parity for different levels of visits. So how do we decide what time parity is? Personally I don't know what you mean by 'fair' when you are looking at different generations of the same program. Aren't you going a little bit too far into anthropomorphism? Is 157 going to have its feelings hurt if we keep claiming that 199 is 'better'? ;-)

Author:  chut [ Mon Jan 14, 2019 3:41 am ]
Post subject:  Re: How strong is Leela Zero

It is good to know that LZ is 15 Dan presently. It is very impressive.
What elo number would corresponds to human 9 Dan strength?
I am curious at who point LZ left human behind, and the number of self-played games to get to that point.

Author:  Uberdude [ Mon Jan 14, 2019 4:38 am ]
Post subject:  Re: How strong is Leela Zero

Saying LZ is 15 dan is pretty meaningless IMO, especially if you aren't clear what 1 dan difference means (is it give 1 stone handicap or a winrate difference?). What I can say is watching LZ play Japanese pros on wbaduk LZ always wins, sometimes forcing a resign before move 100, and that's not just against lower strength pros but also against Shibano Toramaru (who is goratings #68, though maybe he is fooling around a bit in these practice games). But then my LZ messed up a ladder on Fox and lost to a 3p down at goratings #600 (fastish game on moderate hardware but it made the same mistake even with 30k+ playouts).

chut wrote:
I am curious at who point LZ left human behind, and the number of self-played games to get to that point.

In the match with Haylee back around May last year LZ network 136 or thereabouts (7.2 million self-play games) games beat her with ease on even and 2 stones but lost on 3. Haylee is a lot weaker than a top pro like Ke Jie and I'm not aware of any serious matches between players of his calibre and LZ to test when LZ passed top humans. You could trawl through the game records of a player called something like LeelaZer15 on Fox who I believe uses a 15-block network (157 I guess, maybe switched to bijxo super-trained) on a 1080 GPU (I briefly chatted with him) to see if he's played any top pros by cross-referencing with Professional Player Usernames on Fox Go Server. That account usually plays 2 handi plus komi games against Fox 9ds (a wide range of strength) and mostly wins.

Author:  chut [ Mon Jan 14, 2019 6:14 am ]
Post subject:  Re: How strong is Leela Zero

Uberdude wrote:
Saying LZ is 15 dan is pretty meaningless IMO, especially if you aren't clear what 1 dan difference means (is it give 1 stone handicap or a winrate difference?). What I can say is watching LZ play Japanese pros on wbaduk LZ always wins, sometimes forcing a resign before move 100, and that's not just against lower strength pros but also against Shibano Toramaru (who is goratings #68, though maybe he is fooling around a bit in these practice games). But then my LZ messed up a ladder on Fox and lost to a 3p down at goratings #600 (fastish game on moderate hardware but it made the same mistake even with 30k+ playouts).

I stone handicap per rank difference only applies to amateur ranking. The strength difference for professional is much closer, if I am not mistaken the difference between 1p and 9p is at most two stones handicap.

In some systems 1 rank difference = 100 points elo difference. But different ranking systems has different scaling and different winning probability between the ranks.

Author:  jlt [ Mon Jan 14, 2019 6:24 am ]
Post subject:  Re: How strong is Leela Zero

Uberdude wrote:
watching LZ play Japanese pros on wbaduk LZ always wins


Do you have an estimate of the number of visits at each move?

Author:  Uberdude [ Mon Jan 14, 2019 6:35 am ]
Post subject:  Re: How strong is Leela Zero

jlt wrote:
Uberdude wrote:
watching LZ play Japanese pros on wbaduk LZ always wins


Do you have an estimate of the number of visits at each move?


No idea. Maybe it would be possible to estimate this by guessing a network you think they are using (e.g. latest best 40 block) and then seeing if the game LZ ends up playing moves which only rise to become the first choice of your locally run LZ after some number of playouts e.g. I run LZ and it wants to play move A until 20k but then after 20k move B is number 1 (and let's assume run for a further 100k and B stays number 1), then if game LZ played B you have some evidence visits was over 20k but no ceiling. If after 150k local testing the number 1 move switched to C then you have evidence game playouts was also <150k. Just based on random enthusiasts having a 1080 Ti or two I'd expect the Wbaduk LZ to be at least something like that so probably several 10s of thousands playouts a move, but that's just a guess. Note I think these games are actually played on the Japanese 'Yugen No Ma' server and wbaduk has some relay thingy.

Page 1 of 1 All times are UTC - 8 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/