How strong is Leela Zero

[chut]

A question - in the current Elo scale used in leela zero Strength Graph, where would human professional be? Can someone give a rough range? How many dan rank is LZ stronger than human now?

[EdLee]

Hi chut,

Suppose we have anecdotal evidence that the current best engine (which may not be LZ) is roughly 2 or 3 stones better than top human pros, how does that affect your life ? ( What would you do about it ? )

Same question: for ~5 stones.
Curious about the human feelings behind the question.
How much faster is the current fastest car than the fastest running human ?
For some, it really matters a car reaches speed X in N seconds.

[chut]

Hi chut,

Suppose we have anecdotal evidence that the current best engine (which may not be LZ) is roughly 2 or 3 stones better than top human pros, how does that affect your life ? ( What would you do about it ? )

Same question: for ~5 stones.
Curious about the human feelings behind the question.
How much faster is the current fastest car than the fastest running human ?
For some, it really matters a car reaches speed X in N seconds.

I am primarily a computer scientist, and being a go player is only for casual amusements. My interest would be - to use the car analogy - what else can be done to make the car go faster, and more fuel efficient.

I would like to see the different engines competing against each other, to see analysis of how different strategies lead to better/worst performance. And having some milestone markers - human, Alphago Fan, Alphago Lee etc - will make comprehension a lot easier for my feeble mind.

I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly. Is that due to a non-linear scaling used in AlphaGo curve?

[sorin]

I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly.

My guess is that LZ is still far-far away from AGZ. In terms of steepness of the progress curve, I guess it depends how you plot it, but I think the reason why LZ may still improve relatively fast is because it has a lot more room to grow, being weaker.

I am also fascinated by your original question, by the way, I would like very much to see more pros playing experimental games with handicap against LZ to measure the gap.

[Bill Spight]

I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly.

My guess is that LZ is still far-far away from AGZ. In terms of steepness of the progress curve, I guess it depends how you plot it, but I think the reason why LZ may still improve relatively fast is because it has a lot more room to grow, being weaker.

I agree. I think with AlphaGo Zero we are seeing the law of diminishing returns, while Leela Zero isn't yet strong enough to see that.

I am also fascinated by your original question, by the way, I would like very much to see more pros playing experimental games with handicap against LZ to measure the gap.

Me, too.

[jlt]

A question - in the current Elo scale used in leela zero Strength Graph, where would human professional be? Can someone give a rough range? How many dan rank is LZ stronger than human now?

The strength depends on the hardware and on thinking time, but from the thread "LeelaZero adventures on Fox", and from petgo3's rank on KGS, I guess that on medium hardware, and with relatively fast time settings, LZ is about professional strength but not top pro.

[Uberdude]

The only game I played as LZ against a known pro on Fox was here. It was a low dan Chinese pro Li Jianyu who I think is this guy: https://www.goratings.org/en/players/1730.html, goratings #441 with 3079 elo (very top pros being 3600+). LZ 191 with a few thousand playouts (probably a 20 or 30 second byoyomi with my GeForce 1060 GPU) beat him with ease, the winrate was a one-way and rapid decline. How much better would a top pro do?

[Uberdude]

Here's a game from WBaduk, an account called leelazero (black) vs Ichiriki Ryo 8p (goratings #43). I don't know the hardware / time settings, but LZ pretty much crushed him in a practically one-way victory much like my LZ did against a lower grade pro (when it didn't mess up a ladder!).



Winrate grpah from my running LZ #198

LZ Ichirki winrate 198.PNG (161.26 KiB) Viewed 18103 times

[ez4u]

You might be interested in this link: https://github.com/breakwa11/GoAIRatings#go-ai-ratings

[ez4u]

...
I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly. Is that due to a non-linear scaling used in AlphaGo curve?

I was thinking about this comment and so was playing a little with the LZ historical data. Below are two graphs that I made just to emphasize the change in net size over the life of the LZ project to date.

LZ Elo ratings versus cumulative games (in millions), showing the different net sizes. LZ is still far away from the 30 million or so games used by AG. In addition, the project has only recently moved to the 40x256 block size that is (or at least was originally intended as) the final target.

LZ Elo vs Self-Play Games by Net Size.jpg (81.29 KiB) Viewed 17887 times

LZ Elo ratings development over time, showing the different net sizes. Comparing this to the first graph gives us some impression of the effect of the LZ project's strategy of starting with small nets for the early learning and then moving to larger nets as things went along.

LZ Elo Over Time by Net Size.jpg (77.74 KiB) Viewed 17887 times

So far I can't see anything profound here but as long-time readers must realize by now, I like pretty pictures! As always, YMMV.

[Uberdude]

ez4u, I think it would be a good idea to add the rating of LZElf (v0 and/or v1) as calculated in matches versus LZ to those graphs. If the LZ self-play ratings were non inflationary this would be a flat line as it's the same fixed-strength player. However, it actually increases so gives some idea of the background expansion of the inflationary Leelo (Leela Elo) universe against which LZ's progress is measured. (The test matches are at fixed playouts so Elf will stay the same strength, but as LZ moves to more blocks it will get more time than Elf so you expect it to do better, I think time parity would be fairer so this method will under-report the inflation).
https://www.lifein19x19.com/viewtopic.p ... 75#p235675

[ez4u]

ez4u, I think it would be a good idea to add the rating of LZElf (v0 and/or v1) as calculated in matches versus LZ to those graphs. If the LZ self-play ratings were non inflationary this would be a flat line as it's the same fixed-strength player. However, it actually increases so gives some idea of the background expansion of the inflationary Leelo (Leela Elo) universe against which LZ's progress is measured. (The test matches are at fixed playouts so Elf will stay the same strength, but as LZ moves to more blocks it will get more time than Elf so you expect it to do better, I think time parity would be fairer so this method will under-report the inflation).
https://www.lifein19x19.com/viewtopic.p ... 75#p235675

My graphs are just the info in the 'best net' table on the LZ homepage. I did nothing but break out the net size with different colors. Feel free to improve on them.

On time parity, somewhere in the discussions on github people commented on the fact that different hardware will yield time parity for different levels of visits. So how do we decide what time parity is? Personally I don't know what you mean by 'fair' when you are looking at different generations of the same program. Aren't you going a little bit too far into anthropomorphism? Is 157 going to have its feelings hurt if we keep claiming that 199 is 'better'?

[chut]

It is good to know that LZ is 15 Dan presently. It is very impressive.
What elo number would corresponds to human 9 Dan strength?
I am curious at who point LZ left human behind, and the number of self-played games to get to that point.

[Uberdude]

Saying LZ is 15 dan is pretty meaningless IMO, especially if you aren't clear what 1 dan difference means (is it give 1 stone handicap or a winrate difference?). What I can say is watching LZ play Japanese pros on wbaduk LZ always wins, sometimes forcing a resign before move 100, and that's not just against lower strength pros but also against Shibano Toramaru (who is goratings #68, though maybe he is fooling around a bit in these practice games). But then my LZ messed up a ladder on Fox and lost to a 3p down at goratings #600 (fastish game on moderate hardware but it made the same mistake even with 30k+ playouts).

I am curious at who point LZ left human behind, and the number of self-played games to get to that point.

In the match with Haylee back around May last year LZ network 136 or thereabouts (7.2 million self-play games) games beat her with ease on even and 2 stones but lost on 3. Haylee is a lot weaker than a top pro like Ke Jie and I'm not aware of any serious matches between players of his calibre and LZ to test when LZ passed top humans. You could trawl through the game records of a player called something like LeelaZer15 on Fox who I believe uses a 15-block network (157 I guess, maybe switched to bijxo super-trained) on a 1080 GPU (I briefly chatted with him) to see if he's played any top pros by cross-referencing with Professional Player Usernames on Fox Go Server. That account usually plays 2 handi plus komi games against Fox 9ds (a wide range of strength) and mostly wins.

[chut]

Saying LZ is 15 dan is pretty meaningless IMO, especially if you aren't clear what 1 dan difference means (is it give 1 stone handicap or a winrate difference?). What I can say is watching LZ play Japanese pros on wbaduk LZ always wins, sometimes forcing a resign before move 100, and that's not just against lower strength pros but also against Shibano Toramaru (who is goratings #68, though maybe he is fooling around a bit in these practice games). But then my LZ messed up a ladder on Fox and lost to a 3p down at goratings #600 (fastish game on moderate hardware but it made the same mistake even with 30k+ playouts).

I stone handicap per rank difference only applies to amateur ranking. The strength difference for professional is much closer, if I am not mistaken the difference between 1p and 9p is at most two stones handicap.

In some systems 1 rank difference = 100 points elo difference. But different ranking systems has different scaling and different winning probability between the ranks.