A question - in the current Elo scale used in leela zero Strength Graph, where would human professional be? Can someone give a rough range? How many dan rank is LZ stronger than human now?

Hi chut,

Suppose we have anecdotal evidence that the current best engine (which may not be LZ) is roughly 2 or 3 stones better than top human pros, how does that affect your life ? ( What would you do about it ? )

I am primarily a computer scientist, and being a go player is only for casual amusements. My interest would be - to use the car analogy - what else can be done to make the car go faster, and more fuel efficient.

I would like to see the different engines competing against each other, to see analysis of how different strategies lead to better/worst performance. And having some milestone markers - human, Alphago Fan, Alphago Lee etc - will make comprehension a lot easier for my feeble mind.

I am marveled that LZ is still progressing at near linear rate, while the AlphaGo Zero curve went horizontal rather quickly. Is that due to a non-linear scaling used in AlphaGo curve?

My guess is that LZ is still far-far away from AGZ. In terms of steepness of the progress curve, I guess it depends how you plot it, but I think the reason why LZ may still improve relatively fast is because it has a lot more room to grow, being weaker.

I am also fascinated by your original question, by the way, I would like very much to see more pros playing experimental games with handicap against LZ to measure the gap.

_________________
Sorin - 361points.com

I agree. I think with AlphaGo Zero we are seeing the law of diminishing returns, while Leela Zero isn't yet strong enough to see that.



Me, too.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

The strength depends on the hardware and on thinking time, but from the thread "LeelaZero adventures on Fox", and from petgo3's rank on KGS, I guess that on medium hardware, and with relatively fast time settings, LZ is about professional strength but not top pro.

The only game I played as LZ against a known pro on Fox was here. It was a low dan Chinese pro Li Jianyu who I think is this guy: https://www.goratings.org/en/players/1730.html, goratings #441 with 3079 elo (very top pros being 3600+). LZ 191 with a few thousand playouts (probably a 20 or 30 second byoyomi with my GeForce 1060 GPU) beat him with ease, the winrate was a one-way and rapid decline. How much better would a top pro do?

Here's a game from WBaduk, an account called leelazero (black) vs Ichiriki Ryo 8p (goratings #43). I don't know the hardware / time settings, but LZ pretty much crushed him in a practically one-way victory much like my LZ did against a lower grade pro (when it didn't mess up a ladder!).



Winrate grpah from my running LZ #198

You might be interested in this link: https://github.com/breakwa11/GoAIRatings#go-ai-ratings

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

I was thinking about this comment and so was playing a little with the LZ historical data. Below are two graphs that I made just to emphasize the change in net size over the life of the LZ project to date.

LZ Elo ratings versus cumulative games (in millions), showing the different net sizes. LZ is still far away from the 30 million or so games used by AG. In addition, the project has only recently moved to the 40x256 block size that is (or at least was originally intended as) the final target.

LZ Elo ratings development over time, showing the different net sizes. Comparing this to the first graph gives us some impression of the effect of the LZ project's strategy of starting with small nets for the early learning and then moving to larger nets as things went along.

So far I can't see anything profound here but as long-time readers must realize by now, I like pretty pictures! As always, YMMV.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

ez4u, I think it would be a good idea to add the rating of LZElf (v0 and/or v1) as calculated in matches versus LZ to those graphs. If the LZ self-play ratings were non inflationary this would be a flat line as it's the same fixed-strength player. However, it actually increases so gives some idea of the background expansion of the inflationary Leelo (Leela Elo) universe against which LZ's progress is measured. (The test matches are at fixed playouts so Elf will stay the same strength, but as LZ moves to more blocks it will get more time than Elf so you expect it to do better, I think time parity would be fairer so this method will under-report the inflation).
viewtopic.php?p=235675#p235675

My graphs are just the info in the 'best net' table on the LZ homepage. I did nothing but break out the net size with different colors. Feel free to improve on them.

On time parity, somewhere in the discussions on github people commented on the fact that different hardware will yield time parity for different levels of visits. So how do we decide what time parity is? Personally I don't know what you mean by 'fair' when you are looking at different generations of the same program. Aren't you going a little bit too far into anthropomorphism? Is 157 going to have its feelings hurt if we keep claiming that 199 is 'better'?

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

It is good to know that LZ is 15 Dan presently. It is very impressive.
What elo number would corresponds to human 9 Dan strength?
I am curious at who point LZ left human behind, and the number of self-played games to get to that point.

Saying LZ is 15 dan is pretty meaningless IMO, especially if you aren't clear what 1 dan difference means (is it give 1 stone handicap or a winrate difference?). What I can say is watching LZ play Japanese pros on wbaduk LZ always wins, sometimes forcing a resign before move 100, and that's not just against lower strength pros but also against Shibano Toramaru (who is goratings #68, though maybe he is fooling around a bit in these practice games). But then my LZ messed up a ladder on Fox and lost to a 3p down at goratings #600 (fastish game on moderate hardware but it made the same mistake even with 30k+ playouts).


In the match with Haylee back around May last year LZ network 136 or thereabouts (7.2 million self-play games) games beat her with ease on even and 2 stones but lost on 3. Haylee is a lot weaker than a top pro like Ke Jie and I'm not aware of any serious matches between players of his calibre and LZ to test when LZ passed top humans. You could trawl through the game records of a player called something like LeelaZer15 on Fox who I believe uses a 15-block network (157 I guess, maybe switched to bijxo super-trained) on a 1080 GPU (I briefly chatted with him) to see if he's played any top pros by cross-referencing with Professional Player Usernames on Fox Go Server. That account usually plays 2 handi plus komi games against Fox 9ds (a wide range of strength) and mostly wins.

I stone handicap per rank difference only applies to amateur ranking. The strength difference for professional is much closer, if I am not mistaken the difference between 1p and 9p is at most two stones handicap.

In some systems 1 rank difference = 100 points elo difference. But different ranking systems has different scaling and different winning probability between the ranks.

Do you have an estimate of the number of visits at each move?

No idea. Maybe it would be possible to estimate this by guessing a network you think they are using (e.g. latest best 40 block) and then seeing if the game LZ ends up playing moves which only rise to become the first choice of your locally run LZ after some number of playouts e.g. I run LZ and it wants to play move A until 20k but then after 20k move B is number 1 (and let's assume run for a further 100k and B stays number 1), then if game LZ played B you have some evidence visits was over 20k but no ceiling. If after 150k local testing the number 1 move switched to C then you have evidence game playouts was also <150k. Just based on random enthusiasts having a 1080 Ti or two I'd expect the Wbaduk LZ to be at least something like that so probably several 10s of thousands playouts a move, but that's just a guess. Note I think these games are actually played on the Japanese 'Yugen No Ma' server and wbaduk has some relay thingy.