Zhao Baolong 2p played Leela Zero on consumer-grade hardware

[nasdaq]

wow, no idea it was that strong on only a few thousand playouts!

I did beat it a couple of times using the best 10x128 network on about 100 playouts.

One game was a bad ladder.

The other was a large capture race that leela zero could have won easily if it connected, but because of the size of the groups and very low playouts it missed.
I won that capture race by one move,

On low playouts it does seem susceptible to reading mistakes, as to be expected. Still its very tricky to get it into a position to make anything work.

[Gomoto]

Thanks, your post is a very valuable input on the strength on consumer-grade hardware and notebooks.

I am happy it turns out to be reasonable strong on notebooks

[hydrogenpi7]

this is very interesting news and great outcome for LeelaZero and open source AI Go in general.

I think there are still some room to grow for the 15 block and hope the move to 20 block will squeeze some more power out of it still...

Also, what is considered "consumer-grade hardware" is an ever moving goal post, which means that likely in the future in next few years, even today's TitanV will seem "weak" in comparison.

So its safe to say that the age of AI Go for the masses, where we can each sport a bot stronger than the top Pros, is fastly approaching. We have on one hand bots getting stronger, and the other hand hardware getting stronger... so its pretty inevitable.

if any top pro would like a game please contact me, I will set up using latest 125 network and on 8X V100 *basically like 10 X TitanV gpu (about 10000 nodes per second avg. in real games)

[Uberdude]

On reddit you said Baolong played the opening fast and the endgame slowly, which surprises me as with the benefit of Leela's clues and hindsight he seems to make quite a few moves which seem fairly obviously bad and rushed/greedy even to much weaker me. For example for move 18 he split at s11 instead of q13 attach on top Leela recommended (and my instinct), which to me says he thought letting black connect was soft as the resulting wall feels a bit useless with o16 there but it's patient and fairly solid and good topologically. Not a super big deal imo, -2% says Leela, but then with the r17 attach rather than following up on the split of s11 to attack r9 with the p9 cap like Leela recommended he attaches in the top right, ends up trying to have his cake and eat it too and ends up starving on crumbs. Maybe he's played too many weak amateurs and wasn't in serious playing-a-stronger-player mode?

Also it's interesting he said move 12 was bad, this reminds me of Ke Jie vs AlphaGo game 1 where he too tried to develop from a 3-3 point with a far high extension but ended up with a bad result (this knight's move and nozomi of 13 wasn't played in that game but I recall it in some of the variations discussed as being no good for Ke).

[hydrogenpi7]

On reddit you said Baolong played the opening fast and the endgame slowly, which surprises me

That also surprised me. Before the game he said that he played a few bots on Chinese servers but only won very few games against them, usually by tactical complications; he mentioned one game where he got a double ko and the AI kept making ko threats. He said he didn't think he could win the game against Leela Zero. Maybe he tried to create such tactical complications in the beginning.

But I also wondered why the endgame took so long. Maybe he was looking for ways to make it lose points in the endgame. He said the AI did lose a few points.

Leela's endgame was very slow as well as it took the usual amount of time even for completely obvious endgame sequences, which means it took a long time to actually finish the match. After the last dame was filled, Leela started playing in its own territory, so I finished the game on Leela's behalf.

ends up trying to have his cake and eat it too and ends up starving on crumbs

That was my impression as well.

Also it's interesting he said move 12 was bad,

Baolong agreed with Leela's suggestion of k3.

Also, for move 56 (e15) he said he did see that Black could squeeze White from the outside, as happened in the game up to 69, but he didn't have anything better.

By the way, we also talked about the effect that pro-level AI on standard laptops - and presumably, soon even smartphones - will have on professionals' jobs. They make their money from tournaments, but I guess more from teaching amateurs. He said this aspect is going to get more difficult.

And when he said he thought he could win one or two games out of a hundred, I thought of the network matches on the Leela Zero training page, where networks get promoted if they have a success rate of at least 55% against the current best network. Effectively what Baolong says is that his "grey matter neural network" has a 2% success rate. I find it interesting to put this in perspective with the evolution of the network strength graph.

network 125 should be way stronger than the network that played against Haylee a while back, even on time parity. I had tested the haylee network version vs network 117 a while back and at 3600 po the 117 was at least 90% stronger.

But Haylee match the LZ used 4x 1080 (non TI) and I'm thinking with network 125, the same level of strength can be now obtained on only 1 gtx 1080 or even slightly less.

I think in this day and age a 1080 is pretty much minimum baseline and commodity hardware.

[Tryss]

I think in this day and age a 1080 is pretty much minimum baseline and commodity hardware.

Really? Not everyone is a gamer you know. This card cost over 500€, it's not a "baseline" GPU.

[Mike Novack]

I think in this day and age a 1080 is pretty much minimum baseline and commodity hardware.

Really? Not everyone is a gamer you know. This card cost over 500€, it's not a "baseline" GPU.

We possibly are discussing different meanings of "commodity hardware". For consumers? For industry? The sort of workstations a company might be providing staff whose jobs needed serious power. And which might be coming off lease in just a couple years, being refurbished, and sold << if you know the right places >> Sold for prices which be less than the prices of their CPU and GPU if you were replacing them.

Even the grade of laptops a company might be providing non-technical workers might be far more powerful/advanced than "consumer grade" machines.

[Tryss]

We possibly are discussing different meanings of "commodity hardware". For consumers? For industry? The sort of workstations a company might be providing staff whose jobs needed serious power. And which might be coming off lease in just a couple years, being refurbished, and sold << if you know the right places >> Sold for prices which be less than the prices of their CPU and GPU if you were replacing them.

Even the grade of laptops a company might be providing non-technical workers might be far more powerful/advanced than "consumer grade" machines.

In a couple years, yeah, of course. But not "in this day and age".

In march 2018, only a very small part of the Steam users (= the gamers population) have a 1080 or better GPU (under 5%).

And about workstations? Unless they are build for machine learning, they will probably have a worse GPU than a 1080. I mean, I checked what Dell propose by default for its 6500$ workstation : a Quadro P4000, so it's worse than the 1080.

[hydrogenpi7]

We possibly are discussing different meanings of "commodity hardware". For consumers? For industry? The sort of workstations a company might be providing staff whose jobs needed serious power. And which might be coming off lease in just a couple years, being refurbished, and sold << if you know the right places >> Sold for prices which be less than the prices of their CPU and GPU if you were replacing them.

Even the grade of laptops a company might be providing non-technical workers might be far more powerful/advanced than "consumer grade" machines.

In a couple years, yeah, of course. But not "in this day and age".

In march 2018, only a very small part of the Steam users (= the gamers population) have a 1080 or better GPU (under 5%).

And about workstations? Unless they are build for machine learning, they will probably have a worse GPU than a 1080. I mean, I checked what Dell propose by default for its 6500$ workstation : a Quadro P4000, so it's worse than the 1080.

So here is the thing, when I got my first gtx 1080Ti extreme OC edition last October, even back then it couldn't even play Ghost Recon with max out settings while at 1440p without the occasional hiccup and the internal benchmark gave only about 90fps/ so I could forget about 4k. But the 1080Ti was bragged as the first truly 4k card by Nvidia and cohorts.

As for what I mean, anyone who is doing 3ds max, vray, unreal gaming development, or any kind of deep learning, tensorflow, etc would do well with their TIME to acquire at least a 1080 if not a titanv. The extra couple hundred bucks would be more than worth it. Since the general audience here is not the general public (the general public doesn't play Go and isn't interested in it) but actaully those in the computer go arena, I would say a 1080 is the baseline. The 2080 should be out any time now.

Right now, single GPU of any kind, is "commodity" unless we are talking about TitanV which is proconsumer and V100 which is enterprise.

There is no doubt that Leela Zero on commodity hardware has far suprassed AlphaGo Fan. The remarkable part about this is the fact that AlphaGo Fan used hundreds of GPUs and thousands of CPUs. The fact that on a single 1080 with latest LZ net we can already surpass ALphaGO Fan is something I personally wouldn't have fathomed would happen just a mere three years later in terms of scaling down to a laptop or single desktop with one gpu from the datacenter of a beast google used back in 2015.

Argueably AlphaGoLee is not far away. On very high end hardware (aka not commodity) I'd say its already AlphaGo Lee strength.

Franjkly its just a matter of time and resources, given enough client volunteers I don't see why LZ can't someday catch up with and even surpass AGZ.

[Uberdude]

So I just randomly looked at the 1st game of the most recent test match and current 192x15 network #125 lost with a ladder misread (there's a black stone in the path of the ladder but it's still good for white): http://zero.sjeng.org/viewmatch/cb7e08e ... viewer=wgo.

That game info says there were 3200 visits (do I understand args correctly or is there additional time restriction?), do we still expect such ladder misreads on a 15 block network with that many visits? Are these test matches similar so vs Baolong it could have been as silly against him if he gave it the opportunity?

[dfan]

Leela Zero still has issues with ladders, yeah. Part of the problem is that to really learn about them, they have to be played out, or at least partially played out, at some reasonable frequency in the its self-play games that it uses to train itself. Someone did a nice experiment where they got Leela Zero to improve at ladders just by having it look at relevant positions from the self-play games more often. (Here is the discussion on the Leela Zero GitHub page.

Of course you could also add ladder input features, as the original AlphaGo did. Part of it comes down to whether the point is to create the strongest possible go program, or to do so within the "minimal human knowledge" constraint as AlphaGo Zero did.

[hydrogenpi7]

That game info says there were 3200 visits (do I understand args correctly or is there additional time restriction?), do we still expect such ladder misreads on a 15 block network with that many visits? Are these test matches similar so vs Baolong it could have been as silly against him if he gave it the opportunity?

In the Baolong game it got 3000-4000 playouts, which should be more than 3200 visits, if I understand this correctly. Even so, some of the variations it generated (but didn't end up playing) also featured non-working ladders. I definitely think that it still makes reading mistakes, especially with long ladders and long(ish) capturing races.

But as a human player, betting on those mistakes also seems risky. If you try to trick it but it does end up responding correctly, it could lose the game even more quickly. (That is my weak-level understanding.)

Makes sense.

Even if LZ still makes ladder/large group mistakes 5% of the time, in a real match against a pro, the pro can't really bet on it failing a ladder and using that as a strategy to defeat it in an even game. If I recall, even the AGZ didn't completely master ladders until well into the middle of its training, which means given the # of games LZ has self played, it is not there yet.

[hydrogenpi7]

Leela Zero still has issues with ladders, yeah. Part of the problem is that to really learn about them, they have to be played out, or at least partially played out, at some reasonable frequency in the its self-play games that it uses to train itself. Someone did a nice experiment where they got Leela Zero to improve at ladders just by having it look at relevant positions from the self-play games more often. (Here is the discussion on the Leela Zero GitHub page.

Of course you could also add ladder input features, as the original AlphaGo did. Part of it comes down to whether the point is to create the strongest possible go program, or to do so within the "minimal human knowledge" constraint as AlphaGo Zero did.

Originally this was supposed to be LZ's first run.
Based on the sentiment my guess is that they will go to 20block with this at least, if not also 40 blocks eventually. (meaning there will not be a second or third run) ....
But so I think what is happening is that the dev wants to keep LZ as "pure" as possible and not do the things that could get it strongest right now, and then as other people find ways to make it stronger, it goes off the main LZ channel... so in the computer GO arena, LZ may always lag behind a bit due to other AI bots taking advantage of things that LZ refrains from. But LZ could be the bedrock that everything else branches out from...

I think the cgos benchmark is getting way inflated... What used to be 3900 elo is no longer 3900 elo. Bots are getting stronger and stronger and the elo is squashed to be normalized.

A more useful and imho more interesting benchmark is to test AI bots against top pros (best humans) and also to try to scale it down to as commodity of hardware as possible.