jokkebk wrote:Looked through half of the pro vs. elf games. Facebook's bot is just frighteningly powerful (to my untrained 1k eyes at least). If you trust the "black value" evaluation (I assume black value goes from -1 to 1 meaning almost certain loss to almost certain win) in the comments, it seems most games are nearly lost after 30-50 moves.
I also had a look at one of the games with Lizzie, and Leela doesn't give as strong edge (the scales are different of course) to FB bot as quickly, taking maybe 10-20 moves extra for pro chances to drop below 35 %, but still it's showing the bot is crushing most games by move 50, with pro odds usually around 25 %.
I'm hoping the Leela Zero project will gain from Facebook's contribution. Not sure if they should "borrow" the network weights, use ELF OpenGo to help in the training, or just look for ways to improve LZ's training indirectly, though.
My comments on gh:
So… overall ELF is at least >500+ Elo stronger than current best LZ network and possibily up to 800 elo stronker. But it appears to have some latent issues such as being even more prone to ladder fallability and other certain fragilities at less than superhigh playouts, which ironically are things that aren’t as prelevant in the current LZ net arch. By using the approach of hybrid mix 50% LZ/ 50% ELF self-games and continuing to train the LZ net but using ELF as a part-time strength-gainer games-generator (at least until which point LZ catches up to ELF or even surpasses it) may patch the current weak spots in both net archs and set up a framework that is repeatable into the future, such as if AGZ weights get released or facebook teams releases a (new) and even stronger ELF OpenGo network sometime in the future, etc this can be repeated again. And it gives the LZ project the chance to figure out how to resolve two of the biggest issues of superhuman Go AI, that is ladders, large group deaths, and things like high handicap and variable komi and whatnots. Since the current 15b still has enough capacity left, the adding of ELF games will allow LZ to catch up enough to give the devs a way to figure out how to lower gating or remove gating etc. For sure the continuation of “things as usual” for the LZ project is no longer an option, if for no other reason than the strength gap being too great for volunteer clients to still be willing to go the business as usual path, and as gcp stated, its not an option to train the ELF network directly either. (not to mention that switching directly to the ELF network would be in essence killing off the Leela Zero itself) So looks like the roadmap going forward is to train LZ hybridized with ELF, so it seems like there is a way to make good on the fb post about ELF helping project like Leela Zero after all!
1000 games of Facebook OpenGo vs Leela Zero on Nvidia Tesla V100 at 80,000 playouts per move.
(acuatlly its 998 games and Leela Zero had a 18:998 winrate, which means the ELF is ~700 elo stronger)
What is not in debate is this, in terms of open source Go, we have well passed without a shadow of any doubt into “superhuman” territory. ELF is estimated to be 4400 real elo on max hardware while top players are about 3700.
