Is leela zero really that strong?

[Uberdude]

Is there a way to configure it not to resign? When I play leelaz at 5 stones, it will resign when I think it still has good chances due to its skill advantage.

running leeaz --help gives

-r [ --resignpct ] arg (=-1) Resign when winrate is less than x%.
-1 uses 10% but scales for handicap.

so "-r 0"

[betgo]

I changed the configuration to not resign. I then played a game where it edged me out at the end by 3 points at 6 stones and I won easily at 7 stones. I couldn't often beat a weak professional player at 7 stones and I play regular leela at 4 stones. I might be able to beat leelaz at 5 stones with practice.

I noticed that it generally did not launch invasions enough in a high handicap game. Also, it appeared to make clear errors in capturing stones that were pretty dead. I find it hard to believe it could beat top professionals, but it could be my computer and configuration also make it weaker.

[Uberdude]

If you want to know if LZ can beat top professionals in an even game then look at the result of it playing top professional in an even game. Trying to extrapolate from a high handicap against a 1k and how many stones pros can give a 1k is answering a different question. You are using the assumption if A beats C at x handis and B loses to C at x handis that implies A will beat B in an even game. Whilst that is often (but not always) true for humans, it is often not true for bots which are trained to play even games and bad at handicap.

Here is LZ on my £200 GPU easily beating a Japanese 8p (goratings ~#500) in a one sided win by over 20 points. That was with a few thousand playouts a move, strong hardware is in the hundreds of thousands so will be stronger. viewtopic.php?p=240789#p240789

[Uberdude]

I'm not aware of many recent even games between top pros and LZ. Here is a review from over a year ago of LZ (a 20 blocks network, much weaker than latest 40 block ones) on 4 beefy Tesla V100 GPUs beating top 10 pro Fan Tingyu 9p.
https://youtu.be/QZY2erUtJ_0

[Aram]

The "AlphaGo Style" bots do not play a good handicap game. Their whole design is to learn by themselves on how to play even games on 19x19.
They dont train handicap, and since their judgement has been so honed and perfected, they constantly think that they are loosing in handicap, making them sometimes play silly moves.
When every move is a loosing move, it doesn't really matter what you play.. they're incapable of understanding that you will play sub-optimally. They just think the opponent will play as good as they can.

Is Leela Zero stronger than any human professional alive (and probably dead) on a mid range GPU? Yes, has been for ages, and can probable give 0 komi or 2-3 handicap.

A small handicap is easier for the bot to claw back, as the judgement of winning percentages is still in a sane range. When you go to high handicap every move is a loosing move in the eyes of the bot.


A F1 car doesn't really work well as a rally car, and a rally car doesn't really work well as a F1 car. Both are racing cars built to win
Just because a bot can destroy any professional in an even game, doesn't mean it can play handicap (sadly)

[betgo]

I don't know how to post a smart go format game, but I just crushed it at 5 stones. I sacrificed 3 stones early on and it removed them from the board when that was totally unnecessary. It let me play the 3x4 point under my handicap stone with an shimari to the 7x3 point, solidifying a huge territory when it had easy invasions at various points.

[Aram]

As i said earlier, when every move is a sure loss, it will just play one of them more or less at random, since they all lead to the same result in its opinion.

It simply does not handle high handicap games. If you want a handicap opponent to play against, Leela Zero is the incorrect choise (you might be able to get away with 1-2 stones, depending a bit)

[iopq]

Try 7 stones, but this time against KataGo. It will crush, most likely. KataGo tries to maximize score.

[Vargo]

I think the best network to play handicap go with regular LZ is THIS ONE

I don't know how to post a smart go format game.

If you want to post a game, you can for example
1) upload it
2) click "place inline"
3) click "preview"
4) right-click on the attachment, and choose "copy link location"
5) paste it
6) select what you just pasted, and click the "sgf" button

c.jpg (95.4 KiB) Viewed 11366 times

it should be ok, the preview should look like this :

d.jpg (433.56 KiB) Viewed 11366 times

It's probably not the best method, but it works.
_________________________________________________________

BTW, network 14a3a5f wins ~80% of its H8 games, with komi 7.5 against network #48, which is supposed to be 1d, according to THIS SITE (I'm not sure about the exact settings they used for #48 to be 1d)

settings :
Black #48 : --noponder -r 1 -p 1601
White 14a3a5f : --noponder -r 0 -p 1601
(here, W uses 15 times more time than B at equal playouts !)

48v14a3a_h8.sgf

(2.26 KiB) Downloaded 1593 times

[betgo]

I will look into some of the other alternatives mentioned.

leelaz-model-swa-24-192000_quantized Is this the right weight file? https://www.youtube.com/watch?v=FV6dSjnx7jY It was recommended by that video.

I lost to leelaz by 3 points at 3 stones, so I have a hard time believing it could beat a top pro at 3 stones.

[Uberdude]

I lost to leelaz by 3 points at 3 stones, so I have a hard time believing it could beat a top pro at 3 stones.

Bear in mind LeelaZero is not a single Go player, but an infinite family of them. Your LeelaZero is a combination of [network file, number of playouts/visits/time settings, LZ engine version, other factors]. My and anyone else's LeelaZero is a different set of those things and can thus be stronger or weaker. Did you run without the time setting parameters and just the playouts, so you can measure how long your computer takes to get 2500 playouts per move?

Here is a version of LeelaZero (LeelaMaster 20 block weights, beefy hardware) beating Kim Youngsam 8d on 3 stones (with komi) over a year ago.

[betgo]

I tried it without the time settings and it took about 40 seconds per move, so at 5 seconds, it is getting like 300 playouts. How much weaker would that make it?

[AloneAgainstAll]

Well, 300 playouts is very very low, so dont expect pro-crushing strenght. Also you said 5s per move - you responed in 5 secs too?

Giving 3 stones (komi=0) to top pros might be a bit too much.FineArt which is not only damn strong in even games, but also extremely good at handi games has special account which plays 2 stones with 7,5 komi.Against one Taiwan 9p he is 45 wins - 8 loses, so i guess they might be around even with just 2h (no komi).This FA played 2 games with Ke Jie with result 1-1.It was more than year ago, so FA might be stronger now, but 3 stones to top pros might be too much for bots.

[Gomoto]

Is somebody really that strong at computer science?

Sorry, could not resist.

[betgo]

I was averaging significantly less than 5 seconds per move, but since I am not a computer I wasn't taking exactly 5 seconds for each move.

So I know the breakthrough was using AI techniques, and that chess programs were successful mostly without them. However, I guess computational power is still really important.Those programs beating top pros are presumably running on powerful computers and looking at a lot more than 2500 playouts.