LZ's progression

[iopq]

Benchmark with batching, it would be faster than just threading

Could you please show me the command option for batching? It seems that Sabaki always choose batch size 1 by default, while the autogtp chooses something different.

Something like --batchsize I believe

[hoa803]

Could you please show me the command option for batching? It seems that Sabaki always choose batch size 1 by default, while the autogtp chooses something different.

If you use Leela Zero engine version 0.17 it should choose the optimal thread and batch size for your hardware by default. Note that you cannot have more batches than threads.

Try this command in your LZ directory:

leelaz -h

You will see a list of some things you can do.

In terms of engine strength, this person has done a ton of work:

https://github.com/guitanj/AutoPK/blob/master/README.md

These data show that if we are interested in strength as a function on thinking time, LZ will be noticeably weaker since Elfv2 is only a 20-block network and therefore evaluates more quickly. In terms of pure network strength regardless of thinking time, Leela Zero is generally stronger except at high playouts. I find that very interesting.

Very cool data and a lot of work. This person deserves much credit for doing all these tests.

[iopq]

I believe it will do something like batch size 5 and 10 threads.

Anyone try the new 15b net vs. the 40b at time parity?

[Aram]

There was a new 40b network promoted today, and it was also tested officially against a other 40b network from 10 networks ago.

The result is a 69% winrate against the older 40b network from 10 networks back. Rather impressive!
The 40b network is clearly still growing and learning and getting stronger at a good pace!

[Uberdude]

There was a new 40b network promoted today, and it was also tested officially against a other 40b network from 10 networks ago.

The result is a 69% winrate against the older 40b network from 10 networks back. Rather impressive!
The 40b network is clearly still growing and learning and getting stronger at a good pace!

I wonder what the improvement in winrate of this latest 40b compared to 10 networks ago would be versus a non-LZ bot (for fairest comparison would be time parity but even visit parity I expect same effect). Eg maybe #215 beats Elfv2 at 1600 visit parity 65% of the time (and maybe <50% at time parity), and #225 beats it only 67%. I expect quite a lot less difference than 19% (or equivalent ratio difference).

[splee99]

Here is a game between LZ-225 (B) + LZ0.17 and ELF (W) + LZ0.16 in time parity (20 seconds per move).
The ELFv2 was far ahead with winrate close to 90%, until move 135, when LZ-225 suddenly started a complicated fight. After that ELF still had lots of chance to win the fight, according to LZ-225, but eventually ELF gave up.

225-elf-20s.sgf

(3.77 KiB) Downloaded 2253 times

[Aram]

One shouldnt use ELF winrates or ELF in general for analysis.

They sharpened the network a rather huge amount to make it stronger in self play and play versus other bots.
(Probably because they plateud and needed to get some more strength out of the relatively small 20-ish block network)
So basically ELFs possibility to grow and explore new and different moves was terminated and it became stronger on a narrow path of moves.

This also sadly led the network to believe that every match is either a 100% win or a 100% loss.

This is quite visible for anyone using both ELF and LZ networks. ELF will always be very black and white and whats good or bad, while the LZ networks provide several moves which are playable.


LZ could do the same and would probably see a boost in strength, but at the same time it probably wouldnt improve anymore and it wouldnt explore either.
Since the LZ network is larger, it still has capacity to grow and improve by searching and exploring, and as such its not needed to try to force it to its full strength right now.

Same thing with learning rates, one could probobly get a small increase in LZ strength by just dropping the learning rate, but that would kill any further increase in playing strength from exploration.

[Vargo]

20 game match LZ017#225 v LZ017#215
no pondering, komi 7.5, -v 1600
twogtp 1.5.1 , no error, no duplicate game.

#225 wins 17-3
Wow ! It confirms the 69% , and it's only a 10 networks difference...

The stats :

225.gif (46.02 KiB) Viewed 12591 times

The games (225 is B in the odd-numbered games) :

225_215_v 1600.rar

(15.32 KiB) Downloaded 497 times

[nbc44]

Сity legend check (l0XXX 1600 visits vs elfv2 3200 visits):

2 x 1080ti

[Engine_Master]
enginedir = C:\APPS\l0gpu17\
engine = leelaz.exe -g --gpu 0 --gpu 1 -t 24 --batchsize 12 -r 5 --noponder --timemanage off --precision single
weightdir = C:\APPS\net\
weight = 05dbca15.gz
name = ELFV2

[Engine_01]
enginedir = C:\APPS\l0gpu17\
engine = leelaz.exe -g --gpu 0 --gpu 1 -t 24 --batchsize 12 -r 5 --noponder --timemanage off --precision single
weightdir = C:\APPS\net\
weight = a20c31da.gz
name = LZ226

[Engine_02]
enginedir = C:\APPS\l0gpu17\
engine = leelaz.exe -g --gpu 0 --gpu 1 -t 24 --batchsize 12 -r 5 --noponder --timemanage off --precision single
weightdir = C:\APPS\net\
weight = d42f950a.gz
name = LZ225

[Engine_03]
enginedir = C:\APPS\l0gpu17\
engine = leelaz.exe -g --gpu 0 --gpu 1 -t 24 --batchsize 12 -r 5 --noponder --timemanage off --precision single
weightdir = C:\APPS\net\
weight = f3be3bf8.gz
name = LZ224

Code: Select all

===================================================================================================================
From:May-07-19 07:12:33 to May-08-19 05:08:00. Spend 78927.58483s
                         Game << 400 >>
LZ224                    1600 visits  win(All) = 249  62.25%        win(B) = 115  57.50%        win(W) = 134  67.00%
ELFV2                    1600 visits  win(All) = 151  37.75%        win(B) =  66  33.00%        win(W) =  85  42.50%

===================================================================================================================
From:May-09-19 07:42:53 to May-10-19 06:46:29. Spend 83015.170982s
                         Game << 400 >>
LZ224                    1600 visits  win(All) = 212  53.00%        win(B) = 102  51.00%        win(W) = 110  55.00%
ELFV2                    2000 visits  win(All) = 188  47.00%        win(B) =  90  45.00%        win(W) =  98  49.00%

===================================================================================================================
From:May-08-19 05:08:00 to May-09-19 05:47:02. Spend 88741.358819s
                         Game << 400 >>
LZ224                    1600 visits  win(All) = 180  45.00%        win(B) =  91  45.50%        win(W) =  89  44.50%
ELFV2                    2400 visits  win(All) = 220  55.00%        win(B) = 111  55.50%        win(W) = 109  54.50%

===================================================================================================================
From:May-10-19 07:37:42 to May-11-19 07:16:09. Spend 85106.834626s
                         Game << 400 >>
LZ225                    1600 visits  win(All) = 207  51.75%        win(B) =  94  47.00%        win(W) = 113  56.50%
ELFV2                    2000 visits  win(All) = 193  48.25%        win(B) =  87  43.50%        win(W) = 106  53.00%

===================================================================================================================
From:May-11-19 07:16:09 to May-12-19 07:23:49. Spend 86860.492488s
                         Game << 400 >>
LZ225                    1600 visits  win(All) = 172  43.00%        win(B) =  81  40.50%        win(W) =  91  45.50%
ELFV2                    2200 visits  win(All) = 228  57.00%        win(B) = 109  54.50%        win(W) = 119  59.50%

===================================================================================================================
From:May-14-19 06:55:16 to May-15-19 06:12:54. Spend 83858.237752s
                         Game << 400 >>
LZ226                    1600 visits  win(All) = 212  53.00%        win(B) = 100  50.00%        win(W) = 112  56.00%
ELFV2                    2000 visits  win(All) = 188  47.00%        win(B) =  88  44.00%        win(W) = 100  50.00%

===================================================================================================================
From:May-15-19 06:12:54 to May-16-19 05:59:42. Spend 85607.877284s
                         Game << 400 >>
LZ226                    1600 visits  win(All) = 196  49.00%        win(B) =  94  47.00%        win(W) = 102  51.00%
ELFV2                    2200 visits  win(All) = 204  51.00%        win(B) =  98  49.00%        win(W) = 106  53.00%

P.S. If someone wants the games, I can upload them.

[kityanhem]

Can I have the games, nbc44?
For studying, of course!

[splee99]

For higher playouts, for example 2500 vs 5000, ELF becomes weaker than 226. It's just my initial observation.

[SHKD]

Сity legend check (l0XXX 1600 visits vs elfv2 3200 visits):

P.S. If someone wants the games, I can upload them.

I'd like to have these games, interesting to take a look at the next LZ vs ELF episode)

[Vargo]

Interesting topic HERE about a new LZ network.

The provided network (14a3a) looked very promising...
I've tested it, using regular LZ017, first at H9 (komi 0.5) against gnugo (5 kyu)
it won 2 games out of 3 !
(laptop with gtx965, ~3 sec/move effectively used)

Sabaki settings for GnuGo:

Code: Select all

--mode gtp  --level 10

Sabaki settings for for 14a3a

Code: Select all

C:\...\leela-zero-0.17-win64\leelaz.exe   AND   --gtp -w C:...\14a3a5f70.gz --noponder -r 0  AND  time_settings 0 5 1    -komi 0.5

One of the H9 games : here




Second test : 50-game match between 14a3a and #226 (--noponder -v 1600)
14a3a won 52% (gogui 1.5.1, no error, no duplicate game, ~200 moves/game, ~170s/game for #226 or for 14a3a, all games by resignation)

the stats :

stat.gif (160.02 KiB) Viewed 13924 times

the games :

lz_226v14a3.zip

(42.17 KiB) Downloaded 639 times

So, a very interesting network, despite the poor "official" result against 226...

[And]

Second test : 50-game match between 14a3a and #226 (--noponder -v 1600)
14a3a won 52% (gogui 1.5.1, no error, no duplicate game, ~200 moves/game, ~170s/game for #226 or for 14a3a, all games by resignation)

the stats :

stat.gif

the games :

lz_226v14a3.zip

So, a very interesting network, despite the poor "official" result against 226...

the less games, the less reliable result. “Official” result obtained from 158 games.
99a8d3a3 - 226:174 and 115:132 in the official test. maybe even 400 games is not enough

[And]

I wonder why you chose gnugo for handicap tests? the program is quite old. Now programs play differently. possible choose something newer with adjustable strength. By the way, have you tried gnugo with the --cache-size option? (e.g. 256mb)