LZ's progression

[roy7]

For those who might prefer it, there's an experimental "new" home page at http://zero.sjeng.org/home.

[Vargo]

400 is the magic number , so, I extended the match between #196 (40x256) and 92297ff2 (15x192) to 400 games (more than 10 hours!).

time parity : 1 min per side per game, komi 7.5, no pondering, 1x1080, LZ0.16, twogtp V1.5.0
0 duplicate game, 0 error, all games by resignation, no contested scores, the games look normal and seem to have varied beginnings.


The first 10 results were
1, 0, 1, 0, 0, 0, 0, 0, 1, 0 .... (1 if #196 wins)
So, the cumulative winrate for #196 at move n, begins like this :
1, 0.5 , 0.66 , 0.5 , 0.4 , 0.33 , 0.29 , 0.25, 0.33 , 0.3 , ....
you can see these numbers in the left of each graph.


There are huge swings for the winrate in the first few games, and then, the winrate gets more and more stable with the number of games increasing.
The first picture is after 10, 50, and 100 games, with a dotted red line at 50%.
The second one is after 400 games, the dotted red lines are at 0.31 and 0.36, and the "final" win percentage is 0.33

I'd be very interested if someone could duplicate this experiment, I bet the graph after 400 games would look the same, but who knows...

3gr.gif (11.67 KiB) Viewed 15115 times

400.gif (9.82 KiB) Viewed 15115 times

[And]

thanks! do you think that with more time control 92297ff2 will also be stronger than lz196?
please upload the games 1min/game and 5min/game

[And]

accidentally discovered a shortened version of an elf - 136mb (same values, only rounded, original - 248mb) http://gridmaster.tengen.nl/engines/armv7l/. I launched a benchmark with the original, then with a shortened one. shortened runs about 12 percent faster. I wonder how this rounding affects the strength of the game? (LZ cpuonly, Athlon x4 950, leelaz.exe -w theNetworkfile.txt --benchmark -v 300, results: 90 sec and 80 sec)

[Vargo]

I wanted to see if another 400 game match would end up with a graph similar to the first one, so, I ran a second 400 game match with the same parameters.
The answer is yes,the graphs look the same, but... the two final winrates seem very different (33% and 39%), particularly if you consider it's 400 games. Go figure

The .bat file contains :

Code: Select all

gogui-twogtp -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\196.gz --noponder" -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\92297ff_15x192.gz --noponder" -time 1 -games 400 -sgffile 196_15x192_1min -auto -komi 7.5 -alternate

Because of -alternate, in the even-numbered games #196 is always W, (and always B for the odd ones)
Again, there was 0 error, 0 duplicate game, and no contested score, all games by resignation.

The dotted red lines are at 0.38 and 0.42

400.gif (9.84 KiB) Viewed 14940 times

The zip contains the 400 games, and the detailed stats (the html file)

196_15x192_1min.zip

(331.61 KiB) Downloaded 720 times

[Vargo]

Third and last 400 game match between #196 and #92297ff at 1 min per side per game.
Same parameters.

This time, there were 3 duplicate games among the 400, but no error or unagreed score.

The three final results are 33%, 39%, and 39%

gr.gif (11.89 KiB) Viewed 14813 times

The zip contains the 400 games of this last match, and the detailed stats (the html file)

196_15x192_1min_bis.zip

(343.21 KiB) Downloaded 768 times

[Uberdude]

Just one more data point for the 9229 network: I just used it against petgo3 on KGS which was using #197 on similar hardware to my own, 10x20s byoyomis, I/9229 won, see viewtopic.php?p=240161#p240161. So at 20 seconds per move (more playouts than the minutes per game here) on a medium GPU those extra playouts were still more important than the stronger intuition of the deeper network.

[And]

new network 15x192 from bjiyxo: 8b23a0ea VS 92297ff2 229 : 191 (54.52%). the strongest 15x192?
http://zero.sjeng.org/networks/8b23a0ea ... 8cf3b3a.gz

[Vargo]

20 game match : #200 v. #157 (newest 40x256 v. last 15x192)

Time parity, 5min per side and per game, GPU: 1x1080, no pondering, komi 7.5, twogtp 1.5.0, ~205 moves/game (min=114, max=324), ~2sec/move
All games by resignation, no duplicate, no error.

Result : #200 wins 17-3
WOAW !

Stats :

200v157_5min.zip

(11.33 KiB) Downloaded 725 times

[nbc44]

20 game match : #200 v. #157 (newest 40x256 v. last 15x192)

Why not the latest 15x192 199-96k quantized?

Code: Select all

2019-01-08 07:01	f438268e VS  92297ff2 230 : 174 (56.93%)	404 / 400	PASS
2018-12-22 02:42	92297ff2 VS  d351f06e 260 : 144 (64.36%)	404 / 400	PASS

[Vargo]

Why not the latest 15x192 199-96k quantized?

Why not, indeed !

20 game match : #200 v. #f438268

LZ0.16, Time parity, 5min per side and per game, GPU: 1x1080, no pondering, komi 7.5, twogtp 1.5.0, ~198 moves/game (min=106, max=304), ~2sec/move
All games by resignation, no duplicate, no error.

Result : #200 wins 11-9

STATS:

200_f438_5min.zip

(11.22 KiB) Downloaded 717 times

The games (200 is Black in the games 200_f438_5min_2-XXX, 200 is White for 200_f438_5min-XXX )

200_f438_5min_SGF.zip

(17.01 KiB) Downloaded 720 times

[Mike Novack]

Result : #200 wins 11-9

Because it is likely that the majority here have never had a course in statistics, it might be useful, when posting something like this to add "no meaningful difference".

In other words, if you flipped an honest coin 20 times you would expect a result at least as unbalanced as 11-9 << 10-10 might be the single most likely outcome BUT there are so many other outcomes possible that we expect something else. For example, 11-9 or 9-11, one of these or the other, are together more likely than 10-10>>

[Vargo]

You're absolutely right. As the result was very close, I immediately ran a second 20 game match, with...
the same result.

STATS

200isB.gif (96.6 KiB) Viewed 14133 times

STATS

200isW.gif (95.09 KiB) Viewed 14133 times

200vf438_5min.zip

(17.55 KiB) Downloaded 685 times

But I completely agree with you, it could very well have been 8-12, 9-11, 10-10, 11-9, or 12-8.
What it (probably) means, is they're about the same strength.
On the other hand, even with only 20 games, 17-3 probably means that the winner IS stronger (if they were the same strength, it would be very infrequent to get 17-3)

[nbc44]

...I immediately ran a second 20 game match, with...
the same result.

Now then:

Code: Select all

2019-01-11 08:30  #200 VS f438268e 22 : 18 (55.00%)   40 / 40   PASS

But this is a modest result for bigger network. Long live 15x192 .

[Vargo]

Long live 15x192 .

I'm not rooting for or against #200, I just like to run this kind of matches, and I always give the results exactly as they come.
I don't know your parameters and hardware.

Here is my command line for a 20 game match (twogtp 1.5.0)

Code: Select all

gogui-twogtp -white "C:\***PATH TO LZ***\leelaz.exe --gtp --weights=C:\***PATH TO NETWORK***\200.gz --noponder" -black "C:\***PATH TO LZ***\leelaz.exe --gtp --weights=C:\***PATH TO NETWORK***\f438268.gz --noponder" -time 5 -games 10 -sgffile ***NAME OF SGF*** -auto -komi 7.5
gogui-twogtp -black "C:\***PATH TO LZ***\leelaz.exe --gtp --weights=C:\***PATH TO NETWORK***\200.gz --noponder" -white "C:\***PATH TO LZ***\leelaz.exe --gtp --weights=C:\***PATH TO NETWORK***\f438268.gz --noponder" -time 5 -games 10 -sgffile ***NAME OF SGF*** -auto -komi 7.5

If someone wants to run the "same" match at the same strength, but with a different GPU, it can be done :
time parity at 5min per side per game with my GTX1080 is around 2sec/move, and it corresponds to something like -v 1600 for #200 and -v 5000 for #f438.

Long live LZ