LZ's progression

[Vargo]

20-games match between LZ_0.15#165 and LZ_0.15#157 at visits parity.

twogtp V1.4.10 --noponder --visits=1601 -komi 7.5 for both.
If you look at the .dat reports, you'll see that #165 takes at least twice as long as #157 to think.

#165 wins 14-6 (9 times as W, 5 times as B, all games by resignation)

reports.zip

(1.14 KiB) Downloaded 634 times

165_157_v1601_157isBlack.zip

(8.82 KiB) Downloaded 646 times

165_157_v1601_157isWhite.zip

(8.69 KiB) Downloaded 606 times

Could you try #165? It is noticeably more aggressive than before.

Ok. Tomorrow, I'll run a 20-games match at time parity (#165 v #157). It will be interesting to compare results at visits parity and at time parity.

[splee99]

Thanks for the report. I had run one game between #165 and #157 with time parity and #157 won. However the battle was so heavy that only bots can deal with it. Anyway I hope we can see some progress from #161 to #165 from your test tomorrow.

[Vargo]

40-game match at time parity between #165 and #157
(40 games, because I made a mistake, the first 20 games were all with #157 as White, so I had to run 20 more)

5 min per game, komi 7.5, no pondering, 1xGTX1080, twogtp V1.4.10
5 min per game on my computer is similar to 1600 visits for #165 or 3200 visits for #157

#157 wins 27-13 (15 times as W, 12 times as B, all games by resignation)

13 wins for #165, that's 32.5 %, much better than #161, which scored only 15% in a 20-game match at 5 min against #157.

The 40 games :

163_157_5min_157isWhite.zip

(17.74 KiB) Downloaded 590 times

165_157_5min_157isBlack.zip

(18.34 KiB) Downloaded 606 times

[Uberdude]

There's a new 40block network out: http://zero.sjeng.org/networks/e2be4815 ... 1c836a3.gz. I wonder how would this would do vs #157 on 5 mins time parity? It got a very respectable 42% vs Elf v1 (presumably at visits parity) and 83% vs #162. So in a 5 mins game it'll probably get about 800 visits (Edit: is that true, it seems not approx. linear in blocks if 15 to 20 is a halving of visits!) to #157's 3200. Will its superior intuition and judgement win out, or will it make tactical blunders like ladders due to not enough playouts and lose?

[Vargo]

There's a new 40block network out: http://zero.sjeng.org/networks/e2be4815 ... 1c836a3.gz. I wonder how would this would do vs #157

20-game match between LZ0.15 with e2be48 (256x40) and LZ0.15 with #157 (192x15)

e2be48 at 1600 visits, and #157 at 6400 visits.

I think time parity is very roughly :
1x visits for 256x40 or
2x visits for 256x20 or
4x visits for 192x15

So, e2be48 at 1600 visits v. #157 at 6400 visits is at time parity.

#157 wins 11-9 (5 times as W, 6 times as B)

20 games.zip

(18.62 KiB) Downloaded 601 times

I just realized (better late than never !) that twogtp can give a nice and complete .html report, with min, max, standard deviation...

LIKE THIS

AND THIS

With 2xGTX1080Ti,
Average time used per game is around 3min10sec (see html for exact numbers)
Average length is around 220 moves

[Tryss]

So the latest 40b has about the same strenght as #157 at time parity ? Nice !

[moha]

So the latest 40b has about the same strenght as #157 at time parity ? Nice !

I doubt this could be said in a general sense.

So, e2be48 at 1600 visits v. #157 at 6400 visits is at time parity.

#157 wins 11-9 (5 times as W, 6 times as B)

The point of these deep and strong nets is to direct the search, shape the search tree. But with 1600 visits you don't have much of a search tree to be shaped (just a few moves lookahead). In those cases the bot won't be really strong anyway (even AGZ was only high dan amateur without search), and the most important is getting more visits (which #157 does).

But in serious games these bots do a lot more visits per move on good gpus. I suggest you repeat the test with at least 10k visits vs 40k. I would be surprised if 40b won't crush #157.

[Vargo]

I doubt this could be said in a general sense.

You must be right, because I've tested the new 40b (e2be48) against #157 at time parity : only 5min on 1x1080.
For e2be48, it corresponds to only 500-600 visits(?)
#157 wins 14-6
6 wins as W here , and 8 wins as B

Maybe I'll try tomorrow a mini match at time parity, with 3200 or 6400 visits for e2be48.
e2be48 should be stronger, we'll see.

[Uberdude]

Well, I did a game (it took 2 hours) at large visits and didn't get the result I expected: 15 block #157 as white with 80k playouts beat 40 block 40b_157_360k e2be with 20k playouts which is very close to equal time (15b took 3114s, 40b 3201s). Looks like it was a half pointer but black did a load of inside territory nonsense instead of resigning, should I adjust some parameter?



2nd game 40 block won, same opening for first 27 moves, #157 diverged.

[Tryss]

Here is a 40k vs 10k match I just did :


40d win as white against #157, (and it's close to time parity, with 2407s for B and 2559s for W)

[Vargo]

should I adjust some parameter?

You could maybe add -r 10

Mini 4-game match at time parity (40b_e2be48 v. #157)
5000 visits for e2be48, 20000 visits for #157
#157 wins 4-0
I was very surprised, I would have bet 40b would win , go figure

Stats : 40b is B and 40b is W

40b_e2be_157_v5kv20k.zip

(4.05 KiB) Downloaded 599 times

@Uberdude@Tryss :games at 40k or 80 k visits... woaw

[Tryss]

If fact, I was mistaken, it's not a 40k/10k playouts match, because the LZ bots were not running at full playouts. By default, LZ thinking time is capped, so the real playouts numbers were lower (I'm not sure by how much, probably a little higher than 10k/2.5k).


I suggest everyone doing this to verify if you're indeed running at the correct amount of playouts

[Vargo]

...verify if you're indeed running at the correct amount of playouts

My 4-game match was around 5 sec/move, far from the limit.

[splee99]

That's indeed the big issue of a large network. Not only more time is needed per playout, but also more training games (possibly much much more training games) are needed to get a stable, good performing network.

[moha]

Even if #157 could win a statistically significant (20+ games) match at 10k visits vs 40k (still seems unlikely), there surely is a visit limit where it falls apart. This is a race between a linear speed advantage vs an exponential search advantage. That's why LZ always entered competitions with experimental 20 and 40 blocks networks. The only question is whether the turning point is within the reach of an average user (which may be around 10-20 sec per move on single 1080ti - both for playing and for analysis/review).

But I would still bet on 40 blocks at 10k vs 40k visits already. One could also test if everything is ok (wrt settings / corrupted network files / etc) by a quick test at equal 1600 visits - this should reproduce the official result of 85+%. (high visits might also need --timemanage off)

Also remember this 40b is just a supervised network that haven't done any selfplay improvement yet. This will also change sooner or later.