LZ's progression

[Vargo]

could you share the link

HERE

this sort of thing requires REASONABLE time controls

With 2min/game and good gpu, these networks are already (very) strong, I don't see why it's not reasonable.



I've let it run for 6x20 more games.

After 6x30 games, results are :

202-Elf 12-18
202-f43 17-13
202-x89 12-18
x89-Elf 12-18
x89-f43 14-16
Elf-f43 17-13

I'll let it run up to 6x50 games, results in the evening.
_________________________________________________________

Assigning arbitrarily 4000 Elo to each network at the start of the tournament, after 30 games, I get

1)Elf_ v1 4047
2)LM_GX89 3994
3)f438268 3981
4)LZ_#202 3978

(and yes, I know it's questionable )

[And]

when using gpu 1080, these networks play very strongly and at 2 min/game. even using cpu only, will many be able to defeat them?
but the results are different if you use only cpu, for example:

100 games f438268e - elf.v1 62:38 (5 sec/move, no pondering, LZ0.15-cpuonly, twogtp V1.4.9)

for an elf there is not enough computational power, it is probably better to take a 15 block network for a CPU only (with little time control)
that is, when the strength of the networks does not differ much, the results of the matches depend not only on the number of games (if there are few of them, then the result is doubtful), but also from time to move + power gpu / cpu

[Vargo]

After 6x50 games, results are :

202-Elf 21-29
202-f43 21-29
202-x89 28-22
x89-Elf 22-28
x89-f43 26-24
Elf-f43 27-23

A little oddity : usually, all games are by resignation, but there was one won by 1.5 after... 423 moves

gx89_elf.jpg (143.29 KiB) Viewed 11635 times

After 6x50 games, starting with 4000 for each network, I get relative Elos :

1)Elf_ v1 4034
2)LM_GX89 4003
3)f438268 3985
4)LZ_#202 3978
but 6x50 games is too few to be really meaningful.

[Mike Novack]

Maybe I should make clearer what I am saying.

The whole point of this is determining the question is A stronger than B. It is entirely possible that A runs on less resources than B. In other words, at short enough time control, A "has enough to be strong" while B does not. However while A would be stronger given more time, the rate at which it gets stronger might be less than the rate at which B gets stronger with more time.

That is why am suggesting should be reasonable time controls (no faster than human blitz)

There is ANOTHER question that we could be asking. There is trade off between strength of hardware and time. We MIGHT be asking "what is the strongest program, playing at reasonable time controls, but on weaker hardware". THAT we can model at shorter time controls (like being used here) on this powerful hardware. The "weaker hardware" question IS important to many of us. We just need to be clear about what question we are asking.

[Vargo]

Hardware, parameters and time control are all reasonable, and you know it. Other tests with weaker or stronger hardware, with different networks, or different time controls are obviously possible. Feel free to show us your results, it would be interesting.

[Satorian]

Of course everyone is free to do as desired with their own processing power and time, but I agree with Mike that a super short playing/move time has a bias towards smaller/quicker networks by allowing them more playouts within a generally low-playout regime, while those networks would plateau much quicker with longer, more reasonable time controls. I also don't find these super short time controls very interesting. Using an engine for review I would probably give it at least 30-60 minutes total or 10-15s per move, same for playing against it.

[Uberdude]

How many playouts a move is 40 block LZ getting in these 2 minute test games on a 1080 Ti? If it's at least 3k (what I get in 20 seconds on my 1060, and would take 5 minutes on my crappy laptop) then I think that's about enough for a test where the deeper network can show its strength and not be handicapped by being unable to do basic reading (there was a nice chart on the LZ github showing this I linked a while back).

[Satorian]

At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.

[And]

Anyone with a good gpu can test the minigo network?
2019-01-10 12:49 50b98433(40x256) VS d13c4099(elf 20x224) 266 : 134 (66.50%) - elf 33.5%
2019-01-24 03:56 b9efeafa(minigo 19x256) VS ffe8ba44(40x256) 30 : 53 (36.14%) - minigo 36.14%
http://zero.sjeng.org

Minigo v15-990 was stronger than ELFv1 at 10s and 120s (10s/ 120s was at least 10k/ 120k-po on ELFv1)/ a move.
https://github.com/gcp/leela-zero/issues/2179

[Uberdude]

At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.

I asked a similar question a while back so we can estimate it:

20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6 (8 wins as B, 6 wins as W)

So that's about 800 playouts for #184 and 1600 for Elfv1?

So that's about 800 playouts for #184 and 1600 for Elfv1?

Something like 800 visits for #184 and 2000 visits for Elfv1

So that was 5 mins per side per game, these test are 1 minute. 184 is a 40 block so (assuming there isn't an LZ 0.16 fp16 speedup between the two tests, and there may be) 800/5 = 160 so a lot short of the several thousand I like to give the 40 blocks to show their strength. If there is fp16 difference that'll help at most a factor of 2.

[And]

MiniGo v15:
https://cloudygo.com/converted_model/v15-19x19/models/

[lichigo]

Can we use this txt.gz file with lizzie by replacing the weight file?
I heard minigo is very strong.

[And]

yes, i tried 990, ok

[Uberdude]

I just used 000513-revenge in a 40 second game on Fox, beat a Fox 9d who I presume was also a bot by half a point (benefiting from being black with 6.5 komi?). viewtopic.php?p=241098#p241098

[seigenblues]

The latest converted v16 model, 000513-revenge, has a rating of 4202, according to https://cloudygo.com/v16-19x19/eval-graphs .

Now I'm not sure whether these ratings are comparable or whether they are within one run. That is, one model may be rated 4430 according to the v15 scale, and the other model may be rated 4202 according to the v16 scale, but I'm not sure whether these scales are the same.

Hi! Minigo author here The ratings are not directly comparable. We'll get some cross-run ratings in a bit when v16 is finished. But I'd love any help folks can provide in figuring out what the best settings to use and how to run it effectively -- we don't really have the charter to make things as strong as possible, so i'm hoping folks can help us there, but that's another story