It is currently Thu Mar 28, 2024 2:13 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1 ... 3, 4, 5, 6, 7, 8, 9 ... 21  Next
Author Message
Offline
 Post subject: Re: LZ's progression
Post #101 Posted: Wed Aug 15, 2018 3:59 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
Uberdude wrote:
should I adjust some parameter?
You could maybe add -r 10

Mini 4-game match at time parity (40b_e2be48 v. #157)
5000 visits for e2be48, 20000 visits for #157
#157 wins 4-0
I was very surprised, I would have bet 40b would win , go figure :scratch:

Stats : 40b is B and 40b is W
Attachment:
40b_e2be_157_v5kv20k.zip [4.05 KiB]
Downloaded 427 times


@Uberdude@Tryss :games at 40k or 80 k visits... woaw :tmbup:

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #102 Posted: Wed Aug 15, 2018 5:36 am 
Lives in gote

Posts: 502
Liked others: 1
Was liked: 153
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
If fact, I was mistaken, it's not a 40k/10k playouts match, because the LZ bots were not running at full playouts. By default, LZ thinking time is capped, so the real playouts numbers were lower (I'm not sure by how much, probably a little higher than 10k/2.5k).


I suggest everyone doing this to verify if you're indeed running at the correct amount of playouts

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #103 Posted: Wed Aug 15, 2018 6:55 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
Tryss wrote:
...verify if you're indeed running at the correct amount of playouts
My 4-game match was around 5 sec/move, far from the limit.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #104 Posted: Wed Aug 15, 2018 4:47 pm 
Dies with sente

Posts: 101
Liked others: 2
Was liked: 16
Rank: KGS 2 D
That's indeed the big issue of a large network. Not only more time is needed per playout, but also more training games (possibly much much more training games) are needed to get a stable, good performing network.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #105 Posted: Thu Aug 16, 2018 2:03 am 
Lives in gote

Posts: 311
Liked others: 0
Was liked: 45
Rank: 2d
Even if #157 could win a statistically significant (20+ games) match at 10k visits vs 40k (still seems unlikely), there surely is a visit limit where it falls apart. This is a race between a linear speed advantage vs an exponential search advantage. That's why LZ always entered competitions with experimental 20 and 40 blocks networks. The only question is whether the turning point is within the reach of an average user (which may be around 10-20 sec per move on single 1080ti - both for playing and for analysis/review).

But I would still bet on 40 blocks at 10k vs 40k visits already. One could also test if everything is ok (wrt settings / corrupted network files / etc) by a quick test at equal 1600 visits - this should reproduce the official result of 85+%. (high visits might also need --timemanage off)

Also remember this 40b is just a supervised network that haven't done any selfplay improvement yet. This will also change sooner or later.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #106 Posted: Thu Aug 16, 2018 6:45 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
moha wrote:
One could also test if everything is ok (wrt settings / corrupted network files / etc) by a quick test at equal 1600 visits
Everything seems ok, I did a 20-game check #157 v 1fdfb1 (40b) at --visits=1601.
1fdfb1 won 75% which is reasonable (the "official" score it got was 84.25% : 2018-08-02).
40b is W
40b is B
And 4 additional e2be48 v #157 games at --visits=1601. As expected, e2be48 won them (taking 3 times more time than #157).
40b is W
40b is B
Seing this, I'm still surprised that 40b doesn't perform better at time parity.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #107 Posted: Mon Sep 03, 2018 11:37 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
20 game match between LZ0.15#157 and LZ0.15#173
twogtp 1.4.10, 5 min per side and per game, no ponder, komi 7.5 (GPU 1x1080)

#157 wins 14:6 (8 wins as W, 6 wins as B) all games by resignation

Still a lot of catching up to do for the new networks...

If someone wants the games, I can upload them.


This post by Vargo was liked by: Uberdude
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #108 Posted: Tue Sep 04, 2018 10:42 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
20 game match between LZ0.15#157 andLZ0.15#174 (the new official best network is 256x40)
twogtp 1.4.10, time parity : 5 min per side and per game, no ponder, komi 7.5 (GPU 1x1080)

#157 v #174 --> 10:10 (7 wins as W, 3 wins as B) all games by resignation

A good surprise, 256x40 seems much stronger than the 256x20 series...

I'll run some more tests tomorrow, to be sure it's not a fluke ;-)

Attachment:
157_174_174isW.zip [8.63 KiB]
Downloaded 412 times
Attachment:
157_174_174isB.zip [9.49 KiB]
Downloaded 376 times

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #109 Posted: Wed Sep 05, 2018 12:30 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
40 more games between #157 and #174.

In all, it's a 60 game match, at time parity (5 min per game, GPU: 1x1080, komi 7.5, no pondering)

Final result : #157 wins 35:25 (58% , 17 wins as W, 18 wins as B)

So, maybe #174(256x40) is not as strong as #157, but it seems stronger than the 256x20 networks.


The 40 more games :
Attachment:
157v174_174isW.rar [16.62 KiB]
Downloaded 423 times
Attachment:
157v174_174isB.rar [16.01 KiB]
Downloaded 376 times

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #110 Posted: Wed Sep 05, 2018 6:38 am 
Gosei

Posts: 1733
Location: Earth
Liked others: 621
Was liked: 310
Vargo is doing time parity

Matches on https://zero.sjeng.org/ are played at 1600 visits


This post by Gomoto was liked by: Waylon
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #111 Posted: Wed Sep 05, 2018 7:26 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Even without the extra time advantage the larger networks get with equal playouts in test matches, the Leelo scale from successive 55% promotions is highly inflated. Based on comparisons to Elf I estimated it as around a factor of 5. So if one network is 500 above another the Elo formula says it'd win 95% but in reality it's more likely a 100 difference for 65% (I don't have the resources to actually do a test, it's possible LZ is particularly bad against Elf compared to old versions of itself). Another way to get a similar ball-park figure: Top pros are 3600 on goratings which is kind of a continuation of EGf ratings where a beginner is about 0, whilst LZ is about 12000 now and started at 0 for random (lower than beginner), and 12000 / 3600 is about 5 too.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #112 Posted: Wed Sep 05, 2018 7:41 am 
Gosei

Posts: 1733
Location: Earth
Liked others: 621
Was liked: 310
have a look at the recent matches, the network is now the offical top dog

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #113 Posted: Wed Sep 05, 2018 7:42 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
A little postscriptum :

5 min per game with 1x1080 is roughly equivalent to --visits=3201 for #157, and to --visits=801 for #174

At time parity, #157 has 4 times more visits and wins.
At visits parity, #174 takes 4 times more time than #157 and wins.

I still feel it would be more natural to determine Elo at time parity (but maybe it would be difficult to do ?)


This post by Vargo was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #114 Posted: Wed Sep 05, 2018 8:19 am 
Gosei

Posts: 1733
Location: Earth
Liked others: 621
Was liked: 310
Does anybody know why visits are used instead of time?

I think because different hardware does not matter this way.

I also think this is a possible error source for further improvement of the networks.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #115 Posted: Thu Sep 06, 2018 2:25 pm 
Dies in gote

Posts: 44
Liked others: 2
Was liked: 14
Rank: EGF 1 kyu
KGS: finity
Match games are run with visit parity because the architecture was originally designed for fixed size net, so time and visit parity would be essentially the same.

With the Leela Zero project, there has been a size upgrade every few months or so, and as the change is usually done manually, it doesn't matter since all games after that are again at time parity. Breaks the ELO graph though (or not if you would want visit parity). Self-play ELO graph is not absolute in any case but kind of relative, so this is probably not seen as such a huge issue.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #116 Posted: Wed Sep 12, 2018 9:52 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
40 games between #157 and #176.

Time parity, 5 min per game, GPU: 1x1080, komi 7.5, no pondering.

#157 wins 29:11 (17 wins as W, 12 wins as B)

Well, almost 2 months since #157... am I the only one to be so disappointed in the new networks ?

Could someone run a 157 v 176 match (at time-parity, with no pondering), just to be sure of these results.

Attachment:
176isW.zip [16.83 KiB]
Downloaded 384 times
Attachment:
176isB.zip [17.46 KiB]
Downloaded 374 times

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #117 Posted: Wed Sep 12, 2018 1:50 pm 
Lives in gote

Posts: 311
Liked others: 0
Was liked: 45
Rank: 2d
Vargo wrote:
Well, almost 2 months since #157... am I the only one to be so disappointed in the new networks ?
You probably won't see a fast improvement in these 1s/move games, even if the slower networks are getting stronger and stronger, because that strength still needs a meaningful sized search tree to do it's work. Below a certain limit more search beats smarter search, no way around that.

But this doesn't mean the new networks are weaker at "time parity" in general - just in these very fast games.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #118 Posted: Thu Sep 13, 2018 2:10 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
All these time parity matches are 5min per game and per side, so, it's a bit more than 2 sec per move (with 1x1080), which corresponds roughly to 800 visits for #176, and to 3200 visits for #157.
The matches I ran with longer time settings for other networks had similar results...
Anyway, I've begun a new 157 v 176 match, with 2x1080Ti, 12800 visits for 157 and 3200 visits for 176, which should be approximately at time parity (I'll check, with the .dat)

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #119 Posted: Thu Sep 13, 2018 3:42 am 
Lives in gote

Posts: 311
Liked others: 0
Was liked: 45
Rank: 2d
Vargo wrote:
All these time parity matches are 5min per game and per side, so, it's a bit more than 2 sec per move (with 1x1080), which corresponds roughly to 800 visits for #176, and to 3200 visits for #157.
I think in this visit range more search still beats better search. There are around 300 candidate MOVES in a position, so (even if most of them are pruned) this doesn't mean a significantly deep search. Looking a bit deeper is more valuable than the order you FIRST look at the moves (which is all network strength is about).

So in this range I won't expect to see a spectacular improvement between successive networks even if they actually improve. Just lowering official matches from 3200 to 1600 visits had a noticeable negative effect (results more random, promotions scarcer even on a new size).

Quote:
Anyway, I've begun a new 157 v 176 match, with 2x1080Ti, 12800 visits for 157 and 3200 visits for 176, which should be approximately at time parity (I'll check, with the .dat)
Thanks, this will be interesting. I never saw a statistically significant 40b vs 15b match at more realistic time controls (someone posted similar results on github but also only 1-2 sec/move).

OC testing like this is faster, but if someone uses LZ for serious analysis, he probably would allow at least 10-20 sec per move on 1080ti (nearly 10k visits - which is where search quality should start to overcome the visit disadvantage). And the tournaments these high block nets were first used saw much more visits. On the other hand it is good to know that users with weaker hardware are better off with 15 blocks for now. There probably will be unofficial 15b nets trained on 40 block selfplay data in the future as well.


Last edited by moha on Thu Sep 13, 2018 6:03 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #120 Posted: Thu Sep 13, 2018 4:33 am 
Oza
User avatar

Posts: 2408
Location: Ghent, Belgium
Liked others: 359
Was liked: 1019
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
Has LZ also built up a model of the game for itself? Has AlphaGo? I'm confused as to the AI aspect. I understand how it uses MTCS and NN to solve the computation problem, but there's no AI in there, is it? Do these programs always rebuild their intelligence particular to the game? Or has LZ also trained itself like AG has? And what has been the result of AG's training? Did it have an impact only on the MTCS and NN parameters? Or did it rebuild some domain knowledge for itself?

Any articles on that?

Sorry to sound confused.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1 ... 3, 4, 5, 6, 7, 8, 9 ... 21  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group