It is currently Tue Dec 10, 2019 1:11 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 425 posts ]  Go to page Previous  1 ... 3, 4, 5, 6, 7, 8, 9 ... 22  Next
Author Message
Offline
 Post subject: Re: LZ's progression
Post #101 Posted: Tue Aug 14, 2018 2:35 pm 
Judan

Posts: 6225
Location: Cambridge, UK
Liked others: 356
Was liked: 3372
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Well, I did a game (it took 2 hours) at large visits and didn't get the result I expected: 15 block #157 as white with 80k playouts beat 40 block 40b_157_360k e2be with 20k playouts which is very close to equal time (15b took 3114s, 40b 3201s). Looks like it was a half pointer but black did a load of inside territory nonsense instead of resigning, should I adjust some parameter?



2nd game 40 block won, same opening for first 27 moves, #157 diverged.


Attachments:
W147B40b157-1.sgf [2.5 KiB]
Downloaded 1192 times
W147B40b157-0.sgf [2.71 KiB]
Downloaded 1240 times
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #102 Posted: Tue Aug 14, 2018 3:52 pm 
Lives in gote

Posts: 450
Liked others: 1
Was liked: 128
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
Here is a 40k vs 10k match I just did :


40d win as white against #157, (and it's close to time parity, with 2407s for B and 2559s for W)

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #103 Posted: Wed Aug 15, 2018 3:59 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
Uberdude wrote:
should I adjust some parameter?
You could maybe add -r 10

Mini 4-game match at time parity (40b_e2be48 v. #157)
5000 visits for e2be48, 20000 visits for #157
#157 wins 4-0
I was very surprised, I would have bet 40b would win , go figure :scratch:

Stats : 40b is B and 40b is W
Attachment:
40b_e2be_157_v5kv20k.zip [4.05 KiB]
Downloaded 59 times


@Uberdude@Tryss :games at 40k or 80 k visits... woaw :tmbup:

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #104 Posted: Wed Aug 15, 2018 5:36 am 
Lives in gote

Posts: 450
Liked others: 1
Was liked: 128
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
If fact, I was mistaken, it's not a 40k/10k playouts match, because the LZ bots were not running at full playouts. By default, LZ thinking time is capped, so the real playouts numbers were lower (I'm not sure by how much, probably a little higher than 10k/2.5k).


I suggest everyone doing this to verify if you're indeed running at the correct amount of playouts

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #105 Posted: Wed Aug 15, 2018 6:55 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
Tryss wrote:
...verify if you're indeed running at the correct amount of playouts
My 4-game match was around 5 sec/move, far from the limit.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #106 Posted: Wed Aug 15, 2018 4:47 pm 
Dies with sente

Posts: 94
Liked others: 2
Was liked: 15
Rank: KGS 2 D
That's indeed the big issue of a large network. Not only more time is needed per playout, but also more training games (possibly much much more training games) are needed to get a stable, good performing network.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #107 Posted: Thu Aug 16, 2018 2:03 am 
Lives with ko

Posts: 263
Liked others: 0
Was liked: 31
Rank: 2d
Even if #157 could win a statistically significant (20+ games) match at 10k visits vs 40k (still seems unlikely), there surely is a visit limit where it falls apart. This is a race between a linear speed advantage vs an exponential search advantage. That's why LZ always entered competitions with experimental 20 and 40 blocks networks. The only question is whether the turning point is within the reach of an average user (which may be around 10-20 sec per move on single 1080ti - both for playing and for analysis/review).

But I would still bet on 40 blocks at 10k vs 40k visits already. One could also test if everything is ok (wrt settings / corrupted network files / etc) by a quick test at equal 1600 visits - this should reproduce the official result of 85+%. (high visits might also need --timemanage off)

Also remember this 40b is just a supervised network that haven't done any selfplay improvement yet. This will also change sooner or later.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #108 Posted: Thu Aug 16, 2018 6:45 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
moha wrote:
One could also test if everything is ok (wrt settings / corrupted network files / etc) by a quick test at equal 1600 visits
Everything seems ok, I did a 20-game check #157 v 1fdfb1 (40b) at --visits=1601.
1fdfb1 won 75% which is reasonable (the "official" score it got was 84.25% : 2018-08-02).
40b is W
40b is B
And 4 additional e2be48 v #157 games at --visits=1601. As expected, e2be48 won them (taking 3 times more time than #157).
40b is W
40b is B
Seing this, I'm still surprised that 40b doesn't perform better at time parity.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #109 Posted: Mon Sep 03, 2018 11:37 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
20 game match between LZ0.15#157 and LZ0.15#173
twogtp 1.4.10, 5 min per side and per game, no ponder, komi 7.5 (GPU 1x1080)

#157 wins 14:6 (8 wins as W, 6 wins as B) all games by resignation

Still a lot of catching up to do for the new networks...

If someone wants the games, I can upload them.


This post by Vargo was liked by: Uberdude
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #110 Posted: Tue Sep 04, 2018 10:42 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
20 game match between LZ0.15#157 andLZ0.15#174 (the new official best network is 256x40)
twogtp 1.4.10, time parity : 5 min per side and per game, no ponder, komi 7.5 (GPU 1x1080)

#157 v #174 --> 10:10 (7 wins as W, 3 wins as B) all games by resignation

A good surprise, 256x40 seems much stronger than the 256x20 series...

I'll run some more tests tomorrow, to be sure it's not a fluke ;-)

Attachment:
157_174_174isW.zip [8.63 KiB]
Downloaded 48 times
Attachment:
157_174_174isB.zip [9.49 KiB]
Downloaded 39 times

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #111 Posted: Wed Sep 05, 2018 12:30 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
40 more games between #157 and #174.

In all, it's a 60 game match, at time parity (5 min per game, GPU: 1x1080, komi 7.5, no pondering)

Final result : #157 wins 35:25 (58% , 17 wins as W, 18 wins as B)

So, maybe #174(256x40) is not as strong as #157, but it seems stronger than the 256x20 networks.


The 40 more games :
Attachment:
157v174_174isW.rar [16.62 KiB]
Downloaded 54 times
Attachment:
157v174_174isB.rar [16.01 KiB]
Downloaded 57 times

Top
 Profile  
 
Online
 Post subject: Re: LZ's progression
Post #112 Posted: Wed Sep 05, 2018 6:27 am 
Lives in gote

Posts: 594
Liked others: 253
Was liked: 306
Vargo wrote:
#157 wins 35:25


On http://zero.sjeng.org/ #157 has ELO 11806 but #174 has 12463. How is it possible for the last 15-block network to still win against all the networks with higher ELO? And almost all of the networks in-between had a win rate of at least 55% (some winrates dropped later) against the next-lower network.

Does this mean even within LZ's ELO scale the rating is not so meaningful?

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #113 Posted: Wed Sep 05, 2018 6:38 am 
Gosei

Posts: 1347
Location: Earth
Liked others: 511
Was liked: 218
Vargo is doing time parity

Matches on https://zero.sjeng.org/ are played at 1600 visits


This post by Gomoto was liked by: Waylon
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #114 Posted: Wed Sep 05, 2018 7:26 am 
Judan

Posts: 6225
Location: Cambridge, UK
Liked others: 356
Was liked: 3372
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Even without the extra time advantage the larger networks get with equal playouts in test matches, the Leelo scale from successive 55% promotions is highly inflated. Based on comparisons to Elf I estimated it as around a factor of 5. So if one network is 500 above another the Elo formula says it'd win 95% but in reality it's more likely a 100 difference for 65% (I don't have the resources to actually do a test, it's possible LZ is particularly bad against Elf compared to old versions of itself). Another way to get a similar ball-park figure: Top pros are 3600 on goratings which is kind of a continuation of EGf ratings where a beginner is about 0, whilst LZ is about 12000 now and started at 0 for random (lower than beginner), and 12000 / 3600 is about 5 too.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #115 Posted: Wed Sep 05, 2018 7:41 am 
Gosei

Posts: 1347
Location: Earth
Liked others: 511
Was liked: 218
have a look at the recent matches, the network is now the offical top dog

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #116 Posted: Wed Sep 05, 2018 7:42 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
A little postscriptum :

5 min per game with 1x1080 is roughly equivalent to --visits=3201 for #157, and to --visits=801 for #174

At time parity, #157 has 4 times more visits and wins.
At visits parity, #174 takes 4 times more time than #157 and wins.

I still feel it would be more natural to determine Elo at time parity (but maybe it would be difficult to do ?)


This post by Vargo was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #117 Posted: Wed Sep 05, 2018 8:19 am 
Gosei

Posts: 1347
Location: Earth
Liked others: 511
Was liked: 218
Does anybody know why visits are used instead of time?

I think because different hardware does not matter this way.

I also think this is a possible error source for further improvement of the networks.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #118 Posted: Thu Sep 06, 2018 2:25 pm 
Dies in gote

Posts: 41
Liked others: 2
Was liked: 14
Rank: EGF 1 kyu
KGS: finity
Match games are run with visit parity because the architecture was originally designed for fixed size net, so time and visit parity would be essentially the same.

With the Leela Zero project, there has been a size upgrade every few months or so, and as the change is usually done manually, it doesn't matter since all games after that are again at time parity. Breaks the ELO graph though (or not if you would want visit parity). Self-play ELO graph is not absolute in any case but kind of relative, so this is probably not seen as such a huge issue.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #119 Posted: Wed Sep 12, 2018 9:52 am 
Lives with ko

Posts: 242
Liked others: 4
Was liked: 57
40 games between #157 and #176.

Time parity, 5 min per game, GPU: 1x1080, komi 7.5, no pondering.

#157 wins 29:11 (17 wins as W, 12 wins as B)

Well, almost 2 months since #157... am I the only one to be so disappointed in the new networks ?

Could someone run a 157 v 176 match (at time-parity, with no pondering), just to be sure of these results.

Attachment:
176isW.zip [16.83 KiB]
Downloaded 36 times
Attachment:
176isB.zip [17.46 KiB]
Downloaded 51 times

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #120 Posted: Wed Sep 12, 2018 1:50 pm 
Lives with ko

Posts: 263
Liked others: 0
Was liked: 31
Rank: 2d
Vargo wrote:
Well, almost 2 months since #157... am I the only one to be so disappointed in the new networks ?
You probably won't see a fast improvement in these 1s/move games, even if the slower networks are getting stronger and stronger, because that strength still needs a meaningful sized search tree to do it's work. Below a certain limit more search beats smarter search, no way around that.

But this doesn't mean the new networks are weaker at "time parity" in general - just in these very fast games.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 425 posts ]  Go to page Previous  1 ... 3, 4, 5, 6, 7, 8, 9 ... 22  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group