It is currently Thu Mar 28, 2024 1:56 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1 ... 6, 7, 8, 9, 10, 11, 12 ... 21  Next
Author Message
Offline
 Post subject: Re: LZ's progression
Post #161 Posted: Sun Oct 21, 2018 9:54 pm 
Beginner

Posts: 12
Liked others: 5
Was liked: 0
Rank: 15k
Ah, but I have no GPU, and to get a game that takes less than two hours I have to restrict LZ considerably more than you did. On my computer, time parity is 600 visits for LZ 157 and 153 visits for LZ 183. Does 183 still take the lead under those conditions?

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #162 Posted: Mon Oct 22, 2018 1:07 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
abcd_z wrote:
Does 183 still take the lead under those conditions?

Most probably no.

I think 157 would win at least 60% of the games with so few visits.

Maybe I'll set up such a match and tell you the result ;-)


EDIT :
Here it is !
#157 v #184 (the latest best) 600 visits for #157 and 153 visits for #184

# 157 wins 14-6 (70%)

it's even more than 60 % :D


Last edited by Vargo on Mon Oct 22, 2018 4:29 am, edited 1 time in total.

This post by Vargo was liked by: abcd_z
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #163 Posted: Mon Oct 22, 2018 4:24 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
I wonder how does #183 do versus Elf v1 at time-parity? I saw it's now stronger (57% win) at visit parity in the test match: http://zero.sjeng.org/match-games/5bcaa ... 3e27abce47.

P.S. some interesting analysis of different nets strength scaling with time: https://github.com/gcp/leela-zero/issues/1914. The relevant summary for here is around 20-300 playouts for 40b #181 the best 15 block network #157 needs 3 times the playouts to score 50% win, but is ~4 times faster so is stronger at equal time (which given current hardware is sensible time per move for human analysis). Once you get into more playouts 181's strength improves faster than the 15 block so 157 needs more than 3 and then crosses the equal time ~4 times threshold and 40block wins more.

A randomly-chosen LZ #183 (a54cd) win (Elf got captured in a ladder! :lol: ):

Analysing a bit with Elfv1 myself, Elf thought it was doing slightly better (55%) at n15 extend. LZ's n15 is a move which surprised and impressed me (thought shares features with a move Blackie showed me at KPMC) and Elf similarly thinks it's good (it starts off not considering it much but by 1200 playouts thinks it's better than normal l17 defence), does this mean push and cut wasn't best? But the big swing happens (q12 and q10 attachments surprised me but not Elf) with LZ starting the ko with s12 (Elf win collapse to 13%) instead of r13 (Elf win 45%). I suppose r13 is more often the better shape way to start the ko, but here it gives white decent local threats and a playable position. As for the ladder, at move 72 it thinks black will n8 atari allowing white to trade with p7 atari then o7 connect and ko (k15 is ignored threat), despite p12 atari for squeezy ladder having a few playouts and the ladder all the way to the edge of the board being the principal variation.

An Elf v1 win:


P.P.S Had a look at a few more LZ wins, quite a few are Elf blunders of ladders or shortage of liberties (though maybe Elf already thought it was losing from previous non-blunders so is doing bot harakiri). Seems like the 1600 playout limit is not enough to stop Elf making dumb mistakes, maybe Elf would do better at say 10k each. But in my experience it can be quite ladder blunder and blindspot-prone even at higher playouts.


This post by Uberdude was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #164 Posted: Mon Oct 22, 2018 4:41 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
I've launched a 20 game match at 5 min per side per game with GPU 1080. Hopefully, result in 2-3 hours.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #165 Posted: Mon Oct 22, 2018 5:32 am 
Dies in gote

Posts: 50
Liked others: 0
Was liked: 3
My old test:
Code:
The first net is worse than the second
#157a v #181 ( 398 games)
           wins        black       white
#157a  194 48.74%   85 48.57%  109 48.88%
#181   204 51.26%   90 51.43%  114 51.12%
                   175 43.97%  223 56.03%

#157a is the strongest 15x192 net:
Code:
2018-07-23 00:13 fc5e0a50 VS d351f06e 220 : 192 (53.40%) 412 / 400 fail)

L0 015 options (2 x 1080ti):
Code:
-g -v 4801 --gpu 1 --gpu 0 --noponder -t 12 -q -d --timemanage off -w C:\APPS\net\fc5e0a50.gz
-g -v 1601 --gpu 0 --gpu 1 --noponder -t 12 -q -d --timemanage off -w C:\APPS\net\68824bbc.gz

Poor #157a network :sad:, but...
x-axis: # of game,
y-axis: winrate of #157a net (0 == 50%)
Attachment:
match157a-181.png
match157a-181.png [ 35.29 KiB | Viewed 9296 times ]


How many games are needed to determine the winner?

:scratch:

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #166 Posted: Mon Oct 22, 2018 6:45 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
nbc44, were the games for your graph played in series of parallel? One effect that's been observed in the official LZ test matches is when the games are played in parallel and start around the same time (as they are with distributing the match tasks to LZ volunteers) then there can be a bias for the early games to be losses by the player who makes early blunders. So for example in Elf vs LZ matches when the games go to counting Elf tends to win, but Elf will blunder ladders and resign early so these data points have a bias to arrive earlier and makes LZ's winrate after 50 games be higher than it is after the full 400.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #167 Posted: Mon Oct 22, 2018 7:16 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6
(8 wins as B, 6 wins as W)

Average game length : ~209 moves, average time used per side and per game : ~205"

Maybe I'll try a 20 game match with 2x1080TI, and 7 min games, i.e. 4-5 times more visits for both.


Attachments:
184_elf2_elf2isW.rar [7.95 KiB]
Downloaded 410 times
184_elf2_elf2isB.rar [7.91 KiB]
Downloaded 411 times
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #168 Posted: Mon Oct 22, 2018 8:17 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Vargo wrote:
20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6
(8 wins as B, 6 wins as W)


So that's about 800 playouts for #184 and 1600 for Elfv1?

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #169 Posted: Mon Oct 22, 2018 8:37 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
I looked at a few more of the official test match Elf v1 vs LZ #183 match at 1600 playouts and thought I'd characterise a few.
- Game #1: LZ 183 won by resign in 93 moves, Elf blundered itself in shortage of liberties and then walked into net harakiri. At move 61 it expects white to block instead of capture with the throw in as LZ and any 10k could find. Strangely it then planned to r18 block and then expected white to k19, so it could see this tesuji, but dreamed that it could get j15 kikashi first?

- #402 (on previous post): LZ 183 won by resign in 92 moves (given the many 92/93 moves games I suppose that's the earliest they are allowed to resign), Elf got laddered. Elf thought it was doing slightly better (55%) at n15 extend. LZ's n13 is a move which surprised and impressed me (thought shares features with a move Blackie showed me at KPMC) and Elf similarly thinks it's good (it starts off not considering it much but by 1200 playouts thinks it's better than normal l17 defence), does this mean push and cut wasn't best? Some other games have same opening. But the big swing happens (q12 and q10 attachments surprised me but not Elf) with LZ starting the ko with s12 (Elf win collapse to 13%) instead of r13 (Elf win 45%). I suppose r13 is more often the better shape way to start the ko, but here it gives white decent local threats and a playable position. As for the ladder, at move 72 it thinks black will n8 atari allowing white to trade with p7 atari then o7 connect and ko (k15 is ignored threat), despite p12 atari for squeezy ladder having a few playouts and the ladder all the way to the edge of the board being the principal variation.

- #25 LZ 183 won in 112 moves. Not obvious Elf blunder, LZ won an early fight and semeai; I was impressed Elf even managed to make a semeai it lost by 1 liberty instead of just simply dying. Same opening for a while as #402.

- #59 LZ won in 93 moves. Elf got caught in an early ladder, game devolved to nonsense because not allowed to resign.Before white played atari for ladder Elf initially (1 playout) though it was ~65% but this dropped to about 60% by 1600. Before k13 on my weak PC Elf wanted to g16 peep, but k13 soon overtakes in winrate but not playouts. So this is a problem caused by Elf assuming ladders not working, wasting some of the limited playouts on some sente move prior to the failed ladder escape, and then going "hey wait a minute, if I run out this stone it's great for me" because Elf didn't have enough playouts remaining to read it to the edge of the board and capture. With just over 2k playouts it realizes it's a ladder and doesn't want to escape, so LZ is probably benefiting from being in a sweet spot in which it has enough playouts to read ladders whilst Elf doesn't.

- #5 Elf won in relatively short 129 moves. Nice fight with a ko, Elf thought it was losing (40%) when ko started but it found a cool 1st line threat and LZ's g5 threat was no good.

- #2 LZ won in 108 moves, but not Elf blunder: LZ push and clamp of 80 was a sharp tesuji. Elf resisted, led to a 30-move fight which LZ won by a liberty. I'd expect high dans to be able to read that as not too much branching, I could maybe on a good day, but probably wouldn't consider the clamp. Reviewing with Elf at 1600 playouts it thought not much deviation around black (Elf) 50-47% until e13 turn (did seem a bit thank you move to push behind on 4th line!) and at 2k playouts it decides f16 nice table shape is quite a bit better instead of going down to 42%. h15 cut I didn't understand (h17/k17 aim?) and Elf despite playing it Elf thinks it's bad (-10%) and just after 1600 realises e12 directly is better. But then it's back to almost even again when LZ doesn't j13 cut but k18. But with the LZ's push and clamp it did look at that a bit but soon after decides tenuki to s14 is better. As for the fight, Elf's prefers to push at h3 for g2 and expects white to g4 not g3: g3 leaving many cuts but taking a liberty and putting black in deep trouble. Once g3 blindspot played winrate drops.

- #3 Elf won in 198 moves, nice game from both, impressive tesujis from Elf on right lower corner, and then seal the win with kill on left.

- #76. LZ won in 92 moves, same opening 51 moves as #5, then same ko, "cool" 1st line threat was a fake theat and LZ didn't believe it this time! I tried analysing with LZ 183, and indeed on this game it quickly (<100 playouts) sees the threat as fake and wants to connect (but thinks winning even if answer threat), whereas with game 5 it took longer (about 400 playouts, so still within the 1600, I guess just bad luck in the real game).


This post by Uberdude was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #170 Posted: Mon Oct 22, 2018 9:02 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
Uberdude wrote:
So that's about 800 playouts for #184 and 1600 for Elfv1?
Something like 800 visits for #184 and 2000 visits for Elfv1


This post by Vargo was liked by: Uberdude
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #171 Posted: Mon Oct 22, 2018 11:51 am 
Dies in gote

Posts: 50
Liked others: 0
Was liked: 3
Uberdude wrote:
nbc44, were the games for your graph played in series of parallel? One effect that's been observed in the official LZ test matches is when the games are played in parallel and start around the same time (as they are with distributing the match tasks to LZ volunteers) then there can be a bias for the early games to be losses by the player who makes early blunders. So for example in Elf vs LZ matches when the games go to counting Elf tends to win, but Elf will blunder ladders and resign early so these data points have a bias to arrive earlier and makes LZ's winrate after 50 games be higher than it is after the full 400.

It was "validation.exe" test. 50,100,200,300,400 games per match, what are you offering? And what about of mathematical statistics?

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #172 Posted: Mon Oct 22, 2018 8:55 pm 
Oza
User avatar

Posts: 2401
Location: Tokyo, Japan
Liked others: 2338
Was liked: 1332
Rank: Jp 6 dan
KGS: ez4u
Uberdude wrote:
I wonder how does #183 do versus Elf v1 at time-parity? I saw it's now stronger (57% win) at visit parity in the test match: http://zero.sjeng.org/match-games/5bcaa ... 3e27abce47.

...

I was looking at this latest match as well last night (my time). It was interesting to compare it to the previous match with the 178 net vs ELF on September 20th. (This is about to scroll off the bottom of the list of test matches - grab it while it is hot!).

As we know, ELF favors black at the beginning of the game, in contrast to almost all other strong bots. In the earlier (178) match Black indeed won 54% of the games. Black won 194 out of 362 games with both ELF and the 178 net winning 97 times while White won 168 times with ELF winning 83 times and 178 winning 85 times.

However, in the latest match with the 183 net, the situation reversed and White won 54% of the games. Now White won 216 of the 402 games played with the 183 net winning 124 times and ELF winning 92 times. Black won 186 games with 183 winning 103 times and ELF winning 83 times. So 183 had it 55% White versus 45% Black while ELF had 53% White versus 47% Black.

In the earlier match 17 of the games were counted rather than resigned. ELF won all the counted games. In the latest match, however, the 183 net won 4 of the 7 counted games. Indeed ELF played out one game that it lost by 16.5 points. I did not look at any of the games. I wonder if this game contains some kind of life and death error?

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #173 Posted: Tue Oct 23, 2018 1:01 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Interesting observation about the colour win% changing.

ez4u wrote:
It was interesting to compare it to the previous match with the 178 net vs ELF on September 20th. (This is about to scroll off the bottom of the list of test matches - grab it while it is hot!).

Saving this link which should keep working: http://zero.sjeng.org/match-games/5ba32 ... 3773940dd8

I wonder in the last #178 match how many of LZ's wins were Elf ladder/damezumari blunders. Obviously well done to LZ 183 for being able to keep roughly even with Elf long enough and to get into a position where Elf can mess up ladders, but these wins still feel a bit cheap to me (particularly as Elf would likely not make many of them if it was given same time instead of visits as LZ). So I wonder if we have gone from with 178 LZ getting say 10% wins from Elf blunders, 40% sensible wins, 50% sensible losses, to with 183 getting 10% wins from Elf blunders, 45% sensible wins, 45% sensible losses; or does 183 get 15% Elf blunder wins, 40$ sensible wins, 45% sensible losses.

Update:
As a quick check, I counted the number of 92 and 93 move games (all were LZ wins), assuming they are Elf ladder/liberty blunders*. In the 178 match there were 13 92 move (LZ B+res) 10 93 move (LZ W+Res). In the 183 match there were 19 92 move (LZ B+res) and 18 93 moves (LZ W+Res). So that's 5.7% cheap early wins in the 178 match, 9.2% for 183 match. So more than half of the match winrate improvement from 50.3 to 56.5 came from making Elf early blunder more.

* Characterising as blunders does sometimes diminish LZ's win, e.g. 178 game 11 below Elf got damezumarid, but that's because LZ played a nice loose net so to avoid that Elf would've needed to read that the net worked and played differently strategically beforehand to avoid giving LZ that possibility (actually playing around with Elf I think white can avoid capture, one line with n9 at l10 is seki). In fact when I analysed that game for 1600 playouts at move 46 Elfv1 wanted to first come out at n9 and then switched to p17 and didn't even consider s10 which is an aggressive move putting attack before defence and what got it into a net-able position. So that's a bit weird, but when played it does like s10 so I guess Elf just got randomly "lucky" that it gave a few playouts to s10 and discovered it liked it that in that instance it rose to the top of #playouts. On the other hand running out a stone captured in a ladder where tenuki would give a playable position is just Elf being super dumb and Elf losing rather than LZ winning.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #174 Posted: Tue Oct 23, 2018 2:47 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
14 game match between #184 and Elfv1
LZ015, GPU:2x1080Ti, 7 min per side per game, no pondering, komi 7.5 (Elfv1 is always W, sorry)

Elfv1 wins 12-2 ... Even with Elf always taking W, #184 seems far from Elf, at time parity.

(a 7 min game with 2x1080Ti is equivalent to ~8000 visits for Elfv1 and ~ 3200 visits for #184)


Attachments:
elf2_184_elf2isW.zip [13.33 KiB]
Downloaded 382 times

This post by Vargo was liked by: Uberdude
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #175 Posted: Tue Oct 23, 2018 5:12 am 
Dies in gote

Posts: 50
Liked others: 0
Was liked: 3
Uberdude wrote:
So that's about 800 playouts for #184 and 1600 for Elfv1?

No chance:
Code:
The first net is worse than the second
#184 v #elf1 ( 175 games)
          wins        black       white
#184   77 44.00%   44 44.44%   33 43.42%
#elf1  98 56.00%   55 55.56%   43 56.58%
                   99 56.57%   76 43.43%

Code:
C:\APPS\l0gpu15.999\leelaz.exe -g -v 1600 --gpu 0 --gpu 1 --noponder -t 12 -q -d --timemanage off --precision half -w C:\APPS\net\d13c4099.gz
vs
C:\APPS\l0gpu15.999\leelaz.exe -g -v 800 --gpu 1 --gpu 0 --noponder -t 12 -q -d --timemanage off --precision half -w C:\APPS\net\2da87ea8.gz


P.S. There are too many short games (~1/4).


Attachments:
184-elf1.zip [138.12 KiB]
Downloaded 370 times
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #176 Posted: Tue Oct 23, 2018 8:22 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
I don't know what you mean.

Here is a very short test, anybody with a 1080 can do it :

I launched two 3 game matches #184 v Elf v1 (twogtp)

One match with --visits=801 for #184 , and --visits=2001 for Elf
and another match with -time 5 (minutes per side per game)

As you can see, the numbers are very similar :
Attachment:
5 min.jpg
5 min.jpg [ 63 KiB | Viewed 9025 times ]
Attachment:
visits.jpg
visits.jpg [ 59.01 KiB | Viewed 9025 times ]

Global time per move is around 1.8 or 1.9" in both cases.
Two times 3 games is very few, but I don't want to run a 1000 game match, just to find the same numbers within 10%

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #177 Posted: Tue Oct 23, 2018 8:42 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Hmmm. I gather that the training games are all self play. Is it possible to have Elf vs. LZ training games? If so, I expect that both could benefit from that. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #178 Posted: Tue Oct 23, 2018 3:26 pm 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Some Elf games are/were (I think they are being thinned/phased out recently) used in the LZ training. I'm not sure if these are real Elf engine+weights self-play games released in some huge dataset by Facebook, the LZ engine with converted Elf weights playing against itself on LZ volunteer machines, or LZ engine with Elf weights vs LZ engine with LZ weights playing on volunteer machines, but suspect it's the middle one. Can anyone confirm? When these Elf games were added to the training pipeline LZ improved very fast initially as it absorbed the Elvish wisdom. You can see latest LZ has a more Elvish style; not so obsessed with 3-3 invasions of 4-4 but approaches them too, the 4 corners then 4 approaches fuseki liked by Elf is now also played by LZ (almost exclusively in the vs elf test matches I looked at, gets kinda boring). Also Elf is somewhat more human in not hating side extensions so much or playing hoshi pincers (kinda retro Japan style, you won't see Koreans do that since the 90s!). Whether LZ would have evolved like this without the influence of the Elf games we can't say, but I think it's fairly safe to say it has speed up that process.


This post by Uberdude was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #179 Posted: Tue Oct 23, 2018 3:38 pm 
Lives in gote

Posts: 311
Liked others: 0
Was liked: 45
Rank: 2d
Bill Spight wrote:
Hmmm. I gather that the training games are all self play. Is it possible to have Elf vs. LZ training games? If so, I expect that both could benefit from that. :)
The nets are not trained on the games themselves but on search results in individual positions. So training on Elf vs LZ games would not be much different to training on some Elf games and some other LZ games (except slightly more variety in the types of positions occurring, and in game results for the value net). What matters is which net was used in the search at each position (whose turn it was).

The Elf games in the training are LZ selfplay with Elf weights AFAIK (the other options are not readily available).

The Elf vs LZ idea sounds most interesting for value net training but also very risky, may introduce some biases: game results would not reliably reflect the position itself anymore but also the strength difference.


This post by moha was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #180 Posted: Tue Oct 30, 2018 4:06 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
In another thread, there was a very heated discussion about LZ 0.14 being stronger or weaker than LZ 0.15

But now, there's LZ 0.15.9999 (not a gag, you can download it HERE)
Discussion about V0.15.999 and V0.15.9999 HERE

V0.15.9999 is a test for V 0.16, and... it seems really stronger than V0.15

20 game match between
LZ 0.15.9999 network #184 and LZ 0.15 network #184
(3 min per side and per game, no pondering, GPU : 1x1080)
LZ 0.15.9999 wins 14-6
Big win...

Edit : another 40 game match 0.15.9999 v 0.15 (#184, same conditions as before)
LZ 0.15.9999 wins 22-18

So, after 60 games : LZ 0.15.9999 wins 36-24
Not bad ;-)


If someone wants the games...

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1 ... 6, 7, 8, 9, 10, 11, 12 ... 21  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group