It is currently Tue Jul 23, 2019 3:39 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8 ... 21  Next
Author Message
Offline
 Post subject: Re: LZ's progression
Post #81 Posted: Tue Jul 31, 2018 9:24 pm 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
@MikeKyle
It's a freely available tool : gogui-twogtp (there's V1.4.9 and V1.4.10)
It automatically plays n games between two programs, then saves the games and gives you a detailed report for each game (winner, score, number of moves, time spent, etc)
It works well with LZ, LZ variable komi, gtp4zen, Ray, and gnugo (and probably with others I haven't tried) but NOT with AQ 2.1.1

V1.4.9 is easy to install under Windows (there's an installer) , it works well for even games, but I think there's a scoring problem with handicap games.
You can download the installer here : https://sourceforge.net/projects/gogui/ ... gui/1.4.9/

V1.4.10 is less easy to use at first (because there's no win installer), but it handles well handicap games.

Once installed, you write a .txt file containing the parameters of the match you want to set up (which programs, nb of playouts, time, pondering, nb of games to play, etc.) and you rename your .txt file as .bat

And you just launch it.

For example,

A file named xxx.bat, and containing :


gogui-twogtp -black "C:\.....path to LZ.....\leelaz.exe --gtp --weights=C:\...path to your weight...\160.txt --visits=1000 --noponder" -white "C:\.....path to LZ.....\leelaz.exe --gtp --weights=C:\...path to your weight...\159.gz --playouts=100 --noponder" -games 30 -sgffile filenameyouwant -auto


would launch a 30 games match between
B : LZ with weights 160.txt (assuming it's the name these weights have on your computer) at 1000 visits and without pondering
and
W : LZ with weights 159.gz (assuming...) at 100 playouts and without pondering

the final report and the games would be named :

filenameyouwant.dat
filenameyouwant01.sgf
filenameyouwant02.sgf
etc...


EDIT :

For gogui-twogtp V1.4.9, The xxx.bat file must be in the same (install) directory as the file "gogui-twogtp.exe"

For both V1.4.9 and V1.4.10, instead of --playouts=xxx or --visits=yyy , you can set the time
-time 5 is the same as -time 5m and means 5min. for the whole game (per side)
-time 1200s means 1200sec
-time 120m+30s/1 means 2hours, plus one 30sec. byoyomi
-time 1s+5s/1 means 5sec. per move (total time must be >0)

For V1.4.10, as there's no installer, the .bat file must be in the installation directory, and must begin with

java.exe -jar lib\gogui-twogtp.jar -black "C:\etc.etc

for example :

If you want to set up an H7 10-games match between LZ and gnugo, you must use twogtp V1.4.10, and you must have one of the lZ variable komi versions , choose for example the one called "dynamic komi for handicap games"

and your .bat file should be something like :

java.exe -jar lib\gogui-twogtp.jar -black "C:\...path to gnugo ...\gnugo.exe --mode gtp" -white "C:\...path to LZvariablekomi ...\leelazVK.exe --gtp --weights=C:\...path to your weights...\40b_155_328k.gz --visits=2000 --noponder" -handicap 7 -games 10 -sgffile whatuwant -auto

for an H7 game between gnugo and LZ_variable_komi


Last edited by Vargo on Mon Aug 06, 2018 12:18 pm, edited 3 times in total.

This post by Vargo was liked by 3 people: ez4u, Uberdude, Waylon
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #82 Posted: Fri Aug 03, 2018 7:36 am 
Judan

Posts: 5948
Location: Cambridge, UK
Liked others: 339
Was liked: 3189
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
sorin wrote:
I saw today two games on WBaduk server, between Ichiriki Ryo 8p (one of Japan's top pros) and LeelaZero. Both were played on even.

LeelaZero won both (one playing black, one white).

Does anyone know what was the event?


I've seen a bunch more, and some by PhoenixGo too. Mostly against young low dan pros. So my guess is it is similar to the online training series against DeepZen from last year. I've not seen a human win. Also I believe the Japan server on WBaduk is a mirror/clone/relay/something of the Japanese Yugen No Ma server. Currently playing is username "aikiller" with 1p Japanese flag, let's see if he or she can live up to that!

Update: Well, aikiller is recorded as winning by 0.5 points, but PhoenixGo played 1 move inside its territory unnecessarily as the last move, presumably because it assumes 7.5 komi but the server rules were 6.5. So not a moral victory even if it technically was one.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #83 Posted: Sun Aug 05, 2018 2:19 am 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
Someone at reddit/cbaduk (here) asked about 161 vs 157 at time parity...

I've run a 20-games match between #157 and #161 at 5 min. per game per side, no ponder, komi 7.5 (twogtp V1.4.10, 1xGTX1080)

LZ_015#157 v. LZ_015#161 :
LZ#157 wins 17-3 (10 wins as W, 7 wins as B)

Ouch... ! 20 games is not enough to be really sigificant, but stilll...

If someone wants the games, I'll upload them.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #84 Posted: Sun Aug 05, 2018 5:33 am 
Lives in gote

Posts: 403
Liked others: 1
Was liked: 110
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
Quote:
Ouch... ! 20 games is not enough to be really sigificant, but stilll...


Actually, if a program A is stronger than B ( the win probability of A against B is >= 0.5), then there is under 0.13% chance that the program A get 3 win or less in 20 games. So the result is statistically significant.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #85 Posted: Sun Aug 05, 2018 7:48 am 
Honinbo

Posts: 8596
Liked others: 2514
Was liked: 2962
Tryss wrote:
Quote:
Ouch... ! 20 games is not enough to be really sigificant, but stilll...


Actually, if a program A is stronger than B ( the win probability of A against B is >= 0.5), then there is under 0.13% chance that the program A get 3 win or less in 20 games. So the result is statistically significant.


But is statistically significant really significant?

This may sound flip, but it is related to the replication crisis.

_________________
There is one human race.
----------------------------------------------------

The Adkins Principle:

At some point, doesn't thinking have to go on?

— Winona Adkins

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #86 Posted: Sun Aug 05, 2018 7:59 am 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
Another 20-games match at 10min/game (1x1080): #157 wins 16-4 (9 times as W, 7 times as B)

And a 10-games match at something like 15min/game #157 wins 6-4 (3 times as W, 3 times as B)
(in fact it's a 5min/game match, but with 2x1080Ti, which corresponds to ~12-16 min/game on a 1080)

Again, if someone wants the games or the reports, I'll upload them.

In the 50 games played in these matches, #161 won only 11, that's 22%, and #161 has a better Elo score than #157, mmmmm... How come ?

Anyway, if someone wants the reports or the games, I'll upload them.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #87 Posted: Sun Aug 05, 2018 9:28 am 
Honinbo

Posts: 8596
Liked others: 2514
Was liked: 2962
Vargo wrote:
In the 50 games played in these matches, #161 won only 11, that's 22%, and #161 has a better Elo score than #157, mmmmm... How come ?


How come, indeed?

_________________
There is one human race.
----------------------------------------------------

The Adkins Principle:

At some point, doesn't thinking have to go on?

— Winona Adkins

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #88 Posted: Sun Aug 05, 2018 9:34 am 
Lives in gote
User avatar

Posts: 471
Liked others: 42
Was liked: 160
Test matches which are used to calculate the Elo score of LZ networks are not with time parity, but with the same number of visits. 20-block networks take more time per visit than 15-block networks.


This post by jlt was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #89 Posted: Sun Aug 05, 2018 9:51 am 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
The Elo scale is at visits parity, that's why I'm doubtful about it... I would prefer something measuring "real" strength, and with the same scale as for human players, but maybe it's hard or impossible to do.

Anyway, I find all this very interesting and I hope LZ will bounce back !

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #90 Posted: Sun Aug 05, 2018 10:41 am 
Judan

Posts: 5948
Location: Cambridge, UK
Liked others: 339
Was liked: 3189
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Vargo wrote:
In the 50 games played in these matches, #161 won only 11, that's 22%, and #161 has a better Elo score than #157, mmmmm... How come ?

Accumulation of errors without calibration, either to older versions of self or an external benchmark. (On a related note, dead-reckoning / inertial guidance systems (as for Apollo or ICBMs) are amazing feats of engineering, I was recently reading about them.)

Uberdude wrote:
In fact during normal 15-block training I think doing some occasionally e.g. #157 vs #147 would be a good idea to see how inflated the incremental self-improvement Elo is (based on the incremental Elo differences from the promotions 147->148->149 etc it went from 11401 -> 11806 which predicts #157 would beat #147 91% of the time, but I would bet it would be quite a lot lower than that in reality).

I think I've managed to get twogtp working and am running a 147 vs 157 match now at 3200 visits.

Update: currently #157 leads only 7-3 as white. #157 6-0 as black before my PC crashed (maybe I shouldn't run this match and LZ and Elf analysis concurrently!).


This post by Uberdude was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #91 Posted: Fri Aug 10, 2018 9:40 am 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
20-games match between LZ015#157 and LZ015#163 at time parity
#163 is 11985 Elo
#157 is 11806 Elo

5-min games, 1x1080, twogtp V1.4.10

#157 wins 16-4 (10 wins as W, 6 wins as B)

I understand the ranking matches are at visits parity, and all that, but there's still something weird with the Elo scale used. For example, the ranking of L-zero has #163 higher than #157 in "Dan scale" ...

Here, #163 wins only 20% of its games against #157... It's only 20 games, but I doubt that #163 is stronger than #157.


Attachments:
163_157_time5_157isB.zip [8.86 KiB]
Downloaded 39 times
163_157_time5_157isW.zip [8.71 KiB]
Downloaded 41 times

This post by Vargo was liked by: Bill Spight
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #92 Posted: Sun Aug 12, 2018 8:49 am 
Dies with sente

Posts: 91
Liked others: 2
Was liked: 15
Rank: KGS 2 D
Vargo wrote:

Here, #163 wins only 20% of its games against #157... It's only 20 games, but I doubt that #163 is stronger than #157.


Could you try #165? It is noticeably more aggressive than before.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #93 Posted: Sun Aug 12, 2018 10:35 am 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
20-games match between LZ_0.15#165 and LZ_0.15#157 at visits parity.

twogtp V1.4.10 --noponder --visits=1601 -komi 7.5 for both.
If you look at the .dat reports, you'll see that #165 takes at least twice as long as #157 to think.

#165 wins 14-6 (9 times as W, 5 times as B, all games by resignation)
Attachment:
reports.zip [1.14 KiB]
Downloaded 55 times

Attachment:
165_157_v1601_157isBlack.zip [8.82 KiB]
Downloaded 45 times

Attachment:
165_157_v1601_157isWhite.zip [8.69 KiB]
Downloaded 42 times

splee99 wrote:
Could you try #165? It is noticeably more aggressive than before.

Ok. Tomorrow, I'll run a 20-games match at time parity (#165 v #157). It will be interesting to compare results at visits parity and at time parity.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #94 Posted: Sun Aug 12, 2018 11:03 am 
Dies with sente

Posts: 91
Liked others: 2
Was liked: 15
Rank: KGS 2 D
Thanks for the report. I had run one game between #165 and #157 with time parity and #157 won. However the battle was so heavy that only bots can deal with it. Anyway I hope we can see some progress from #161 to #165 from your test tomorrow.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #95 Posted: Mon Aug 13, 2018 4:05 am 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
40-game match at time parity between #165 and #157
(40 games, because I made a mistake, the first 20 games were all with #157 as White, so I had to run 20 more)

5 min per game, komi 7.5, no pondering, 1xGTX1080, twogtp V1.4.10
5 min per game on my computer is similar to 1600 visits for #165 or 3200 visits for #157

#157 wins 27-13
(15 times as W, 12 times as B, all games by resignation)

13 wins for #165, that's 32.5 %, much better than #161, which scored only 15% in a 20-game match at 5 min against #157.

The 40 games :
Attachment:
163_157_5min_157isWhite.zip [17.74 KiB]
Downloaded 47 times
Attachment:
165_157_5min_157isBlack.zip [18.34 KiB]
Downloaded 33 times


This post by Vargo was liked by: Uberdude
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #96 Posted: Mon Aug 13, 2018 9:12 am 
Judan

Posts: 5948
Location: Cambridge, UK
Liked others: 339
Was liked: 3189
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
There's a new 40block network out: http://zero.sjeng.org/networks/e2be4815 ... 1c836a3.gz. I wonder how would this would do vs #157 on 5 mins time parity? It got a very respectable 42% vs Elf v1 (presumably at visits parity) and 83% vs #162. So in a 5 mins game it'll probably get about 800 visits (Edit: is that true, it seems not approx. linear in blocks if 15 to 20 is a halving of visits!) to #157's 3200. Will its superior intuition and judgement win out, or will it make tactical blunders like ladders due to not enough playouts and lose?

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #97 Posted: Mon Aug 13, 2018 11:53 pm 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
Uberdude wrote:
There's a new 40block network out: http://zero.sjeng.org/networks/e2be4815 ... 1c836a3.gz. I wonder how would this would do vs #157
20-game match between LZ0.15 with e2be48 (256x40) and LZ0.15 with #157 (192x15)

e2be48 at 1600 visits, and #157 at 6400 visits.

I think time parity is very roughly :
1x visits for 256x40 or
2x visits for 256x20 or
4x visits for 192x15

So, e2be48 at 1600 visits v. #157 at 6400 visits is at time parity.

#157 wins 11-9 (5 times as W, 6 times as B)
Attachment:
20 games.zip [18.62 KiB]
Downloaded 41 times

I just realized (better late than never !) that twogtp can give a nice and complete .html report, with min, max, standard deviation...

LIKE THIS

AND THIS

With 2xGTX1080Ti,
Average time used per game is around 3min10sec (see html for exact numbers)
Average length is around 220 moves


This post by Vargo was liked by 2 people: Uberdude, Waylon
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #98 Posted: Tue Aug 14, 2018 7:29 am 
Lives in gote

Posts: 403
Liked others: 1
Was liked: 110
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
So the latest 40b has about the same strenght as #157 at time parity ? Nice !

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #99 Posted: Tue Aug 14, 2018 8:05 am 
Lives with ko

Posts: 242
Liked others: 0
Was liked: 30
Rank: 2d
Tryss wrote:
So the latest 40b has about the same strenght as #157 at time parity ? Nice !
I doubt this could be said in a general sense.

Vargo wrote:
So, e2be48 at 1600 visits v. #157 at 6400 visits is at time parity.

#157 wins 11-9 (5 times as W, 6 times as B)

The point of these deep and strong nets is to direct the search, shape the search tree. But with 1600 visits you don't have much of a search tree to be shaped (just a few moves lookahead). In those cases the bot won't be really strong anyway (even AGZ was only high dan amateur without search), and the most important is getting more visits (which #157 does).

But in serious games these bots do a lot more visits per move on good gpus. I suggest you repeat the test with at least 10k visits vs 40k. I would be surprised if 40b won't crush #157.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #100 Posted: Tue Aug 14, 2018 8:24 am 
Lives with ko

Posts: 236
Liked others: 4
Was liked: 57
moha wrote:
I doubt this could be said in a general sense.
You must be right, because I've tested the new 40b (e2be48) against #157 at time parity : only 5min on 1x1080.
For e2be48, it corresponds to only 500-600 visits(?)
#157 wins 14-6
6 wins as W here , and 8 wins as B

Maybe I'll try tomorrow a mini match at time parity, with 3200 or 6400 visits for e2be48.
e2be48 should be stronger, we'll see.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8 ... 21  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group