It is currently Thu Mar 28, 2024 11:01 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1 ... 11, 12, 13, 14, 15, 16, 17 ... 21  Next
Author Message
Offline
 Post subject: Re: LZ's progression
Post #261 Posted: Sun Jan 20, 2019 11:12 am 
Gosei
User avatar

Posts: 1348
Liked others: 202
Was liked: 203
gogui-twogtp V1.4.9, leela-zero-0.15-cpuonly 20 games ok. I previously unpacked the network. I have a weak video card now, but I'll try to run the gpu version

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #262 Posted: Sun Jan 20, 2019 11:33 am 
Gosei
User avatar

Posts: 1348
Liked others: 202
Was liked: 203
I started the gpu version (using sabaki) with f438268e - it plays normally, Ms5 - it works, but it makes stupid moves. It seems that Ms5 has problems with gpu

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #263 Posted: Mon Jan 21, 2019 3:53 am 
Gosei
User avatar

Posts: 1348
Liked others: 202
Was liked: 203
40 games f438268e - Ms5 28-12 (10 sec/move, no pondering, LZ0.15-cpuonly, twogtp V1.4.9)

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #264 Posted: Wed Jan 23, 2019 9:01 am 
Gosei
User avatar

Posts: 1348
Liked others: 202
Was liked: 203
60 more games, total
100 games f438268e - Ms5 63:37 (10 sec/move, no pondering, LZ0.15-cpuonly, twogtp V1.4.9)
Ms5 disappointed

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #265 Posted: Thu Jan 24, 2019 9:28 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
I heard networks bjiyxo f438268 (192x15) and LM_GX89 (192x15) were very strong. OK, I added two players : LZ_Elfv1 (224x290) and LZ#202 (256x40) to run a 4 players tournament.

Short time parity (2 min/game per player , with gpu:1x1080) no pondering , komi 7.5, every player playing 10 games against every other one.

Results :

202-Elf 3-7
202-f43 5-5
202-x89 2-8
x89-Elf 5-5
x89-f43 7-3
Elf-f43 5-5


I'm very much surprised by the poor result of #202 (too little time for 40b ?) and by the excellent result of GX89.

It's about 3h for the 60 matches. I'll try to run the same thing tomorrow, to see if I get similar results. Maybe (probably) I should let it run for a lot more games.


Complete tournament code :
Code:
rem 202-elf

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\202.gz --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\elf2.txt --noponder" -games 10 -sgffile 202elf -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze 202elf.dat

rem  202-f43

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\202.gz --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\f438268.gz --noponder" -games 10 -sgffile 202f43 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze 202f43.dat

rem 202-X89

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\202.gz --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\LM_GX89.txt --noponder" -games 10 -sgffile 202x89 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze 202x89.dat


rem X89-elf

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\LM_GX89.txt --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\elf2.txt --noponder" -games 10 -sgffile x89elf -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze x89elf.dat

rem X89-f43

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\LM_GX89.txt --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\f438268.gz --noponder" -games 10 -sgffile x89f43 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze x89f43.dat

rem elf-f43

gogui-twogtp -black "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\elf2.txt --noponder" -white "C:\Users\jm\gogui150\leela-zero-0.16-win64ok\leelaz.exe --gtp --weights=C:\Users\jm\LZnetworks\f438268.gz --noponder" -games 10 -sgffile elff43 -time 2 -auto -komi 7.5 -alternate

gogui-twogtp -analyze elff43.dat
Attachment:
all the stats.zip [32.08 KiB]
Downloaded 389 times

Attachment:
60 games.zip [49.23 KiB]
Downloaded 389 times

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #266 Posted: Thu Jan 24, 2019 11:56 am 
Lives in sente

Posts: 1037
Liked others: 0
Was liked: 180
Vargo wrote:


Short time parity (2 min/game per player , with gpu:1x1080) no pondering , komi 7.5, every player playing 10 games against every other one.

......

I'm very much surprised by the poor result of #202 (too little time for 40b ?) and by the excellent result of GX89.


I think this sort of thing requires REASONABLE time controls. Nothing faster than human blitz play. Saying that this irrelevant, that all that matters is equality, misses an important point.

Think of a race. Which is faster, a human or a race car?

a) Down a quarter mile straightway
b) Ten times around two pylons 50 feet apart

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #267 Posted: Thu Jan 24, 2019 8:49 pm 
Dies with sente

Posts: 101
Liked others: 2
Was liked: 16
Rank: KGS 2 D
Well, a "long game" on my computer (GTX1050) with 20 seconds per move maybe just same as the "2 minutes" game on a GTX1080. #202 lost again to LM-GX89. Sometimes it seems bigger doesn't mean better in the AI world.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #268 Posted: Thu Jan 24, 2019 11:47 pm 
Lives with ko

Posts: 198
Liked others: 4
Was liked: 16
Is it possible to use the gx-89 with lizzie?

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #269 Posted: Fri Jan 25, 2019 2:36 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
lichigo wrote:
Is it possible to use the gx-89 with lizzie?

Should be, I've used GX88 with Lizzie.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #270 Posted: Fri Jan 25, 2019 2:55 am 
Lives with ko

Posts: 198
Liked others: 4
Was liked: 16
Oh great, could you share the link. I would like to try it ^^

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #271 Posted: Fri Jan 25, 2019 4:57 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
lichigo wrote:
could you share the link
HERE


Mike Novack wrote:
this sort of thing requires REASONABLE time controls
With 2min/game and good gpu, these networks are already (very) strong, I don't see why it's not reasonable.



I've let it run for 6x20 more games.

After 6x30 games, results are :

202-Elf 12-18
202-f43 17-13
202-x89 12-18
x89-Elf 12-18
x89-f43 14-16
Elf-f43 17-13


I'll let it run up to 6x50 games, results in the evening.
_________________________________________________________

Assigning arbitrarily 4000 Elo to each network at the start of the tournament, after 30 games, I get

1)Elf_ v1 4047
2)LM_GX89 3994
3)f438268 3981
4)LZ_#202 3978


(and yes, I know it's questionable ;-) )

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #272 Posted: Fri Jan 25, 2019 8:55 am 
Gosei
User avatar

Posts: 1348
Liked others: 202
Was liked: 203
when using gpu 1080, these networks play very strongly and at 2 min/game. even using cpu only, will many be able to defeat them?
but the results are different if you use only cpu, for example:

100 games f438268e - elf.v1 62:38 (5 sec/move, no pondering, LZ0.15-cpuonly, twogtp V1.4.9)

for an elf there is not enough computational power, it is probably better to take a 15 block network for a CPU only (with little time control)
that is, when the strength of the networks does not differ much, the results of the matches depend not only on the number of games (if there are few of them, then the result is doubtful), but also from time to move + power gpu / cpu

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #273 Posted: Fri Jan 25, 2019 10:44 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
After 6x50 games, results are :

202-Elf 21-29
202-f43 21-29
202-x89 28-22
x89-Elf 22-28
x89-f43 26-24
Elf-f43 27-23


A little oddity : usually, all games are by resignation, but there was one won by 1.5 after... 423 moves :D
Attachment:
gx89_elf.jpg
gx89_elf.jpg [ 143.29 KiB | Viewed 8374 times ]


After 6x50 games, starting with 4000 for each network, I get relative Elos :


1)Elf_ v1 4034
2)LM_GX89 4003
3)f438268 3985
4)LZ_#202 3978

but 6x50 games is too few to be really meaningful.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #274 Posted: Fri Jan 25, 2019 2:15 pm 
Lives in sente

Posts: 1037
Liked others: 0
Was liked: 180
Maybe I should make clearer what I am saying.

The whole point of this is determining the question is A stronger than B. It is entirely possible that A runs on less resources than B. In other words, at short enough time control, A "has enough to be strong" while B does not. However while A would be stronger given more time, the rate at which it gets stronger might be less than the rate at which B gets stronger with more time.

That is why am suggesting should be reasonable time controls (no faster than human blitz)

There is ANOTHER question that we could be asking. There is trade off between strength of hardware and time. We MIGHT be asking "what is the strongest program, playing at reasonable time controls, but on weaker hardware". THAT we can model at shorter time controls (like being used here) on this powerful hardware. The "weaker hardware" question IS important to many of us. We just need to be clear about what question we are asking.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #275 Posted: Sat Jan 26, 2019 1:11 am 
Lives in gote

Posts: 337
Liked others: 22
Was liked: 97
Hardware, parameters and time control are all reasonable, and you know it. Other tests with weaker or stronger hardware, with different networks, or different time controls are obviously possible. Feel free to show us your results, it would be interesting.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #276 Posted: Sat Jan 26, 2019 4:51 am 
Dies with sente

Posts: 93
Liked others: 26
Was liked: 13
Rank: Total beginner
Universal go server handle: Satorian
Of course everyone is free to do as desired with their own processing power and time, but I agree with Mike that a super short playing/move time has a bias towards smaller/quicker networks by allowing them more playouts within a generally low-playout regime, while those networks would plateau much quicker with longer, more reasonable time controls. I also don't find these super short time controls very interesting. Using an engine for review I would probably give it at least 30-60 minutes total or 10-15s per move, same for playing against it.


Last edited by Satorian on Sat Jan 26, 2019 4:38 pm, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #277 Posted: Sat Jan 26, 2019 5:50 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
How many playouts a move is 40 block LZ getting in these 2 minute test games on a 1080 Ti? If it's at least 3k (what I get in 20 seconds on my 1060, and would take 5 minutes on my crappy laptop) then I think that's about enough for a test where the deeper network can show its strength and not be handicapped by being unable to do basic reading (there was a nice chart on the LZ github showing this I linked a while back).

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #278 Posted: Sat Jan 26, 2019 4:42 pm 
Dies with sente

Posts: 93
Liked others: 26
Was liked: 13
Rank: Total beginner
Universal go server handle: Satorian
At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #279 Posted: Tue Jan 29, 2019 10:43 am 
Gosei
User avatar

Posts: 1348
Liked others: 202
Was liked: 203
Anyone with a good gpu can test the minigo network?
2019-01-10 12:49 50b98433(40x256) VS d13c4099(elf 20x224) 266 : 134 (66.50%) - elf 33.5%
2019-01-24 03:56 b9efeafa(minigo 19x256) VS ffe8ba44(40x256) 30 : 53 (36.14%) - minigo 36.14%
http://zero.sjeng.org

Minigo v15-990 was stronger than ELFv1 at 10s and 120s (10s/ 120s was at least 10k/ 120k-po on ELFv1)/ a move.
https://github.com/gcp/leela-zero/issues/2179

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #280 Posted: Tue Jan 29, 2019 11:29 am 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Satorian wrote:
At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.


I asked a similar question a while back so we can estimate it:

Uberdude wrote:
Vargo wrote:
20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6
(8 wins as B, 6 wins as W)


So that's about 800 playouts for #184 and 1600 for Elfv1?


Vargo wrote:
Uberdude wrote:
So that's about 800 playouts for #184 and 1600 for Elfv1?
Something like 800 visits for #184 and 2000 visits for Elfv1


So that was 5 mins per side per game, these test are 1 minute. 184 is a 40 block so (assuming there isn't an LZ 0.16 fp16 speedup between the two tests, and there may be) 800/5 = 160 so a lot short of the several thousand I like to give the 40 blocks to show their strength. If there is fp16 difference that'll help at most a factor of 2.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 418 posts ]  Go to page Previous  1 ... 11, 12, 13, 14, 15, 16, 17 ... 21  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group