It is currently Tue Sep 24, 2019 2:19 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 423 posts ]  Go to page Previous  1 ... 12, 13, 14, 15, 16, 17, 18 ... 22  Next
Author Message
Offline
 Post subject: Re: LZ's progression
Post #281 Posted: Sat Jan 26, 2019 4:42 pm 
Dies with sente

Posts: 87
Liked others: 26
Was liked: 12
Rank: Total beginner
Universal go server handle: Satorian
At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #282 Posted: Tue Jan 29, 2019 10:43 am 
Dies with sente

Posts: 120
Liked others: 25
Was liked: 8
Anyone with a good gpu can test the minigo network?
2019-01-10 12:49 50b98433(40x256) VS d13c4099(elf 20x224) 266 : 134 (66.50%) - elf 33.5%
2019-01-24 03:56 b9efeafa(minigo 19x256) VS ffe8ba44(40x256) 30 : 53 (36.14%) - minigo 36.14%
http://zero.sjeng.org

Minigo v15-990 was stronger than ELFv1 at 10s and 120s (10s/ 120s was at least 10k/ 120k-po on ELFv1)/ a move.
https://github.com/gcp/leela-zero/issues/2179

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #283 Posted: Tue Jan 29, 2019 11:29 am 
Judan

Posts: 6074
Location: Cambridge, UK
Liked others: 345
Was liked: 3269
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Satorian wrote:
At 120 seconds per player, I would expect this to be 1s per move or less. I don't think the 1080 Ti outperforms the 1060 at 20:1 to allow the 40b network to reach 3k playouts on a consistent basis. Perhaps Vargo could provide some stats on average/median/min/max playout count and time per move.


I asked a similar question a while back so we can estimate it:

Uberdude wrote:
Vargo wrote:
20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6
(8 wins as B, 6 wins as W)


So that's about 800 playouts for #184 and 1600 for Elfv1?


Vargo wrote:
Uberdude wrote:
So that's about 800 playouts for #184 and 1600 for Elfv1?
Something like 800 visits for #184 and 2000 visits for Elfv1


So that was 5 mins per side per game, these test are 1 minute. 184 is a 40 block so (assuming there isn't an LZ 0.16 fp16 speedup between the two tests, and there may be) 800/5 = 160 so a lot short of the several thousand I like to give the 40 blocks to show their strength. If there is fp16 difference that'll help at most a factor of 2.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #284 Posted: Tue Jan 29, 2019 12:10 pm 
Lives in gote

Posts: 582
Location: Vienna, Austria
Liked others: 252
Was liked: 297
There are Leela Zero-compatible models for the MiniGo v16 series (40x256) as well:

https://cloudygo.com/converted_model/v16-19x19/models/

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #285 Posted: Tue Jan 29, 2019 12:25 pm 
Dies with sente

Posts: 120
Liked others: 25
Was liked: 8
MiniGo v15:
https://cloudygo.com/converted_model/v15-19x19/models/

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #286 Posted: Tue Jan 29, 2019 12:29 pm 
Dies with sente

Posts: 100
Liked others: 0
Was liked: 8
Can we use this txt.gz file with lizzie by replacing the weight file?
I heard minigo is very strong.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #287 Posted: Tue Jan 29, 2019 12:36 pm 
Dies with sente

Posts: 120
Liked others: 25
Was liked: 8
yes, i tried 990, ok

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #288 Posted: Tue Jan 29, 2019 2:24 pm 
Lives in gote

Posts: 582
Location: Vienna, Austria
Liked others: 252
Was liked: 297
The latest converted (to Leela Zero-style) v15 model, 000939-heron, has a rating of 4430 according to https://cloudygo.com/v15-19x19/eval-graphs and has just won a game against ELFv1 at time parity (30 seconds per move on my MacBook Pro, only a few thousand visits); admittedly not statistically significant.

The latest converted v16 model, 000513-revenge, has a rating of 4202, according to https://cloudygo.com/v16-19x19/eval-graphs .

Now I'm not sure whether these ratings are comparable or whether they are within one run. That is, one model may be rated 4430 according to the v15 scale, and the other model may be rated 4202 according to the v16 scale, but I'm not sure whether these scales are the same.

I'm just running a game between v16-000513-revenge and ELFv1 and they look more evenly matched than v15-000939-heron and ELFv1.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #289 Posted: Tue Jan 29, 2019 2:39 pm 
Judan

Posts: 6074
Location: Cambridge, UK
Liked others: 345
Was liked: 3269
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
I just used 000513-revenge in a 40 second game on Fox, beat a Fox 9d who I presume was also a bot by half a point (benefiting from being black with 6.5 komi?). viewtopic.php?p=241098#p241098

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #290 Posted: Tue Jan 29, 2019 3:13 pm 
Dies with sente

Posts: 76
Liked others: 10
Was liked: 47
Rank: AGA 4d
Marcel Grünauer wrote:
The latest converted v16 model, 000513-revenge, has a rating of 4202, according to https://cloudygo.com/v16-19x19/eval-graphs .

Now I'm not sure whether these ratings are comparable or whether they are within one run. That is, one model may be rated 4430 according to the v15 scale, and the other model may be rated 4202 according to the v16 scale, but I'm not sure whether these scales are the same.


Hi! Minigo author here :) The ratings are not directly comparable. We'll get some cross-run ratings in a bit when v16 is finished. But I'd love any help folks can provide in figuring out what the best settings to use and how to run it effectively -- we don't really have the charter to make things as strong as possible, so i'm hoping folks can help us there, but that's another story ;)


This post by seigenblues was liked by: Marcel Grünauer
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #291 Posted: Tue Jan 29, 2019 3:25 pm 
Lives in gote

Posts: 582
Location: Vienna, Austria
Liked others: 252
Was liked: 297
seigenblues wrote:
Hi! Minigo author here :) The ratings are not directly comparable. We'll get some cross-run ratings in a bit when v16 is finished. But I'd love any help folks can provide in figuring out what the best settings to use and how to run it effectively -- we don't really have the charter to make things as strong as possible, so i'm hoping folks can help us there, but that's another story ;)


Oh; I'm just watching your videos on MiniGo [1] and while I'm rather un-knowledgable about neural networks, I think I'm slowly getting it.

IIRC gcp also said that he'd rather stick to the Zero approach as opposed to making Leela Zero as strong as possible. But what *is* your charter then? And how can we help to make the MiniGo models as strong as possible? (Which I assume is what many people here are most interested in.)

[1] MiniGo: TensorFlow Meets Andrew Jackson and Minigo: Building a Go AI with Kubernetes and TensorFlow (Cloud Next '18)


This post by Marcel Grünauer was liked by: ez4u
Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #292 Posted: Wed Jan 30, 2019 2:57 am 
Dies with sente

Posts: 87
Liked others: 26
Was liked: 12
Rank: Total beginner
Universal go server handle: Satorian
Uberdude wrote:
So that was 5 mins per side per game, these test are 1 minute. 184 is a 40 block so (assuming there isn't an LZ 0.16 fp16 speedup between the two tests, and there may be) 800/5 = 160 so a lot short of the several thousand I like to give the 40 blocks to show their strength. If there is fp16 difference that'll help at most a factor of 2.


Interesting. Thank you! So even with the FP16 best case, we'd be at ~320 playouts. That's a far cry from what I would consider reasonable, especially given your experience and reports on playout effects.

I would prefer to have the engines simply play at 10s or 20s or 30s per move, in pure byo-yomi. No time management, no fancy distractions, and a move speed that is realistic either for online go or longer OTB games.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #293 Posted: Wed Jan 30, 2019 4:10 am 
Judan

Posts: 6074
Location: Cambridge, UK
Liked others: 345
Was liked: 3269
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
... though if I can get a decent few thousand playouts in 20 seconds per move on my 1060 GPU, Vargo may be able to get the same in 5 seconds on his top-end 1080 Ti.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #294 Posted: Wed Jan 30, 2019 6:31 am 
Lives with ko

Posts: 238
Liked others: 4
Was liked: 57
mini 10 game match
Elfv1 v. V16_000513 converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 10, time parity, twogtp 1.5.0

Result : 10-0 for Elf... :o
I used -alternate, so, Elf-White seems to win 10 times, but in fact, Elf won 5 times as W and 5 times as B.


Something wrong with the parameters ?
(time is several seconds per move)

Attachment:
mgelf.jpg
mgelf.jpg [ 347.8 KiB | Viewed 1420 times ]

I'll try another match, with an other converted network.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #295 Posted: Wed Jan 30, 2019 7:46 am 
Lives in sente

Posts: 932
Liked others: 0
Was liked: 158
Uberdude wrote:
... though if I can get a decent few thousand playouts in 20 seconds per move on my 1060 GPU, Vargo may be able to get the same in 5 seconds on his top-end 1080 Ti.


THAT is more or less what I have been trying to say. I was NOT saying a particular amount of time but one realistic scaled to what we decide to consider "adequate hardware" << so less actual clock time if tests run on a more powerful machine >>

For example, we might decide "should be a portable machine" so power consumption and cooling requirements limit the gpu to the equivalent of a 1060. Our question would then be "what is the most powerful (go strength( program if run on a powerful, but portable, gaming machine at fast, but playable for humans time control?"

If the actual tests were to be run on a machine X times faster then we would scale the time to 1/x of that.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #296 Posted: Wed Jan 30, 2019 8:38 am 
Dies with sente

Posts: 120
Liked others: 25
Was liked: 8
Vargo wrote:
mini 10 game match
Elfv1 v. V16_000513 converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 10, time parity, twogtp 1.5.0

Result : 10-0 for Elf... :o

v15(20x256) is better:
https://cloudygo.com/v16-19x19/eval-graphs
https://cloudygo.com/all-eval-graphs?bucket=

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #297 Posted: Wed Jan 30, 2019 9:15 am 
Lives in gote

Posts: 582
Location: Vienna, Austria
Liked others: 252
Was liked: 297
Vargo wrote:
Something wrong with the parameters ?


I think that (as of this post) the strongest v16 network isn't as strong as the strongest v15 network.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #298 Posted: Wed Jan 30, 2019 10:02 am 
Lives with ko

Posts: 238
Liked others: 4
Was liked: 57
another mini 10 game match
Elfv1 v. V15-000990-cormorant converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 5, time parity, twogtp 1.5.0

Result : 6-4 for Elf

time is several seconds per move. 10 games is too few, I'll run more games tomorrow.

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #299 Posted: Wed Jan 30, 2019 10:19 am 
Judan

Posts: 6074
Location: Cambridge, UK
Liked others: 345
Was liked: 3269
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Vargo wrote:
mini 10 game match
Elfv1 v. V16_000513 converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 10, time parity, twogtp 1.5.0

Result : 10-0 for Elf... :o
I used -alternate, so, Elf-White seems to win 10 times, but in fact, Elf won 5 times as W and 5 times as B.

Something wrong with the parameters ?
(time is several seconds per move)


I expect what we are seeing here is the reduced playouts that 40-block v16_513 minigo gets in this short time parity are really hurting it against the 20-block elfv1. If you ran a test of fewer games but with enough time for minigo to get around 3k playouts per move (so I guess Elf gets about 6-8k, playouts speed isn't quite linear in number of blocks) then you'd find it not so lopsided. (By analogy with 40-block LZ's, I don't have much feel for minigo's behaviour yet)

Top
 Profile  
 
Offline
 Post subject: Re: LZ's progression
Post #300 Posted: Wed Jan 30, 2019 1:23 pm 
Judan

Posts: 6074
Location: Cambridge, UK
Liked others: 345
Was liked: 3269
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
So I manually played a game between elfv1 (black) and minigo v16.513 (white), giving them >30s a move (approx ~3k for minigo and 6k for elf), giving them both extra time on hard moves, watching for where they agreed and differed. Elf as black (which it tends to favour) thought it was winning a lot, but had a blind spot for white p15 so its o17 was a lot worse than it expected. I don't have the time to play it to counting, but they both agree the game is pretty good for white (Elfv1 says 75%, minigo 85%). Here's the sgfs with winrates from each's point of view. Also elf missed q2 and winrate plunged on that, minigo saw it quickly (but probably not quickly enough for Vargo's setup).





And here's their view of the winrate, minigo's is much more stable/consistent. Does this mean it can't detect tiny mistakes as well as Elf can, or is Elf just being erratic and missing moves which cause it to change evaluation? I tend towards the latter.
Attachment:
elf winrate.PNG
elf winrate.PNG [ 287.94 KiB | Viewed 1292 times ]

Attachment:
minigo winrate.PNG
minigo winrate.PNG [ 287.96 KiB | Viewed 1292 times ]


Attachments:
minigo16.513 vs elfv1 minigo view.sgf [15.27 KiB]
Downloaded 401 times
minigo16.513 vs elfv1 elf view.sgf [10.37 KiB]
Downloaded 398 times

This post by Uberdude was liked by: ez4u
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 423 posts ]  Go to page Previous  1 ... 12, 13, 14, 15, 16, 17, 18 ... 22  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group