It is currently Wed Jul 15, 2020 8:04 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 132 posts ]  Go to page Previous  1 ... 3, 4, 5, 6, 7
Author Message
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #121 Posted: Tue Dec 31, 2019 1:16 am 
Dies with sente

Posts: 87
Location: Pasadena, USA
Liked others: 34
Was liked: 6
Rank: OGS 11 kyu
OGS: Maharani
splee99 wrote:
For higher playouts (visits), Katago is still much weaker than ELFv2.


What amount of playouts are we talking?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #122 Posted: Tue Dec 31, 2019 9:15 am 
Lives in sente

Posts: 955
Liked others: 0
Was liked: 162
And what real time associated with each? Discussing strength in terms of number of playouts if ignoring how much TIME involved for a given amount of hardware does not give a true picture, because go is played with finite time.

Thus telling us that network A is weaker than network B at X playouts each does not mean that network A is weaker than network B on machine Y at time control Z if when running on machine Y network A can do X playouts within Z seconds and network B cannot.

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #123 Posted: Tue Dec 31, 2019 9:23 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
with the same number of playouts, the elf network thinks about 7 times longer than 10x128.
if I play against a human, then I’m interested in how long he thinks, and it’s not at all interesting how much he weighs, what his pulse, temperature, etc. :)
or splee99 implies higher playouts, but with time parity?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #124 Posted: Tue Dec 31, 2019 10:37 am 
Dies with sente

Posts: 87
Location: Pasadena, USA
Liked others: 34
Was liked: 6
Rank: OGS 11 kyu
OGS: Maharani
Even if we're just talking pure number of playouts (which I do find interesting)... At what point does ELF OpenGo v2 start giving much better results than KataGo?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #125 Posted: Tue Dec 31, 2019 11:22 am 
Dies with sente

Posts: 101
Liked others: 2
Was liked: 16
Rank: KGS 2 D
I was running a match between Katago(B) and ELF(W). Yes I used time parity with 20 seconds per move. The playouts are variable, but ELF used roughly 5000 playouts per move. It looks like ELF was ahead from the beginning to the end. I didn't set any resign threshold but at the end both sides indicated that Katago was lost.


Attachment:
kata-elf.sgf [2.88 KiB]
Downloaded 431 times

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #126 Posted: Tue Dec 31, 2019 2:13 pm 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
How many playouts did KataGo typically get during that amount of time each move?

If you haven't tuned KataGo's settings, you should be aware the default config is set to be pretty low-resource usage. The main things are to adjust is the number of threads, and to set it to use FP16 if you are using the CUDA version and have an NVIDIA graphics card with of FP16 tensor cores. Both can give huge boosts if you do have a high-end graphics card.

For tuning threads, you can try running "./katago benchmark" using the exact same arguments you use to run "./katago gtp" and it will try a bunch of different numbers of threads on test positions, report the speed of each one and you can pick the best one (although, err on the low side if speed is similar, since holding speed and/or visits fixed more threads usually hurts strength).

For example, on a V100 GPU with somewhere from 16 to 24 threads and FP16 enabled, KataGo should be at least equally matched with ELF and possibly a little stronger at fixed time settings that reach single-digit thousands of playouts.


This post by lightvector was liked by: MagRes
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #127 Posted: Tue Dec 31, 2019 9:08 pm 
Dies with sente

Posts: 101
Liked others: 2
Was liked: 16
Rank: KGS 2 D
I have attached my configuration file and I'm happy to hear any suggestions to make it better. During that game the root visits was around 10000 to 20000, while the ELF playouts was around 2000 to 5000.

Attachment:
gtp_example .cfg.txt [7.85 KiB]
Downloaded 30 times

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #128 Posted: Wed Jan 01, 2020 12:35 am 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
Thanks for sharing/testing. :tmbup:

You still haven't said if you're using the OpenCL or the CUDA version, but if you're using the CUDA version with a GPU that has tensor cores (such as RTX2080), you want to set cudaUseFP16 and cudaUseNHWC both to true - they currently are not set in your config.

But if you're using the CUDA version on modern yet not quite as cutting-edge GPU that doesn't have tensor cores but still has some FP16 support (for example RTX10** I think?), then setting them either won't work or it won't help much, I think. And if you're using the OpenCL version, that version doesn't have FP16 support at all. It would be straightforward to implement, I've just never gotten around to doing so yet. So assuming you're running ELF on Leela Zero's engine, I would expect ELF to be a little better in these cases, particularly because Leela Zero's engine has code that takes advantage of limited FP16 support even when tensor cores are not available.

Your dynamicScoreUtilityFactor has been modified quite a bit higher from the default - I'm not entirely sure what effect that will have. The default GTP config should have come with is 0.2 and 0.2 for static and dynamic, but you can also try 0.0 and 0.4 which is actually what is used in training. You have 0.2 and 0.5, which puts a lot of weight on score compared to winning/losing.

(Edit: Also numNNServerThreadsPerModel = 2 is interesting if you only have one GPU. If you've specifically benchmarked the difference between setting it to 2 instead of the default of 1, and found it better, great! If you haven't - then I'm not sure why you have a non-default value here).

Besides that your config looks okay. It's hard to compare the numbers you gave due to visits versus playouts difference, assuming you do mean "visits" vs "playouts" the way LZ people usually mean - tree reuse can cause the relationship to vary wildly. But I'd guess both ELF and KataGo should be able to each win a decent number of games against the other. At fixed playouts and smaller numbers of threads on each side I know they are generally fairly similar. And then, which one is better at fixed time is a matter of things like the hardware and implementation details above, which can make as much as a factor of 2 difference in performance one way or another - and which is not small, a factor of 2 is easily more than 100 Elo.

When bots are otherwise close, it's hard to make a blanket statement about what will be best or which bot "is stronger" - messy configuration and hardware details on both sides and simple statistical noise can have a pretty big effect case by case.

Hope that helps? :)


Last edited by lightvector on Wed Jan 01, 2020 8:05 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #129 Posted: Wed Jan 01, 2020 1:45 am 
Dies with sente

Posts: 87
Location: Pasadena, USA
Liked others: 34
Was liked: 6
Rank: OGS 11 kyu
OGS: Maharani
So interesting. I was wondering about that during the recent AI championship streamed by Stephen Hu from the AGA. Does everyone bring their own hardware etc?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #130 Posted: Wed Jan 08, 2020 12:11 pm 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
splee99 wrote:
I was running a match between Katago(B) and ELF(W). Yes I used time parity with 20 seconds per move. The playouts are variable, but ELF used roughly 5000 playouts per move. It looks like ELF was ahead from the beginning to the end. I didn't set any resign threshold but at the end both sides indicated that Katago was lost.

which network KataGo did you use 20x256 or 10x128? if 10x128, then the last, s458837800-d26065887?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #131 Posted: Sat Jan 11, 2020 9:40 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
new network 10x128 https://github.com/lightvector/KataGo/i ... -573324889

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #132 Posted: Mon Jan 20, 2020 4:13 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
KataGo 10x128 g170e - ELF v2 (gt 610, 120 sec/move) 1:1

TIME_B 20630.4 (5h 43m) TIME_W 19088.7 (5h 18m)

TIME_B 16603.3 (4h 36m) TIME_W 19787.4 (5h 29m)


Attachments:
KataGo - ELF.sgf [2.47 KiB]
Downloaded 229 times
ELF - KataGo.sgf [2.39 KiB]
Downloaded 226 times
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 132 posts ]  Go to page Previous  1 ... 3, 4, 5, 6, 7

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group