It is currently Wed Jul 15, 2020 9:39 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 132 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7  Next
Author Message
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #41 Posted: Sun Jul 21, 2019 1:54 pm 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
And wrote:
How can I check performance, for example, visits or n/s? interesting to compare OpenCL and cuda

Download them both, try them out on a variety of positions to adjust how many threads you should use for best performance for each one, and see how long searches with different numbers of visits takes?


This post by lightvector was liked by: And
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #42 Posted: Sun Jul 21, 2019 11:34 pm 
Lives in gote

Posts: 649
Liked others: 281
Was liked: 325
I ran a series of ten-game matches between KataGo and some Leela Zero networks.

The players were:

* KataGo = KataGo v1.2 beta with the g104-b20c256 network

* LZ15b = Leela Zero with the latest 15b-trained-on-40b-games network

* ELFv2 = Leela Zero with ELFv2

* LZ233 = Leela Zero with the #233 network

Results:

* KataGo : LZ15b = 10 : 0

* KataGo : ELFv2 = 8 : 2

* KataGo : LZ233 = 5 : 5

Conditions: Radeon Pro 580X, 1600 visits for each player. KataGo took longer to reach this count than ELFv2.

Some KataGo wins were due to kills; some were due to early ladders across the board.

Attachment:
File comment: SGF files
matches.zip [41.28 KiB]
Downloaded 61 times

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #43 Posted: Mon Jul 22, 2019 1:31 am 
Lives in gote

Posts: 649
Liked others: 281
Was liked: 325
KataGo versus Leela Zero matches In "drunken pro"-mode - that is, just one visit. 100 game match:

KataGo - ELFv2 = 98 : 2
KataGo - LZ-233 = 96 : 4

But many of the games end pretty abruptly, so no idea how reliable these stats are.

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #44 Posted: Mon Jul 22, 2019 4:36 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
in sabaki katago plays on any boards up to 19x19. even 2x2! using one network! unfortunately more than 19x19 does not work, probably a restriction in the release (using sabaki you can play on boards up to 25x25)
Perhaps the strongest program for the game 13x13 and 9x9?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #45 Posted: Mon Jul 22, 2019 8:20 am 
Beginner

Posts: 19
Liked others: 0
Was liked: 2
And wrote:
How can I check performance, for example, visits or n/s? interesting to compare OpenCL and cuda


Unsure if there is a way. I benchmarked by bringing up a complex middlegame position in Sabaki for each engine and then literally timing playouts per second using a stopwatch. You can set a large playout limit for each engine, and then tell it to generate a move.

Note that you must optimize the number of threads manually for KataGo, since I believe the default is one. This is a bit complicated, since as lightvector points out the engine becomes weaker as threads increase. There will be an inflection point where speed gains become small and are no longer overcoming the loss of strength due to increasing threads further.

I settled on 16 threads for my hardware. At this setting KataGo was close to twice as fast generating playouts as the 40-block LZ network. That is similar to Elfv2, which makes sense given that they are both 20-block networks.

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #46 Posted: Mon Jul 22, 2019 9:25 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
hoa803 Thanks, I'm interested in a test like in Lz (--benchmark). By the way, in gtp.log I found information about visits

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #47 Posted: Tue Jul 23, 2019 2:38 pm 
Lives in gote

Posts: 649
Liked others: 281
Was liked: 325
KataGo vs Leela Zero at time parity of 10 sec/move:

KataGo vs ELFv2 = 1 : 9
KataGo vs LZ-157 = 4 : 6
KataGo vs LZ-130 = 9 : 1

For reference, LZ-125 beat Zhao Baolong 2p; he said he might win one or two in a hundred games if he can exploit a ladder weakness. But KataGo has special ladder support, so it's still strong enough for us amateurs. And it can deal with handicap stones.

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #48 Posted: Tue Jul 23, 2019 6:56 pm 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
Marcel Grünauer wrote:
KataGo vs Leela Zero at time parity of 10 sec/move:

KataGo vs ELFv2 = 1 : 9
KataGo vs LZ-157 = 4 : 6
KataGo vs LZ-130 = 9 : 1

For reference, LZ-125 beat Zhao Baolong 2p; he said he might win one or two in a hundred games if he can exploit a ladder weakness. But KataGo has special ladder support, so it's still strong enough for us amateurs. And it can deal with handicap stones.


Are you using only one search thread? If so, I would expect better results if you increase the number search threads KataGo is allowed to use. I should probably make the default larger, although given the wildly different specs on different user's systems, it's hard to pick a particular good number in advance. If you have a strong GPU, the number ideal of search threads can actually be significantly more than the number of cores that your CPU has, since it's more about making sure the GPU is well-fed and the CPU cores will often be idle waiting on the GPU anyways.

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #49 Posted: Tue Jul 23, 2019 7:50 pm 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
hoa803 wrote:
I benchmarked by bringing up a complex middlegame position in Sabaki for each engine and then literally timing playouts per second using a stopwatch. You can set a large playout limit for each engine, and then tell it to generate a move.


Sorry about the stopwatch. :)

I'll add a utility to the next release version that should auto-time the bot on a set of hardcoded test positions with different number of threads and report the speed for each, so that it's easier to choose a number of threads. Or maybe, to even auto-tune it, although that might take a bit more work (OpenCL parameters are auto-tuned and stored already, but the way search threads are handled is slightly different).

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #50 Posted: Wed Jul 24, 2019 3:56 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
what does "special ladder support" mean?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #51 Posted: Wed Jul 24, 2019 9:15 am 
Lives in gote

Posts: 649
Liked others: 281
Was liked: 325
lightvector wrote:
Are you using only one search thread? If so, I would expect better results if you increase the number search threads KataGo is allowed to use. I should probably make the default larger, although given the wildly different specs on different user's systems, it's hard to pick a particular good number in advance. If you have a strong GPU, the number ideal of search threads can actually be significantly more than the number of cores that your CPU has, since it's more about making sure the GPU is well-fed and the CPU cores will often be idle waiting on the GPU anyways.


Yes; I didn't think about that. I changed the config to use "numSearchThreads = 10" and reran the test against ELFv2 at 10s/move time parity. Now KataGo wins 3 out of 10 games against ELFv2.

I have an iMac with a Radeon Pro 580X, so it's not very strong.

Two things I'd really like is a "benchmark" command like Leela Zero's and a way to make the first moves more random, using something like Leela Zero's "--randomcnt" and "--randomvisits".

I'm using KataGo to play handicap games and it always goes into all available 3-3 points, or if I use a free handicap of, say, four komoku, it plays a keima against the first one and then always attaches against the other komoku stones.


This post by Marcel Grünauer was liked by: lightvector
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #52 Posted: Wed Jul 24, 2019 4:10 pm 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
Marcel Grünauer wrote:
Two things I'd really like is a "benchmark" command like Leela Zero's and a way to make the first moves more random, using something like Leela Zero's "--randomcnt" and "--randomvisits".

I'm using KataGo to play handicap games and it always goes into all available 3-3 points, or if I use a free handicap of, say, four komoku, it plays a keima against the first one and then always attaches against the other komoku stones.


Yep, if you've ready my reply to hoa803 right above the next version will have better support for testing and tuning performance. In the meantime, if you're willing to experiment further you might be able to do a bit better than just setting it to a number like 10 (unless you did experiment a fair bit in order to eventually settle on 10 as best!). Otherwise, just stay tuned I guess for future versions. :rambo:

For randomizing early play, you can fiddle with the "chosenMoveTemperatureEarly" parameter and the other parameters near it. Actually, if you want to tune anything about KataGo, you might just want to read through the whole gtp_example.cfg! The comments should explain roughly what most of the settings do. :study:

The policy output does weight 3-3 quite heavily. I agree opening variety is a bit low a common problem for all the 'zero' bots to varying degrees. Still a topic of research how to encourage variety in a non-totally-contrived way and that actually helps training strength, rather than hurting it. For now though in the worst case you could force-play alternative moves for white in one or two of the corners before continuing as a normal handicap game. You can also try free-placement patterns that don't involve a stone in each corner - use the stones for corner enclosures of different kinds, or chinese/star-point/other side formations, and let white have a corner.

And wrote:
what does "special ladder support" mean?


KataGo's neural net gets an input plane that marks all stones that are in or can be put into repeated inescapable atari, both now and over the last 2 turns (to help with seeing when moves are ladder breakers/makers). So it should never be mistaken about the status of any direct ladder. In rare situations when very losing and also very behind in score, you might still see it play out a step or two of a broken ladder though.


This post by lightvector was liked by 2 people: Bonobo, Marcel Grünauer
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #53 Posted: Thu Jul 25, 2019 3:22 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
lightvector, thank you, interestingly, this feature of the KataGo, relating to the ladders, was obtained as a result of training or was it originally set in the program?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #54 Posted: Thu Jul 25, 2019 4:42 am 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
And wrote:
lightvector, thank you, interestingly, this feature of the KataGo, relating to the ladders, was obtained as a result of training or was it originally set in the program?


Re-read what I just wrote. :razz:

lightvector wrote:
KataGo's neural net gets an input plane that marks all stones that are in or can be put into repeated inescapable atari


It's an input to the neural net. So it's not a thing that is being learned or produced by the neural net, it's being provided as an input. Although the neural net will still need to learn how to use the information or how much attention to give to it.

(In case you're unfamiliar with the terminology, in ALL zero-trained programs, the neural net is the thing that gets trained, and is the part of a program responsible for pattern recognition for what moves are worth considering further, as well as judging the whole board to determine whether the result of a possible line of play is good or bad).


This post by lightvector was liked by 2 people: And, Bonobo
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #55 Posted: Thu Jul 25, 2019 5:01 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
lightvector, I know almost nothing about the device of neural networks, but I understood approximately how it works. besides this, I do not know English and I use a translator (and the translator often translates in such a way that nothing is clear at all!). thanks for the answer!

PS I took a position with the beginning of the ladder from one game where the ELFv2 lost. KataGo immediately understood what would happen next and played differently! I like KataGo more and more!

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #56 Posted: Fri Jul 26, 2019 9:58 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
on gt 610 compared to numSearchThreads = 1 with numSearchThreads = 4 position calculation is 2% faster, 8-9%, 16-16%, 32-19%. Is the result better on powerful video cards? Can I change other settings to improve performance? (OpenCL)

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #57 Posted: Sat Jul 27, 2019 8:37 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
hoa803 wrote:
At this setting KataGo was close to twice as fast generating playouts as the 40-block LZ network.

How did you determine this? measuring the time to calculate the position?

Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #58 Posted: Sat Jul 27, 2019 11:35 am 
Lives in gote

Posts: 531
Liked others: 87
Was liked: 567
Rank: maybe 2d
And wrote:
on gt 610 compared to numSearchThreads = 1 with numSearchThreads = 4 position calculation is 2% faster, 8-9%, 16-16%, 32-19%. Is the result better on powerful video cards? Can I change other settings to improve performance? (OpenCL)


Probably there aren't many settings you can change here that will make a big difference, it's more that KataGo's OpenCL implementation just needs more optimization work - not surprising since KataGo is the first time I've ever written any GPU code whatsoever, and I'm learning as I go. :)

Once I get more free-time to work on it, I'll continue to incrementally improve things in addition to having better benchmarking tools. Future versions some weeks or months down the line should be better.


This post by lightvector was liked by: And
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #59 Posted: Sun Jul 28, 2019 4:35 am 
Lives in gote
User avatar

Posts: 378
Liked others: 60
Was liked: 33
thank! waiting for new releases! We are also waiting the CPUonly version!


This post by And was liked by: serbes
Top
 Profile  
 
Offline
 Post subject: Re: A new run of KataGo released - strength comparable to EL
Post #60 Posted: Mon Jul 29, 2019 10:30 am 
Beginner

Posts: 19
Liked others: 3
Was liked: 8
And wrote:
thank! waiting for new releases! We are also waiting the CPUonly version!


You can already run on the CPU if it supports OpenCL. It's quite slow though, since the cpu slowness compounds with the unoptimised opencl code.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 132 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group