Page 5 of 15
Re: KataGo V1.3
Posted: Sun Feb 09, 2020 3:55 am
by Limeztone
lightvector,
When I shift between the 20 block and the 15 block nets is it recommended to run the benchmark and tuner command again?
Re: KataGo V1.3
Posted: Sun Feb 09, 2020 7:36 am
by lightvector
By default, it automatically should, right? So yes, I would say it's recommended.
Although, I don't know how much of a difference it makes, truthfully, compared to if you were to manually rename one tuner file so that the other one would use the same tuning it as well.
Re: KataGo V1.3
Posted: Sun Feb 09, 2020 10:48 am
by Limeztone
I think I misunderstood the tuning thing...
But on my GT650M KataGo is twice as fast after the automatic tuning than if I run the tuner explicit.
On my better GPUs that is not the case.
Re: KataGo V1.3
Posted: Mon Feb 10, 2020 4:15 pm
by go4thewin
With Leela Zero maxing out, I wonder what about katago? Is the 20b still improving near the incredible pace it has been? Will there be many more months or even a year more of this? This go bot has been an amazing thing
Re: KataGo V1.3
Posted: Mon Feb 10, 2020 5:32 pm
by lightvector
KataGo's run will continue! We're moving to 30 and 40 blocks and will continue training the 20 block to be stronger as well.
Sadly, the run has been temporarily paused for the last week. For various reasons, I need to transfer the ongoing training to a different cloud account instead, but in the new account Google Cloud support has refused to allocate and approve the necessary limits to resume the run. I'm attempting to get this sorted out. Ugh bureaucracy.
Re: KataGo V1.3
Posted: Wed Feb 12, 2020 4:42 am
by Vargo
100-game test at visits parity (1600 visits for both)
gogui-twogtp 1.5.1, no error, no duplicate game , average time : 204s/game for LZ, 213 s/game for KG.
KataGo 1.3.2 (b20-s19-d43) v. LZ017(#263)
KataGo wins 72-28 (72%) all games by resignation.
stats :
The games :
10-game test :
KataGo 1.3.2 (b20-s19-d43)
MaxPlayouts=1, numSearchThreads=1
v.
gtp4zen7 -t1 -r20 -s4000 -n3 -o1.4 -p0.7 (8D, according to
this site )
gtp4zen 8D wins 8-2 against KataGo at ... 1 playout
Re: KataGo V1.3
Posted: Wed Feb 12, 2020 5:46 am
by inbae
Here is the benchmark result for short time control:
Code: Select all
KataGo vs Leela Zero STC benchmark - fixed playouts
KataGo: g170-20x256-1.91G, 60 threads, 36000 playouts
Leela Zero: #263, 20 threads, 20000 playouts
Configs: Resign threshold = 10%, pondering off, temperature = 0
Hardware: 2 x RTX2080Ti
Openings: 4-move opening book (3-3 included)
Black White Total
KataGo 40 (61.5%) 51 (78.5%) 91 (70.0%)
Leela Zero 14 (21.5%) 25 (38.5%) 39 (30.0%)
Total 54 (41.5%) 65 (58.5%) 130
Elo rating difference: +147.2 +- 55.0
I halted this benchmark due to the clear statistical significance, and now a new benchmark with 1m per move (~200k playouts for LZ and ~350k playouts for KataGo) is running, with the current status being KG 7-1 LZ.
Re: KataGo V1.3
Posted: Wed Feb 12, 2020 5:56 am
by xela
Vargo wrote:
10-game test :
KataGo 1.3.2 (b20-s19-d43) MaxPlayouts=1...
Why???
Re: KataGo V1.3
Posted: Wed Feb 12, 2020 6:24 am
by Vargo
xela wrote:Why???
To see how the "raw network" performs.
Re: KataGo V1.3
Posted: Thu Feb 13, 2020 9:34 am
by Vargo
20-game test :
KataGo 1.3.2 (b20-s19-d43) MaxPlayouts=1, numSearchThreads=1
v.
gtp4zen7 -t1 -s2400 -n2 -o1.5 -p0.5 (strength : 4d)
KataGo wins 19-1 (all games by resignation, average time : ~0.5s/game for Katago !)
twogtp stats (KG always appears as W, because of the command -alternate)
The raw network is maybe somewhere between 4d and 8d ?
Re: KataGo V1.3
Posted: Fri Feb 14, 2020 9:35 am
by go4thewin
4 games between katago 1.3.2 s191 at max 400 playouts 1 thread against the weakest bot I could find that beat a pro over a real board, leela zero 125 at 4000 playouts (
viewtopic.php?f=18&t=15631). I used 1 thread for leelaz as well. Katago's games as white were very close, but katago won all 4, 2 white and 2 black. So it is maybe 9 dan amateur at 400 playouts.
katago 1.3.2 s191 1 thread 400 po 4 wins
lz (9dan) 125 1 thread 4000 po 0 wins
Katago 1.3.2 s191 1 thread 16 playouts vs gtp4zen 9d : 4-0
katago 1.3.2 15b s167 1 thread 70 playouts vs s191 16po : 4-0
katago 1.3.2 10b s114 1 thread 512 playouts vs s167 70po :2-2
Katago 1.3.2 s191 1 thread 1 playouts vs
zen6 gtp4zen 7d 3:1
Katago's strength starts at the same level zen6 gtp4zen ends, so combining the two in a gui can give you a wide range of playing strengths to choose from for any board size and either ruleset. If you want lower playing strength (high ddk) than zen6, you can use zen7 withe less playouts. Sample 17k-9d (
https://bit.ly/2TbYGeY)
Re: KataGo V1.3
Posted: Sat Feb 15, 2020 9:00 am
by Vargo
Not statistically significant, either : 100 game tests
KG-100po v KG-1po : 100-0
KG-200po v KG 100po : 91-9
KG-400po v KG-200po : 87-13
KG-800po v KG-400po : 81-19
(and KG-400po v KG-100po : 100-0)
KGv1.3.2, network g170-b20c256x2-s1913382912-d435450331, all games by resignation, no error, no duplicate game, gogui-twogtp 1.5.1
Doubling the playouts seems to give a 80-90% winrate, maybe less than that for high numbers of playouts ?
Some stats (highest po always appears as B, because of -alternate)
Re: KataGo V1.3
Posted: Sun Feb 16, 2020 6:46 am
by lightvector
Some bureaucracy has been navigated finally now, and after a pause of about 2 weeks, KataGo's training run has resumed!
Re: KataGo V1.3
Posted: Fri Feb 21, 2020 7:57 pm
by inbae
KataGo vs Leela Zero VLTC benchmark is completed.
Code: Select all
Engine 1: KataGo v1.3.2, g170-20x256-1.91G
Engine 2: Leela Zero v0.17, #263
Hardware: 2xRTX2080Ti
Time control: 1 minute per move
Black White Total
KataGo 25 (67.6%) 30 (81.1%) 55 (74.3%)
Leela Zero 7 (18.9%) 12 (32.4%) 19 (25.7%)
Total 32 (43.2%) 42 (56.8%) 74
Elo rating difference: +184.6 +- 69.6
Diagonal openings: KG 22-4 LZ, Elo: +296.1 +- 96.4
Parallel openings: KG 33-15 LZ, Elo: +137.0 +- 91.1
Game records can be downloaded
here. These include SGF files commented with winrates and scores, with PV's expanded as branches, and image files plotting winrates and scores. I hope these can also help for studying purposes.
The major difference between diagonal and parallel openings lies in that the "flying dagger" joseki, which KataGo does not understand very well, is only playable with parallel openings due to ladders.
Re: KataGo V1.3
Posted: Sat Feb 22, 2020 9:51 am
by go4thewin
When specifying max 1 search thread in the config file, is the 32 bit version the same strength at equal (low) playouts as the 64 bit version? Thanks!