KataGo V1.3

[Limeztone]

lightvector,

When I shift between the 20 block and the 15 block nets is it recommended to run the benchmark and tuner command again?

[lightvector]

By default, it automatically should, right? So yes, I would say it's recommended.

Although, I don't know how much of a difference it makes, truthfully, compared to if you were to manually rename one tuner file so that the other one would use the same tuning it as well.

[Limeztone]

I think I misunderstood the tuning thing...

But on my GT650M KataGo is twice as fast after the automatic tuning than if I run the tuner explicit.

On my better GPUs that is not the case.

[go4thewin]

With Leela Zero maxing out, I wonder what about katago? Is the 20b still improving near the incredible pace it has been? Will there be many more months or even a year more of this? This go bot has been an amazing thing

[lightvector]

KataGo's run will continue! We're moving to 30 and 40 blocks and will continue training the 20 block to be stronger as well.

Sadly, the run has been temporarily paused for the last week. For various reasons, I need to transfer the ongoing training to a different cloud account instead, but in the new account Google Cloud support has refused to allocate and approve the necessary limits to resume the run. I'm attempting to get this sorted out. Ugh bureaucracy.

[Vargo]

100-game test at visits parity (1600 visits for both)
gogui-twogtp 1.5.1, no error, no duplicate game , average time : 204s/game for LZ, 213 s/game for KG.

KataGo 1.3.2 (b20-s19-d43) v. LZ017(#263)
KataGo wins 72-28 (72%) all games by resignation.

stats :

k263.jpg (462.82 KiB) Viewed 10537 times

The games :

KG132_v1600_LZ263_v1600.rar

(85.99 KiB) Downloaded 409 times

10-game test :
KataGo 1.3.2 (b20-s19-d43) MaxPlayouts=1, numSearchThreads=1
v.
gtp4zen7 -t1 -r20 -s4000 -n3 -o1.4 -p0.7 (8D, according to this site )

gtp4zen 8D wins 8-2 against KataGo at ... 1 playout

[inbae]

Here is the benchmark result for short time control:

Code: Select all

KataGo vs Leela Zero STC benchmark - fixed playouts

    KataGo: g170-20x256-1.91G, 60 threads, 36000 playouts
Leela Zero: #263, 20 threads, 20000 playouts
   Configs: Resign threshold = 10%, pondering off, temperature = 0
  Hardware: 2 x RTX2080Ti
  Openings: 4-move opening book (3-3 included)

              Black          White          Total
KataGo       40 (61.5%)     51 (78.5%)     91 (70.0%)
Leela Zero   14 (21.5%)     25 (38.5%)     39 (30.0%)
Total        54 (41.5%)     65 (58.5%)     130

Elo rating difference: +147.2 +- 55.0

I halted this benchmark due to the clear statistical significance, and now a new benchmark with 1m per move (~200k playouts for LZ and ~350k playouts for KataGo) is running, with the current status being KG 7-1 LZ.

[xela]

10-game test :
KataGo 1.3.2 (b20-s19-d43) MaxPlayouts=1...

Why???

[Vargo]

Why???

To see how the "raw network" performs.

[Vargo]

20-game test :
KataGo 1.3.2 (b20-s19-d43) MaxPlayouts=1, numSearchThreads=1
v.
gtp4zen7 -t1 -s2400 -n2 -o1.5 -p0.5 (strength : 4d)

KataGo wins 19-1 (all games by resignation, average time : ~0.5s/game for Katago !)

twogtp stats (KG always appears as W, because of the command -alternate)

k1.jpg (402.93 KiB) Viewed 10435 times

The raw network is maybe somewhere between 4d and 8d ?

[go4thewin]

4 games between katago 1.3.2 s191 at max 400 playouts 1 thread against the weakest bot I could find that beat a pro over a real board, leela zero 125 at 4000 playouts (viewtopic.php?f=18&t=15631). I used 1 thread for leelaz as well. Katago's games as white were very close, but katago won all 4, 2 white and 2 black. So it is maybe 9 dan amateur at 400 playouts.

katago 1.3.2 s191 1 thread 400 po 4 wins
lz (9dan) 125 1 thread 4000 po 0 wins
Katago 1.3.2 s191 1 thread 16 playouts vs gtp4zen 9d : 4-0
katago 1.3.2 15b s167 1 thread 70 playouts vs s191 16po : 4-0
katago 1.3.2 10b s114 1 thread 512 playouts vs s167 70po :2-2
Katago 1.3.2 s191 1 thread 1 playouts vs zen6 gtp4zen 7d 3:1
Katago's strength starts at the same level zen6 gtp4zen ends, so combining the two in a gui can give you a wide range of playing strengths to choose from for any board size and either ruleset. If you want lower playing strength (high ddk) than zen6, you can use zen7 withe less playouts. Sample 17k-9d (https://bit.ly/2TbYGeY)

[Vargo]

Not statistically significant, either : 100 game tests

KG-100po v KG-1po : 100-0
KG-200po v KG 100po : 91-9
KG-400po v KG-200po : 87-13
KG-800po v KG-400po : 81-19
(and KG-400po v KG-100po : 100-0)

KGv1.3.2, network g170-b20c256x2-s1913382912-d435450331, all games by resignation, no error, no duplicate game, gogui-twogtp 1.5.1
Doubling the playouts seems to give a 80-90% winrate, maybe less than that for high numbers of playouts ?

Some stats (highest po always appears as B, because of -alternate)

2.jpg (235.5 KiB) Viewed 10270 times

4.jpg (245.59 KiB) Viewed 10270 times

5.jpg (232.7 KiB) Viewed 10270 times

[lightvector]

Some bureaucracy has been navigated finally now, and after a pause of about 2 weeks, KataGo's training run has resumed!

[inbae]

KataGo vs Leela Zero VLTC benchmark is completed.

Code: Select all

Engine 1: KataGo v1.3.2, g170-20x256-1.91G
Engine 2: Leela Zero v0.17, #263
Hardware: 2xRTX2080Ti
Time control: 1 minute per move

              Black          White          Total
KataGo       25 (67.6%)     30 (81.1%)     55 (74.3%)
Leela Zero    7 (18.9%)     12 (32.4%)     19 (25.7%)
Total        32 (43.2%)     42 (56.8%)     74

Elo rating difference: +184.6 +- 69.6

Diagonal openings: KG 22-4 LZ, Elo: +296.1 +- 96.4
Parallel openings: KG 33-15 LZ, Elo: +137.0 +- 91.1

Game records can be downloaded here. These include SGF files commented with winrates and scores, with PV's expanded as branches, and image files plotting winrates and scores. I hope these can also help for studying purposes.

The major difference between diagonal and parallel openings lies in that the "flying dagger" joseki, which KataGo does not understand very well, is only playable with parallel openings due to ladders.

[go4thewin]

When specifying max 1 search thread in the config file, is the 32 bit version the same strength at equal (low) playouts as the 64 bit version? Thanks!