lightvector,

When I shift between the 20 block and the 15 block nets is it recommended to run the benchmark and tuner command again?

By default, it automatically should, right? So yes, I would say it's recommended.

Although, I don't know how much of a difference it makes, truthfully, compared to if you were to manually rename one tuner file so that the other one would use the same tuning it as well.

I think I misunderstood the tuning thing...

But on my GT650M KataGo is twice as fast after the automatic tuning than if I run the tuner explicit.

On my better GPUs that is not the case.

With Leela Zero maxing out, I wonder what about katago? Is the 20b still improving near the incredible pace it has been? Will there be many more months or even a year more of this? This go bot has been an amazing thing

_________________
set katago to play at your level https://docdro.id/sHZU1ti or experiment with gtp4zen ( https://rb.gy/kx2ilb )

KataGo's run will continue! We're moving to 30 and 40 blocks and will continue training the 20 block to be stronger as well.

Sadly, the run has been temporarily paused for the last week. For various reasons, I need to transfer the ongoing training to a different cloud account instead, but in the new account Google Cloud support has refused to allocate and approve the necessary limits to resume the run. I'm attempting to get this sorted out. Ugh bureaucracy.

100-game test at visits parity (1600 visits for both)
gogui-twogtp 1.5.1, no error, no duplicate game , average time : 204s/game for LZ, 213 s/game for KG.

KataGo 1.3.2 (b20-s19-d43) v. LZ017(#263)
KataGo wins 72-28 (72%) all games by resignation.

stats :The games :

10-game test :
KataGo 1.3.2 (b20-s19-d43) MaxPlayouts=1, numSearchThreads=1
v.
gtp4zen7 -t1 -r20 -s4000 -n3 -o1.4 -p0.7 (8D, according to this site )

gtp4zen 8D wins 8-2 against KataGo at ... 1 playout

Here is the benchmark result for short time control:



I halted this benchmark due to the clear statistical significance, and now a new benchmark with 1m per move (~200k playouts for LZ and ~350k playouts for KataGo) is running, with the current status being KG 7-1 LZ.

Why???

To see how the "raw network" performs.

20-game test :
KataGo 1.3.2 (b20-s19-d43) MaxPlayouts=1, numSearchThreads=1
v.
gtp4zen7 -t1 -s2400 -n2 -o1.5 -p0.5 (strength : 4d)

KataGo wins 19-1 (all games by resignation, average time : ~0.5s/game for Katago !)

twogtp stats (KG always appears as W, because of the command -alternate)The raw network is maybe somewhere between 4d and 8d ?

4 games between katago 1.3.2 s191 at max 400 playouts 1 thread against the weakest bot I could find that beat a pro over a real board, leela zero 125 at 4000 playouts (https://lifein19x19.com/viewtopic.php?f=18&t=15631). I used 1 thread for leelaz as well. Katago's games as white were very close, but katago won all 4, 2 white and 2 black. So it is maybe 9 dan amateur at 400 playouts.

katago 1.3.2 s191 1 thread 400 po 4 wins
lz (9dan) 125 1 thread 4000 po 0 wins
Katago 1.3.2 s191 1 thread 16 playouts vs gtp4zen 9d : 4-0
katago 1.3.2 15b s167 1 thread 70 playouts vs s191 16po : 4-0
katago 1.3.2 10b s114 1 thread 512 playouts vs s167 70po :2-2
Katago 1.3.2 s191 1 thread 1 playouts vs zen6 gtp4zen 7d 3:1
Katago's strength starts at the same level zen6 gtp4zen ends, so combining the two in a gui can give you a wide range of playing strengths to choose from for any board size and either ruleset. If you want lower playing strength (high ddk) than zen6, you can use zen7 withe less playouts. Sample 17k-9d (https://bit.ly/2TbYGeY)

_________________
set katago to play at your level https://docdro.id/sHZU1ti or experiment with gtp4zen ( https://rb.gy/kx2ilb )

Not statistically significant, either : 100 game tests

KG-100po v KG-1po : 100-0
KG-200po v KG 100po : 91-9
KG-400po v KG-200po : 87-13
KG-800po v KG-400po : 81-19
(and KG-400po v KG-100po : 100-0)

KGv1.3.2, network g170-b20c256x2-s1913382912-d435450331, all games by resignation, no error, no duplicate game, gogui-twogtp 1.5.1
Doubling the playouts seems to give a 80-90% winrate, maybe less than that for high numbers of playouts ?

Some stats (highest po always appears as B, because of -alternate)

Some bureaucracy has been navigated finally now, and after a pause of about 2 weeks, KataGo's training run has resumed!

KataGo vs Leela Zero VLTC benchmark is completed.



Game records can be downloaded here. These include SGF files commented with winrates and scores, with PV's expanded as branches, and image files plotting winrates and scores. I hope these can also help for studying purposes.

The major difference between diagonal and parallel openings lies in that the "flying dagger" joseki, which KataGo does not understand very well, is only playable with parallel openings due to ladders.

When specifying max 1 search thread in the config file, is the 32 bit version the same strength at equal (low) playouts as the 64 bit version? Thanks!

_________________
set katago to play at your level https://docdro.id/sHZU1ti or experiment with gtp4zen ( https://rb.gy/kx2ilb )

Thanks, this is very cool! They're now on my (very long) list of game records I want to study. I probably won't get through all 74 though. Are there any games that stand out as highlights?

Including the winrate/score graphs in the download is a nice touch.

So 1 minute per move gives usually a few hundred thousand playouts. I guess the numbers vary a lot depending on the amount of branching in the tree and the amount of tree reuse between moves.

I notice that the first four moves of each game have the word "book" as a comment. You assigned specific opening positions then got the AIs to play from move 5 onwards? How did you choose the openings?

I spectated games when the engine were playing each other, and most of them were very interesting to me. Of course there are some silly games - you can detect them by skimming through winrate graphs and checking if there is a sudden change of winrate. Otherwise I suppose all games are worth studying.


Of course they do, though tree reuse affects visits, not playouts.


I'm using an opening book to avoid semi-duplicate games while not weakening engines with randomness. I attach three sets opening books I made:

openings.zip: All combinations of 4-move openings with 4-4, 4-3 and 3-3, unique under symmetry operations.
openings_small.zip: Same as above, but without 3-3.
openings_verysmall.zip: A handcrafted subset of openings_small.zip.

The smallest opening book was used for this benchmark.

What is "the" flying dagger joseki.

It should be "a" flying dagger joseki in my opinion. One of many, that is.

Nevertheless I have fun with your variant of a flying dagger joseki, although trick moves are not my thing usually. But with this one, if the flying dagger misses its target, it is at least not catastrophic.

And a pro game, the variation seems to be difficult even for 9d professionals:

example variation

Click Here To Show Diagram Code

[go]$$W
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . . . . . . . . . X O . . |
$$ | . . . O . . . . . , . . X O O X O . . |
$$ | . . . . . . . . . . . . O X X O O 8 . |
$$ | . . . . . . . . . . . . . 2 1 X X O . |
$$ | . . . . . . . . . . . . . . 3 4 6 X 9 |
$$ | . . . . . . . . . . . . . . . 5 . 7 . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . O . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . , . . . . . X . . . |
$$ | . . . . . X . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]