KataGo V1.3

[xela]

KataGo vs Leela Zero VLTC benchmark is completed.

Thanks, this is very cool! They're now on my (very long) list of game records I want to study. I probably won't get through all 74 though. Are there any games that stand out as highlights?

Including the winrate/score graphs in the download is a nice touch.

So 1 minute per move gives usually a few hundred thousand playouts. I guess the numbers vary a lot depending on the amount of branching in the tree and the amount of tree reuse between moves.

I notice that the first four moves of each game have the word "book" as a comment. You assigned specific opening positions then got the AIs to play from move 5 onwards? How did you choose the openings?

[inbae]

Are there any games that stand out as highlights?

I spectated games when the engine were playing each other, and most of them were very interesting to me. Of course there are some silly games - you can detect them by skimming through winrate graphs and checking if there is a sudden change of winrate. Otherwise I suppose all games are worth studying.

So 1 minute per move gives usually a few hundred thousand playouts. I guess the numbers vary a lot depending on the amount of branching in the tree and the amount of tree reuse between moves.

Of course they do, though tree reuse affects visits, not playouts.

I notice that the first four moves of each game have the word "book" as a comment. You assigned specific opening positions then got the AIs to play from move 5 onwards? How did you choose the openings?

I'm using an opening book to avoid semi-duplicate games while not weakening engines with randomness. I attach three sets opening books I made:

openings.zip: All combinations of 4-move openings with 4-4, 4-3 and 3-3, unique under symmetry operations.
openings_small.zip: Same as above, but without 3-3.
openings_verysmall.zip: A handcrafted subset of openings_small.zip.

The smallest opening book was used for this benchmark.

[Gomoto]

What is "the" flying dagger joseki.

It should be "a" flying dagger joseki in my opinion. One of many, that is.

Nevertheless I have fun with your variant of a flying dagger joseki, although trick moves are not my thing usually. But with this one, if the flying dagger misses its target, it is at least not catastrophic.

[Gomoto]

And a pro game, the variation seems to be difficult even for 9d professionals:

[Gomoto]

example variation

$$W
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . . . . . . . . . X O . . |
$$ | . . . O . . . . . , . . X O O X O . . |
$$ | . . . . . . . . . . . . O X X O O 8 . |
$$ | . . . . . . . . . . . . . 2 1 X X O . |
$$ | . . . . . . . . . . . . . . 3 4 6 X 9 |
$$ | . . . . . . . . . . . . . . . 5 . 7 . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . O . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . , . . . . . X . . . |
$$ | . . . . . X . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------

Click Here To Show Diagram Code

[go]$$W
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . X . . . . |
$$ | . . . . . X . . . . . . . . . X O . . |
$$ | . . . O . . . . . , . . X O O X O . . |
$$ | . . . . . . . . . . . . O X X O O 8 . |
$$ | . . . . . . . . . . . . . 2 1 X X O . |
$$ | . . . . . . . . . . . . . . 3 4 6 X 9 |
$$ | . . . . . . . . . . . . . . . 5 . 7 . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . O . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . , . . . . . X . . . |
$$ | . . . . . X . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

[inbae]

What is "the" flying dagger joseki.

It should be "a" flying dagger joseki in my opinion. One of many, that is.

Nevertheless I have fun with your variant of a flying dagger joseki, although trick moves are not my thing usually. But with this one, if the flying dagger misses its target, it is at least not catastrophic.

AFAIK the word "flying dagger" is a direct translation of a Chinese word, and more specifically, they call the joseki "Mi Yuting's flying dagger" or something. I do not know if there is any other flying dagger joseki though. And that variation is considered not a trick play but a joseki, a very complex one which most of the modern engines (including even Golaxy) fail to understand thoroughly.

[Gomoto]

Flying dagger: viewtopic.php?f=13&t=16016

[inbae]

Huh, didn't know.

[Maharani]

Bots use internal memory to store their calculation progress. If you move up/down the game tree Leela Zero and KataGo will still remember their progress for previous calculations and will be able to resume new calculations from where they left off earlier. But unfortunately:
- If you restart a bot, all progress is gone. And you have to restart calculations from scratch next time.
- After a long time of calculation, the limit of the memory can be reached. When that happens, it will start clean parts of its memory.
- The memory cannot be dumped/exported to a file right now.

[...]

Even though there have been experiments to load/save memory to files, and it hasn't been supported by leela zero and katago yet.

(From this thread)

Are there any near-, mid- or long-term plans for KataGo to support saving/loading calculation memory?

[lightvector]

Are there any near-, mid- or long-term plans for KataGo to support saving/loading calculation memory?

No, not right now. (This might be a lot of work, I think other things take priority).

[lightvector]

Version 1.3.3 is out!

https://github.com/lightvector/KataGo/releases

This is the first release of some new bigger nets: b30c320, b40c256. These should be noticeably stronger at equal playouts but noticeably weaker at equal time. At least, for low 1000s of playouts - I haven't tested them at very long time controls to see whether they do the thing where the big net scales better than the small net for much longer time controls.

And the 20 block net is still improving too, as we are continuing to train it even after switching to the larger nets for self-play.

Also, this release has various configuration, GTP and other improvements. For example, you can now query KataGo for raw neural net evaluations in GTP, have interactively generate a config for you and do the necessary tuning, or run it on KGS out-of-the-box now.

[Mike Novack]

Version 1.3.3 is out!

........ but noticeably weaker at equal time. At least, for low 1000s of playouts ..........- I haven't tested them at very long time controls to see whether they do the thing where the big net scales better than the small net for much longer time controls.

And the 20 block net is still improving too, as we are continuing to train it even after switching to the larger nets for self-play.

Expected behavior, but uninformative as to what the inflection points might be. I realize that you perhaps don't know the information yet in an exact sense, but how about in at least a ball park of what you mean by "short" and "long" time. When you say "short" do you mean a second, ten seconds, a minute?

[lightvector]

I don't think I did say "short"? Rather I said "low 1000s of playouts". For example say the 20 block gets a time control that lets it get about 4000 playouts per move while the 40 block is able to get a bit more than 2000 playouts per move. Generally slightly more since 2x the size of net is a bit less than a 2x slowdown depending on your GPUs and your CPUs.

At roughly those settings, with a test of a few hundred games I'm seeing about a 60-110 Elo gap between the two. For example, for one of the older 40 block versions (a bit older than the one released publicly, which I haven't directly tested) vs one the 20 block versions from last release (s1.91G), test result was 379.5 vs 230.5 vs in favor of the 20 block. Counting draws as half, randomizing between area and territory rules and group tax and button Go and other options automatically choosing whatever komi the nets believe are most fair, usually 6.5 or 7 depending on the rules.

You'd need to work for yourself on your hardware how much time that many playouts takes.

And how much longer "long" would need to be, or if in fact the 30 or 40 block nets have been trained enough to even have better scaling yet? Nobody has tested yet.

[And]

"OpenCL version will now detect CPU-based OpenCL devices, and might run on some pure CPU machines now with no GPU." (https://github.com/lightvector/KataGo/releases)

can I use cpu if there is a gpu? and how to find out if the cpu supports?

[lightvector]

If you run anything with it (GTP, or benchmark), it should list all the devices it finds in the log. Or in the case of the new genconfig command, it will explicitly tell you the devices in the interactive prompt. Then you can select which device you want, either editing an existing gtp.cfg to use that device id, or using genconfig to make an entirely new config.

You may or may not have a CPU OpenCL implementation. The cloud machines I'm testing on don't have one, but my laptop does. Keep in mind that CPU will be a lot slower.