AI opinions on "fairest" komi by board size

[lightvector]

I did a study of KataGo's opinions about the "fairest" komi for each square board size using the final 40 block network, for every size that the 40-block network has had direct training experience on. These opinions should reflect pretty well KataGo's experiences during self-play about its empirical winning frequency on these board sizes for different values of komi on those board sizes, and although KataGo is only one bot, might be indicative of what "fair" komi would generally be among bots trained to a comparable level on these board sizes.

Firstly, from separate earlier studies, we know (probable) perfect komi for smaller sizes:

Code: Select all

SZ  Japanese  Chinese     
3      8         9
4      0         2
5     24        25 
6      3         4

[lightvector]

Next my interpretation of the raw data. Most "fair" komi, in KataGo's opinion, assuming quality of play at KataGo's self-play-training level, plus some notes about that komi.

Keep in mind the fact that in Chinese rules, absent seki, odd komi are the ones that matter to cross on odd boards, while even komi are the ones that matter to cross on even boards. So, on odd sized boards, 5.5, 6, 6.5 are all the same for example, while 6.5, 7, 7.5 are all different. On even boards, it's the other way around. So there's this weird oscillatory behavior for Chinese rules on board size parity.

Code: Select all

SZ  Japanese  Chinese     
7      8         9        
KataGo is pretty sure about both values, to such a degree that although formal proof is vastly beyond reach, these are probably decent guesses for the perfect komi.

8      9-*       10-*     
KataGo thinks these values are fair, and is *very* confident that it is not higher. But as for being lower... white has high-variance tactics that KataGo thinks probably don't work, but if they do, could push the perfect komi much lower, even by multiple points. The incredible asymmetry between higher komi confidence vs lower komi confidence is fascinating.

9      6-        7--      
These are the most fair, but KataGo thinks white is at least slightly easier to play in each case, especially in Chinese.

10     6-        6++      
Japanese: White is slightly easier to play. 
Chinese: Black is easier to play.

11     6-     5.5,6,6.5+  
Japanese: White is slightly easier to play. 
Chinese: No seki is expected, unclear if perfect komi is 5 or 7, so fairest komi in practice is in between, but more likely to be 7, so black is slightly easier to play. (7 is less fair in practice because even if the most likely perfect komi, it's so much harder for black to play than 5.5,6,6.5 are for white to play).

12     6      6.5,7,7.5-- 
Japanese: Pretty much fair in practice. 
Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is easier for white to play.

13     6+        7-       
Japanese: Black is slightly easier to play. 
Chinese: White is slightly easier to play.

14     6+     6.5,7,7.5-  
Japanese: Black is slightly easier to play. 
Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play.

15    6.5-        7-      
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. 
Chinese: White is slightly easier to play.

16    6.5-     6.5,7,7.5- 
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. 
Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play.

17    6.5-        7-      
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. 
Chinese: White is slightly easier to play.

18    6.5-     6.5,7,7.5 
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. 
Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between.

19    6.5         7       
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5.
Chinese: Pretty much fair in practice.

[lightvector]

Lastly, the raw data. These are tables of the observed winrates for KataGo from the opening position for different board sizes under different rules. This is the full range of board sizes that KataGo has had actual training on - size 7 to size 19. These were done with the final g170 40 block net with 100ks of playouts on the smaller boards, diminishing to 10ks of playouts on the large boards since those took longer, and using somewhat wider-than-normal search settings.

On the large boards the results appear quite stable even if searching longer, since the search is mostly just smoothly averaging over the large variety of permutations of the opening what the neural net says based on its self-play experience. On the smaller boards, tactical complications start happening much faster, and the values never stop fluctuating as the tactics are explored ever deeper, I often just chose an arbitrary point to cut it off, different runs might get different results.

Also of course, technically these are "KataGo Japanese" and "KataGo Chinese" rules, which might differ from the actual Chinese and Japanese rules in sufficiently pathological positions.

Japanese rules winrates:

Code: Select all

SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7J                                      99  50   3
8J              95  94  93  91  89  89  90  85  84  50   2
9J          89  84  80  70  48  26  17  13
10J     87  82  77  70  60  47  32  22  16
11J             76  68  59  48  37  28  22
12J                 67  59  51  42  35  28
13J                 67  59  52  45  38  31
14J                 65  59  52  46  40  34
15J                 65  60  54  48  42  37
16J                 63  58  53  48  43  38
17J                     57  52  48  44  39
18J                     56  53  49  45  41
19J                     56  52  49  45  42

Chinese rules winrates:

Code: Select all

SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7C                                              99  50   2
8C                      95  92  89  88  90  79  79  80  85  50   1
9C              93  83  69  69  70  41  13  17  16  12   8
10C     93  85  76  75  77  60  31  34  32  18   7
11C             85  74  59  59  59  39  21
12C                     73  59  41  43  42  29  18
13C             83  72  59  59  60  45  31
14C                     71  60  45  46  46  34  23
15C             78  69  59  59  60  48  37
16C                     68  58  47  48  49  39  30
17C             74  65  56  57  57  48  39 
18C                     65  56  48  48  49  41  33
19C             70  63  55  56  56  49  41

And the same two tables, interleaved and paired if you want to more easily compare Japanese and Chinese together for just a single komi:

Code: Select all

SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7J                                      99  50   3
7C                                              99  50   2

8J              95  94  93  91  89  89  90  85  84  50   2
8C                      95  92  89  88  90  79  79  80  85  50   1

9J          89  84  80  70  48  26  17  13
9C              93  83  69  69  70  41  13  17  16  12   8

10J     87  82  77  70  60  47  32  22  16
10C     93  85  76  75  77  60  31  34  32  18   7

11J             76  68  59  48  37  28  22
11C             85  74  59  59  59  39  21

12J                 67  59  51  42  35  28
12C                     73  59  41  43  42  29  18

13J                 67  59  52  45  38  31
13C             83  72  59  59  60  45  31

14J                 65  59  52  46  40  34
14C                     71  60  45  46  46  34  23

15J                 65  60  54  48  42  37
15C             78  69  59  59  60  48  37

16J                 63  58  53  48  43  38
16C                     68  58  47  48  49  39  30

17J                     57  52  48  44  39
17C             74  65  56  57  57  48  39 

18J                     56  53  49  45  41
18C                     65  56  48  48  49  41  33

19J                     56  52  49  45  42
19C             70  63  55  56  56  49  41

[lightvector]

There are of course other things to observe in the raw data too.

Anyways, this was fun. Hope someone finds this interesting.

[Knotwilg]

Fascinating that fair komi hardly changes between board size 9 and 19, and that we've known this all along.

[gennan]

I find this very interesting!

I made charts for odd board sizes 9-19.

So for odd board sizes 9-19, the fairest komi is 6+ for Japanese rules and 7- for Chinese rules.

With Japanese rules, 6 komi looks like perfect komi.
When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then white has an advantage on smaller boards).

With Chinese rules, 7 komi looks like perfect komi.
When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then black has an advantage, more so on smaller boards). With 7.5 komi, white has a fairly large advantage.

Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much.
Does anyone know if there are plans in China or the US to lower the komi to 6.5?

Has any AI research been done on area scoring with integer komi and a Button to break ties?

[Tryss]

Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much.
Does anyone know if there are plans in China or the US to lower the komi to 6.5?

With area scoring rules, you have 361 pts to share, so if there's no seki, the score difference on the board is always an odd integer (If black score on the board is B, and white score on the board is B, then W+B=361)

The only case where there's a difference between 5.5 and 6.5 komi is when there's an odd number of irreductible dame.

[gennan]

Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules.

But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi.

[Bill Spight]

Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules.

But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi.

Well, why not a komi of 7 points? Even pros would probably have ties less than 5% of the time. Chess gets by with a fairly high rate of draws.

I still like Button Go, though.

[ez4u]

@lightvector

How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard.

[lightvector]

I still like Button Go, though.

KataGo supports area scoring + button Go as an evaluation method. I have not studied what its evaluations are or how they differ from those here (I suspect they would largely match the Japanese rules evaluations, but I haven't checked). I welcome anyone to do their own study.

@lightvector

How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard.

Generally KataGo played easily 10x (order of magnitudewise) as many 19x19 games as any other board size. But also smaller boards are easier to learn - whereas if you only a have a few GPUs it takes weeks or months to reach pro level on 19x19 for example, I think it could be done on 9x9 in days. Also an enormous amount of knowledge generalizes between the board sizes, especially on the larger sizes. You can see this in the smoothness of the winrate estimates for the larger board sizes. If it were systematically the case that smaller board size play were abnormally "lower-quality" by a lot than the 19x19 play, you'd expect that empirically in self-play a given degree of komi-unfairness would not result in as pronounced an advantage for one side or the other, and then the neural net's output would reflect those statistics (fitting smooth-ish simple curves to statistics is very easy for neural nets), so you might expect a discontinuity in the sharpness of the winrates as you go from 19x19 to any smaller size. But to the contrary it looks quite regularly increasing in sharpness, as you'd expect if quality of play was fairly regular and the dominant effect was simply the higher variance of the large board due to the board being bigger / games lasting longer, given comparably high-quality play.

This makes intuitive sense to me. I'd think that 95+% of the "knowledge" between the board sizes is shared for board sizes 13x13 and larger. You learn the subtle ways that playing on 15x15 should adjust your overall judgment of territory vs influence and other fuzzy aspects of evaluation, and then modulo those shades of difference in judgment, almost everything you know about fighting and good shapes and what moves are effective in obtaining different kinds of trades carries over.

You can take the winrates as a fairly direct reflection of the self-play training statistics. For example on 9x9 with Japanese rules, KataGo must have experienced roughly fair results with 6 komi, and adjusting it merely to 5.5 komi gives black 70%/30%, more than a 2-1 advantage, whereas adjusting it to 6.5 gives white 74%/26%, almost a 3-1 advantage. While more extreme komis than this will have increasingly limited data, there should be quite many games at all of 5.5, 6, and 6.5, so presumably these are roughly the statistics from the games. Also self-play conditions have added noise and use only 200-1000 playouts - so basically you can say that slightly drunk ultra-blitz KataGo is roughly of a level on 9x9 where, at least against itself, a half-point advantage somewhere between doubles and triples your winning likelihood over the opponent. With the tens of thousands of playouts you'd get in match conditions, it should be yet much sharper.

[lightvector]

One thing I find interesting:

For 3,4,5,6,7 you see a pattern where the odd sizes have a larger advantage for black than the even sizes. Something to do with tengen being a uniquely effective move on small boards, whereas when the center is a 2x2 square, white can also split the board and you get a more balanced result? It sort of makes intuitive sense. But, size 8 breaks the pattern. It has a *larger* advantage for black than either 7 or 9.

Size 9 is the last size in which tengen is a good move - and in fact, contrary to what you hear circulated in some forums, it's not even necessarily the best move - size 9 is where non-tengen moves finally "catch up" and become just as good in quality, 4-4 and 4-5 openings are probably also draws under fair komi. After size 9, non central moves overtake tengen. Past size 9, it also becomes increasingly clear that the bots are no longer "nearly solving" the game, and the game becomes smoother and not dominated by one single highly swingy tactical fight. Both players have enough choices of how to play and maintain average move efficiency that aren't dictated by a high density of one-way-street tactics, that I'd conjecture that perfect komi remains near 6 or 7 perpetually after, near the practical fair komi. As opposed to being much more jittery like the small boards and it merely being beyond anyone's reach to find the needle-in-a-haystack line that makes you multiple points better than the current komis that appear to be fair.

So anyways, that would seems to leave board size 8 as an oddity, where the single-big-tactical-fight of the game results in black having a bigger advantage despite an even size, rather than a smaller advantage, and one of the last board sizes for a chance for this to happen before one might conjecture smooth averaging behavior to start to dominate.

[Uberdude]

Great stuff lightvector. I didn't realise KataGo now prefers white on Japanese 6.5, it used to prefer black. The 20b network I just tried in the BadukAI app still does. Any idea when this changed?

[lightvector]

Uberdude - Not sure. But all the nets are here, feel free to check.

https://d3dndmfyhecmj0.cloudfront.net/g ... index.html

[Elom0]

An update of this data on

Also, I am actually quite interested in KataGo's estimation for 6x6, 5x5, 4x4 and 3x3, perhaps even 2x2 as it would be interesting to see KataGo's thought process in getting to those likely correct komi