It is currently Tue Apr 20, 2021 1:30 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 14 posts ] 
Author Message
Offline
 Post subject: AI opinions on "fairest" komi by board size
Post #1 Posted: Thu Aug 27, 2020 9:41 pm 
Lives in gote

Posts: 654
Liked others: 108
Was liked: 789
Rank: maybe 2d
I did a study of KataGo's opinions about the "fairest" komi for each square board size using the final 40 block network, for every size that the 40-block network has had direct training experience on. These opinions should reflect pretty well KataGo's experiences during self-play about its empirical winning frequency on these board sizes for different values of komi on those board sizes, and although KataGo is only one bot, might be indicative of what "fair" komi would generally be among bots trained to a comparable level on these board sizes.

Firstly, from separate earlier studies, we know (probable) perfect komi for smaller sizes:
Code:
SZ  Japanese  Chinese     
3      8         9
4      0         2
5     24        25
6      3         4


This post by lightvector was liked by: Knotwilg
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #2 Posted: Thu Aug 27, 2020 9:42 pm 
Lives in gote

Posts: 654
Liked others: 108
Was liked: 789
Rank: maybe 2d
Next my interpretation of the raw data. Most "fair" komi, in KataGo's opinion, assuming quality of play at KataGo's self-play-training level, plus some notes about that komi.

Keep in mind the fact that in Chinese rules, absent seki, odd komi are the ones that matter to cross on odd boards, while even komi are the ones that matter to cross on even boards. So, on odd sized boards, 5.5, 6, 6.5 are all the same for example, while 6.5, 7, 7.5 are all different. On even boards, it's the other way around. So there's this weird oscillatory behavior for Chinese rules on board size parity.

Code:
SZ  Japanese  Chinese     
7      8         9       
KataGo is pretty sure about both values, to such a degree that although formal proof is vastly beyond reach, these are probably decent guesses for the perfect komi.

8      9-*       10-*     
KataGo thinks these values are fair, and is *very* confident that it is not higher. But as for being lower... white has high-variance tactics that KataGo thinks probably don't work, but if they do, could push the perfect komi much lower, even by multiple points. The incredible asymmetry between higher komi confidence vs lower komi confidence is fascinating.

9      6-        7--     
These are the most fair, but KataGo thinks white is at least slightly easier to play in each case, especially in Chinese.

10     6-        6++     
Japanese: White is slightly easier to play.
Chinese: Black is easier to play.

11     6-     5.5,6,6.5+ 
Japanese: White is slightly easier to play.
Chinese: No seki is expected, unclear if perfect komi is 5 or 7, so fairest komi in practice is in between, but more likely to be 7, so black is slightly easier to play. (7 is less fair in practice because even if the most likely perfect komi, it's so much harder for black to play than 5.5,6,6.5 are for white to play).

12     6      6.5,7,7.5--
Japanese: Pretty much fair in practice.
Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is easier for white to play.

13     6+        7-       
Japanese: Black is slightly easier to play.
Chinese: White is slightly easier to play.

14     6+     6.5,7,7.5- 
Japanese: Black is slightly easier to play.
Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play.

15    6.5-        7-     
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: White is slightly easier to play.

16    6.5-     6.5,7,7.5-
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play.

17    6.5-        7-     
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: White is slightly easier to play.

18    6.5-     6.5,7,7.5
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between.

19    6.5         7       
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5.
Chinese: Pretty much fair in practice.


This post by lightvector was liked by: icarusmustburn
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #3 Posted: Thu Aug 27, 2020 9:48 pm 
Lives in gote

Posts: 654
Liked others: 108
Was liked: 789
Rank: maybe 2d
Lastly, the raw data. These are tables of the observed winrates for KataGo from the opening position for different board sizes under different rules. This is the full range of board sizes that KataGo has had actual training on - size 7 to size 19. These were done with the final g170 40 block net with 100ks of playouts on the smaller boards, diminishing to 10ks of playouts on the large boards since those took longer, and using somewhat wider-than-normal search settings.

On the large boards the results appear quite stable even if searching longer, since the search is mostly just smoothly averaging over the large variety of permutations of the opening what the neural net says based on its self-play experience. On the smaller boards, tactical complications start happening much faster, and the values never stop fluctuating as the tactics are explored ever deeper, I often just chose an arbitrary point to cut it off, different runs might get different results.

Also of course, technically these are "KataGo Japanese" and "KataGo Chinese" rules, which might differ from the actual Chinese and Japanese rules in sufficiently pathological positions.

Japanese rules winrates:
Code:
SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7J                                      99  50   3
8J              95  94  93  91  89  89  90  85  84  50   2
9J          89  84  80  70  48  26  17  13
10J     87  82  77  70  60  47  32  22  16
11J             76  68  59  48  37  28  22
12J                 67  59  51  42  35  28
13J                 67  59  52  45  38  31
14J                 65  59  52  46  40  34
15J                 65  60  54  48  42  37
16J                 63  58  53  48  43  38
17J                     57  52  48  44  39
18J                     56  53  49  45  41
19J                     56  52  49  45  42


Chinese rules winrates:
Code:
SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7C                                              99  50   2
8C                      95  92  89  88  90  79  79  80  85  50   1
9C              93  83  69  69  70  41  13  17  16  12   8
10C     93  85  76  75  77  60  31  34  32  18   7
11C             85  74  59  59  59  39  21
12C                     73  59  41  43  42  29  18
13C             83  72  59  59  60  45  31
14C                     71  60  45  46  46  34  23
15C             78  69  59  59  60  48  37
16C                     68  58  47  48  49  39  30
17C             74  65  56  57  57  48  39
18C                     65  56  48  48  49  41  33
19C             70  63  55  56  56  49  41


And the same two tables, interleaved and paired if you want to more easily compare Japanese and Chinese together for just a single komi:
Code:
SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7J                                      99  50   3
7C                                              99  50   2

8J              95  94  93  91  89  89  90  85  84  50   2
8C                      95  92  89  88  90  79  79  80  85  50   1

9J          89  84  80  70  48  26  17  13
9C              93  83  69  69  70  41  13  17  16  12   8

10J     87  82  77  70  60  47  32  22  16
10C     93  85  76  75  77  60  31  34  32  18   7

11J             76  68  59  48  37  28  22
11C             85  74  59  59  59  39  21

12J                 67  59  51  42  35  28
12C                     73  59  41  43  42  29  18

13J                 67  59  52  45  38  31
13C             83  72  59  59  60  45  31

14J                 65  59  52  46  40  34
14C                     71  60  45  46  46  34  23

15J                 65  60  54  48  42  37
15C             78  69  59  59  60  48  37

16J                 63  58  53  48  43  38
16C                     68  58  47  48  49  39  30

17J                     57  52  48  44  39
17C             74  65  56  57  57  48  39

18J                     56  53  49  45  41
18C                     65  56  48  48  49  41  33

19J                     56  52  49  45  42
19C             70  63  55  56  56  49  41


This post by lightvector was liked by 4 people: dfan, icarusmustburn, marvin, Uberdude
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #4 Posted: Thu Aug 27, 2020 9:53 pm 
Lives in gote

Posts: 654
Liked others: 108
Was liked: 789
Rank: maybe 2d
There are of course other things to observe in the raw data too.

Anyways, this was fun. Hope someone finds this interesting. :)

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #5 Posted: Fri Aug 28, 2020 3:49 am 
Gosei
User avatar

Posts: 1843
Location: Ghent, Belgium
Liked others: 283
Was liked: 840
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
Fascinating that fair komi hardly changes between board size 9 and 19, and that we've known this all along.

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #6 Posted: Fri Aug 28, 2020 4:13 am 
Lives in gote

Posts: 394
Location: Netherlands
Liked others: 195
Was liked: 106
Rank: EGF 3d
Universal go server handle: gennan
I find this very interesting!

I made charts for odd board sizes 9-19.

So for odd board sizes 9-19, the fairest komi is 6+ for Japanese rules and 7- for Chinese rules.

With Japanese rules, 6 komi looks like perfect komi.
When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then white has an advantage on smaller boards).

With Chinese rules, 7 komi looks like perfect komi.
When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then black has an advantage, more so on smaller boards). With 7.5 komi, white has a fairly large advantage.

Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much.
Does anyone know if there are plans in China or the US to lower the komi to 6.5?

Has any AI research been done on area scoring with integer komi and a Button to break ties?


Attachments:
komi.png
komi.png [ 67.12 KiB | Viewed 3244 times ]

This post by gennan was liked by: Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #7 Posted: Fri Aug 28, 2020 7:42 am 
Lives in gote

Posts: 499
Liked others: 1
Was liked: 151
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
Quote:
Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much.
Does anyone know if there are plans in China or the US to lower the komi to 6.5?


With area scoring rules, you have 361 pts to share, so if there's no seki, the score difference on the board is always an odd integer (If black score on the board is B, and white score on the board is B, then W+B=361)

The only case where there's a difference between 5.5 and 6.5 komi is when there's an odd number of irreductible dame.

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #8 Posted: Fri Aug 28, 2020 9:11 am 
Lives in gote

Posts: 394
Location: Netherlands
Liked others: 195
Was liked: 106
Rank: EGF 3d
Universal go server handle: gennan
Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules.

But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi.

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #9 Posted: Fri Aug 28, 2020 11:40 am 
Honinbo

Posts: 10676
Liked others: 3594
Was liked: 3339
gennan wrote:
Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules.

But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi.


Well, why not a komi of 7 points? Even pros would probably have ties less than 5% of the time. Chess gets by with a fairly high rate of draws.

I still like Button Go, though. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


This post by Bill Spight was liked by: gennan
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #10 Posted: Sat Aug 29, 2020 12:26 am 
Oza
User avatar

Posts: 2338
Location: Tokyo, Japan
Liked others: 2241
Was liked: 1314
Rank: Jp 6 dan
KGS: ez4u
@lightvector

How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #11 Posted: Sat Aug 29, 2020 3:48 pm 
Lives in gote

Posts: 654
Liked others: 108
Was liked: 789
Rank: maybe 2d
Bill Spight wrote:
I still like Button Go, though. :)


KataGo supports area scoring + button Go as an evaluation method. I have not studied what its evaluations are or how they differ from those here (I suspect they would largely match the Japanese rules evaluations, but I haven't checked). I welcome anyone to do their own study. :)

ez4u wrote:
@lightvector

How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard.


Generally KataGo played easily 10x (order of magnitudewise) as many 19x19 games as any other board size. But also smaller boards are easier to learn - whereas if you only a have a few GPUs it takes weeks or months to reach pro level on 19x19 for example, I think it could be done on 9x9 in days. Also an enormous amount of knowledge generalizes between the board sizes, especially on the larger sizes. You can see this in the smoothness of the winrate estimates for the larger board sizes. If it were systematically the case that smaller board size play were abnormally "lower-quality" by a lot than the 19x19 play, you'd expect that empirically in self-play a given degree of komi-unfairness would not result in as pronounced an advantage for one side or the other, and then the neural net's output would reflect those statistics (fitting smooth-ish simple curves to statistics is very easy for neural nets), so you might expect a discontinuity in the sharpness of the winrates as you go from 19x19 to any smaller size. But to the contrary it looks quite regularly increasing in sharpness, as you'd expect if quality of play was fairly regular and the dominant effect was simply the higher variance of the large board due to the board being bigger / games lasting longer, given comparably high-quality play.

This makes intuitive sense to me. I'd think that 95+% of the "knowledge" between the board sizes is shared for board sizes 13x13 and larger. You learn the subtle ways that playing on 15x15 should adjust your overall judgment of territory vs influence and other fuzzy aspects of evaluation, and then modulo those shades of difference in judgment, almost everything you know about fighting and good shapes and what moves are effective in obtaining different kinds of trades carries over.

You can take the winrates as a fairly direct reflection of the self-play training statistics. For example on 9x9 with Japanese rules, KataGo must have experienced roughly fair results with 6 komi, and adjusting it merely to 5.5 komi gives black 70%/30%, more than a 2-1 advantage, whereas adjusting it to 6.5 gives white 74%/26%, almost a 3-1 advantage. While more extreme komis than this will have increasingly limited data, there should be quite many games at all of 5.5, 6, and 6.5, so presumably these are roughly the statistics from the games. Also self-play conditions have added noise and use only 200-1000 playouts - so basically you can say that slightly drunk ultra-blitz KataGo is roughly of a level on 9x9 where, at least against itself, a half-point advantage somewhere between doubles and triples your winning likelihood over the opponent. With the tens of thousands of playouts you'd get in match conditions, it should be yet much sharper.


This post by lightvector was liked by: gennan
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #12 Posted: Sat Aug 29, 2020 4:07 pm 
Lives in gote

Posts: 654
Liked others: 108
Was liked: 789
Rank: maybe 2d
One thing I find interesting:

For 3,4,5,6,7 you see a pattern where the odd sizes have a larger advantage for black than the even sizes. Something to do with tengen being a uniquely effective move on small boards, whereas when the center is a 2x2 square, white can also split the board and you get a more balanced result? It sort of makes intuitive sense. But, size 8 breaks the pattern. It has a *larger* advantage for black than either 7 or 9.

Size 9 is the last size in which tengen is a good move - and in fact, contrary to what you hear circulated in some forums, it's not even necessarily the best move - size 9 is where non-tengen moves finally "catch up" and become just as good in quality, 4-4 and 4-5 openings are probably also draws under fair komi. After size 9, non central moves overtake tengen. Past size 9, it also becomes increasingly clear that the bots are no longer "nearly solving" the game, and the game becomes smoother and not dominated by one single highly swingy tactical fight. Both players have enough choices of how to play and maintain average move efficiency that aren't dictated by a high density of one-way-street tactics, that I'd conjecture that perfect komi remains near 6 or 7 perpetually after, near the practical fair komi. As opposed to being much more jittery like the small boards and it merely being beyond anyone's reach to find the needle-in-a-haystack line that makes you multiple points better than the current komis that appear to be fair.

So anyways, that would seems to leave board size 8 as an oddity, where the single-big-tactical-fight of the game results in black having a bigger advantage despite an even size, rather than a smaller advantage, and one of the last board sizes for a chance for this to happen before one might conjecture smooth averaging behavior to start to dominate.


This post by lightvector was liked by 3 people: ez4u, gennan, Maharani
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #13 Posted: Wed Nov 04, 2020 3:30 pm 
Judan

Posts: 6686
Location: Cambridge, UK
Liked others: 432
Was liked: 3663
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Great stuff lightvector. I didn't realise KataGo now prefers white on Japanese 6.5, it used to prefer black. The 20b network I just tried in the BadukAI app still does. Any idea when this changed?

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #14 Posted: Thu Nov 05, 2020 11:08 am 
Lives in gote

Posts: 654
Liked others: 108
Was liked: 789
Rank: maybe 2d
Uberdude - Not sure. But all the nets are here, feel free to check. :)

https://d3dndmfyhecmj0.cloudfront.net/g ... index.html

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group