There are of course other things to observe in the raw data too.
Anyways, this was fun. Hope someone finds this interesting.
| Life In 19x19 http://lifein19x19.com/ |
|
| AI opinions on "fairest" komi by board size http://lifein19x19.com/viewtopic.php?f=15&t=17750 |
Page 1 of 1 |
| Author: | lightvector [ Thu Aug 27, 2020 9:41 pm ] |
| Post subject: | AI opinions on "fairest" komi by board size |
I did a study of KataGo's opinions about the "fairest" komi for each square board size using the final 40 block network, for every size that the 40-block network has had direct training experience on. These opinions should reflect pretty well KataGo's experiences during self-play about its empirical winning frequency on these board sizes for different values of komi on those board sizes, and although KataGo is only one bot, might be indicative of what "fair" komi would generally be among bots trained to a comparable level on these board sizes. Firstly, from separate earlier studies, we know (probable) perfect komi for smaller sizes: Code: SZ Japanese Chinese
3 8 9 4 0 2 5 24 25 6 3 4 |
|
| Author: | lightvector [ Thu Aug 27, 2020 9:42 pm ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Next my interpretation of the raw data. Most "fair" komi, in KataGo's opinion, assuming quality of play at KataGo's self-play-training level, plus some notes about that komi. Keep in mind the fact that in Chinese rules, absent seki, odd komi are the ones that matter to cross on odd boards, while even komi are the ones that matter to cross on even boards. So, on odd sized boards, 5.5, 6, 6.5 are all the same for example, while 6.5, 7, 7.5 are all different. On even boards, it's the other way around. So there's this weird oscillatory behavior for Chinese rules on board size parity. Code: SZ Japanese Chinese
7 8 9 KataGo is pretty sure about both values, to such a degree that although formal proof is vastly beyond reach, these are probably decent guesses for the perfect komi. 8 9-* 10-* KataGo thinks these values are fair, and is *very* confident that it is not higher. But as for being lower... white has high-variance tactics that KataGo thinks probably don't work, but if they do, could push the perfect komi much lower, even by multiple points. The incredible asymmetry between higher komi confidence vs lower komi confidence is fascinating. 9 6- 7-- These are the most fair, but KataGo thinks white is at least slightly easier to play in each case, especially in Chinese. 10 6- 6++ Japanese: White is slightly easier to play. Chinese: Black is easier to play. 11 6- 5.5,6,6.5+ Japanese: White is slightly easier to play. Chinese: No seki is expected, unclear if perfect komi is 5 or 7, so fairest komi in practice is in between, but more likely to be 7, so black is slightly easier to play. (7 is less fair in practice because even if the most likely perfect komi, it's so much harder for black to play than 5.5,6,6.5 are for white to play). 12 6 6.5,7,7.5-- Japanese: Pretty much fair in practice. Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is easier for white to play. 13 6+ 7- Japanese: Black is slightly easier to play. Chinese: White is slightly easier to play. 14 6+ 6.5,7,7.5- Japanese: Black is slightly easier to play. Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play. 15 6.5- 7- Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. Chinese: White is slightly easier to play. 16 6.5- 6.5,7,7.5- Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play. 17 6.5- 7- Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. Chinese: White is slightly easier to play. 18 6.5- 6.5,7,7.5 Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play. Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between. 19 6.5 7 Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5. Chinese: Pretty much fair in practice. |
|
| Author: | lightvector [ Thu Aug 27, 2020 9:48 pm ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Lastly, the raw data. These are tables of the observed winrates for KataGo from the opening position for different board sizes under different rules. This is the full range of board sizes that KataGo has had actual training on - size 7 to size 19. These were done with the final g170 40 block net with 100ks of playouts on the smaller boards, diminishing to 10ks of playouts on the large boards since those took longer, and using somewhat wider-than-normal search settings. On the large boards the results appear quite stable even if searching longer, since the search is mostly just smoothly averaging over the large variety of permutations of the opening what the neural net says based on its self-play experience. On the smaller boards, tactical complications start happening much faster, and the values never stop fluctuating as the tactics are explored ever deeper, I often just chose an arbitrary point to cut it off, different runs might get different results. Also of course, technically these are "KataGo Japanese" and "KataGo Chinese" rules, which might differ from the actual Chinese and Japanese rules in sufficiently pathological positions. Japanese rules winrates: Code: SZ\KM 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5 7J 99 50 3 8J 95 94 93 91 89 89 90 85 84 50 2 9J 89 84 80 70 48 26 17 13 10J 87 82 77 70 60 47 32 22 16 11J 76 68 59 48 37 28 22 12J 67 59 51 42 35 28 13J 67 59 52 45 38 31 14J 65 59 52 46 40 34 15J 65 60 54 48 42 37 16J 63 58 53 48 43 38 17J 57 52 48 44 39 18J 56 53 49 45 41 19J 56 52 49 45 42 Chinese rules winrates: Code: SZ\KM 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5 7C 99 50 2 8C 95 92 89 88 90 79 79 80 85 50 1 9C 93 83 69 69 70 41 13 17 16 12 8 10C 93 85 76 75 77 60 31 34 32 18 7 11C 85 74 59 59 59 39 21 12C 73 59 41 43 42 29 18 13C 83 72 59 59 60 45 31 14C 71 60 45 46 46 34 23 15C 78 69 59 59 60 48 37 16C 68 58 47 48 49 39 30 17C 74 65 56 57 57 48 39 18C 65 56 48 48 49 41 33 19C 70 63 55 56 56 49 41 And the same two tables, interleaved and paired if you want to more easily compare Japanese and Chinese together for just a single komi: Code: SZ\KM 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7J 99 50 3 7C 99 50 2 8J 95 94 93 91 89 89 90 85 84 50 2 8C 95 92 89 88 90 79 79 80 85 50 1 9J 89 84 80 70 48 26 17 13 9C 93 83 69 69 70 41 13 17 16 12 8 10J 87 82 77 70 60 47 32 22 16 10C 93 85 76 75 77 60 31 34 32 18 7 11J 76 68 59 48 37 28 22 11C 85 74 59 59 59 39 21 12J 67 59 51 42 35 28 12C 73 59 41 43 42 29 18 13J 67 59 52 45 38 31 13C 83 72 59 59 60 45 31 14J 65 59 52 46 40 34 14C 71 60 45 46 46 34 23 15J 65 60 54 48 42 37 15C 78 69 59 59 60 48 37 16J 63 58 53 48 43 38 16C 68 58 47 48 49 39 30 17J 57 52 48 44 39 17C 74 65 56 57 57 48 39 18J 56 53 49 45 41 18C 65 56 48 48 49 41 33 19J 56 52 49 45 42 19C 70 63 55 56 56 49 41 |
|
| Author: | lightvector [ Thu Aug 27, 2020 9:53 pm ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
There are of course other things to observe in the raw data too. Anyways, this was fun. Hope someone finds this interesting.
|
|
| Author: | Knotwilg [ Fri Aug 28, 2020 3:49 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Fascinating that fair komi hardly changes between board size 9 and 19, and that we've known this all along. |
|
| Author: | gennan [ Fri Aug 28, 2020 4:13 am ] | ||
| Post subject: | Re: AI opinions on "fairest" komi by board size | ||
I find this very interesting! I made charts for odd board sizes 9-19. So for odd board sizes 9-19, the fairest komi is 6+ for Japanese rules and 7- for Chinese rules. With Japanese rules, 6 komi looks like perfect komi. When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then white has an advantage on smaller boards). With Chinese rules, 7 komi looks like perfect komi. When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then black has an advantage, more so on smaller boards). With 7.5 komi, white has a fairly large advantage. Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much. Does anyone know if there are plans in China or the US to lower the komi to 6.5? Has any AI research been done on area scoring with integer komi and a Button to break ties?
|
|||
| Author: | Tryss [ Fri Aug 28, 2020 7:42 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Quote: Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much. Does anyone know if there are plans in China or the US to lower the komi to 6.5? With area scoring rules, you have 361 pts to share, so if there's no seki, the score difference on the board is always an odd integer (If black score on the board is B, and white score on the board is B, then W+B=361) The only case where there's a difference between 5.5 and 6.5 komi is when there's an odd number of irreductible dame. |
|
| Author: | gennan [ Fri Aug 28, 2020 9:11 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules. But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi. |
|
| Author: | Bill Spight [ Fri Aug 28, 2020 11:40 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
gennan wrote: Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules. But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi. Well, why not a komi of 7 points? Even pros would probably have ties less than 5% of the time. Chess gets by with a fairly high rate of draws. I still like Button Go, though.
|
|
| Author: | ez4u [ Sat Aug 29, 2020 12:26 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
@lightvector How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard. |
|
| Author: | lightvector [ Sat Aug 29, 2020 3:48 pm ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Bill Spight wrote: I still like Button Go, though. KataGo supports area scoring + button Go as an evaluation method. I have not studied what its evaluations are or how they differ from those here (I suspect they would largely match the Japanese rules evaluations, but I haven't checked). I welcome anyone to do their own study. ez4u wrote: @lightvector How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard. Generally KataGo played easily 10x (order of magnitudewise) as many 19x19 games as any other board size. But also smaller boards are easier to learn - whereas if you only a have a few GPUs it takes weeks or months to reach pro level on 19x19 for example, I think it could be done on 9x9 in days. Also an enormous amount of knowledge generalizes between the board sizes, especially on the larger sizes. You can see this in the smoothness of the winrate estimates for the larger board sizes. If it were systematically the case that smaller board size play were abnormally "lower-quality" by a lot than the 19x19 play, you'd expect that empirically in self-play a given degree of komi-unfairness would not result in as pronounced an advantage for one side or the other, and then the neural net's output would reflect those statistics (fitting smooth-ish simple curves to statistics is very easy for neural nets), so you might expect a discontinuity in the sharpness of the winrates as you go from 19x19 to any smaller size. But to the contrary it looks quite regularly increasing in sharpness, as you'd expect if quality of play was fairly regular and the dominant effect was simply the higher variance of the large board due to the board being bigger / games lasting longer, given comparably high-quality play. This makes intuitive sense to me. I'd think that 95+% of the "knowledge" between the board sizes is shared for board sizes 13x13 and larger. You learn the subtle ways that playing on 15x15 should adjust your overall judgment of territory vs influence and other fuzzy aspects of evaluation, and then modulo those shades of difference in judgment, almost everything you know about fighting and good shapes and what moves are effective in obtaining different kinds of trades carries over. You can take the winrates as a fairly direct reflection of the self-play training statistics. For example on 9x9 with Japanese rules, KataGo must have experienced roughly fair results with 6 komi, and adjusting it merely to 5.5 komi gives black 70%/30%, more than a 2-1 advantage, whereas adjusting it to 6.5 gives white 74%/26%, almost a 3-1 advantage. While more extreme komis than this will have increasingly limited data, there should be quite many games at all of 5.5, 6, and 6.5, so presumably these are roughly the statistics from the games. Also self-play conditions have added noise and use only 200-1000 playouts - so basically you can say that slightly drunk ultra-blitz KataGo is roughly of a level on 9x9 where, at least against itself, a half-point advantage somewhere between doubles and triples your winning likelihood over the opponent. With the tens of thousands of playouts you'd get in match conditions, it should be yet much sharper. |
|
| Author: | lightvector [ Sat Aug 29, 2020 4:07 pm ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
One thing I find interesting: For 3,4,5,6,7 you see a pattern where the odd sizes have a larger advantage for black than the even sizes. Something to do with tengen being a uniquely effective move on small boards, whereas when the center is a 2x2 square, white can also split the board and you get a more balanced result? It sort of makes intuitive sense. But, size 8 breaks the pattern. It has a *larger* advantage for black than either 7 or 9. Size 9 is the last size in which tengen is a good move - and in fact, contrary to what you hear circulated in some forums, it's not even necessarily the best move - size 9 is where non-tengen moves finally "catch up" and become just as good in quality, 4-4 and 4-5 openings are probably also draws under fair komi. After size 9, non central moves overtake tengen. Past size 9, it also becomes increasingly clear that the bots are no longer "nearly solving" the game, and the game becomes smoother and not dominated by one single highly swingy tactical fight. Both players have enough choices of how to play and maintain average move efficiency that aren't dictated by a high density of one-way-street tactics, that I'd conjecture that perfect komi remains near 6 or 7 perpetually after, near the practical fair komi. As opposed to being much more jittery like the small boards and it merely being beyond anyone's reach to find the needle-in-a-haystack line that makes you multiple points better than the current komis that appear to be fair. So anyways, that would seems to leave board size 8 as an oddity, where the single-big-tactical-fight of the game results in black having a bigger advantage despite an even size, rather than a smaller advantage, and one of the last board sizes for a chance for this to happen before one might conjecture smooth averaging behavior to start to dominate. |
|
| Author: | Uberdude [ Wed Nov 04, 2020 3:30 pm ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Great stuff lightvector. I didn't realise KataGo now prefers white on Japanese 6.5, it used to prefer black. The 20b network I just tried in the BadukAI app still does. Any idea when this changed? |
|
| Author: | lightvector [ Thu Nov 05, 2020 11:08 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
Uberdude - Not sure. But all the nets are here, feel free to check. https://d3dndmfyhecmj0.cloudfront.net/g ... index.html |
|
| Author: | Elom0 [ Mon Aug 14, 2023 4:11 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
An update of this data on Also, I am actually quite interested in KataGo's estimation for 6x6, 5x5, 4x4 and 3x3, perhaps even 2x2 as it would be interesting to see KataGo's thought process in getting to those likely correct komi |
|
| Author: | RobertJasiek [ Mon Aug 14, 2023 6:35 am ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
2x2? To see if KataGo does worse than Bill Taylor's theorem (IIRC, Black's 1 point win under area scoring and superko)? |
|
| Author: | Elom0 [ Mon Aug 14, 2023 12:44 pm ] |
| Post subject: | Re: AI opinions on "fairest" komi by board size |
RobertJasiek wrote: 2x2? To see if KataGo does worse than Bill Taylor's theorem (IIRC, Black's 1 point win under area scoring and superko)? We can see KataGo's thought process for 2x2 on multiple rulesets! Also, an idea is instead of trying multiple komis, simply train only on no komi games and increase the weight of a win according to winning score, so when a net wins by 3 points, the effect of it's weightings on creating the new version are 3 times what it would have been if it won by one point, for example. But it means the winrate for a particular komi is just derived from the proportion of games won by black by that margin or higher in these no komi games. The total number of games between area rulesets Chinese, AGA, Ing, New Zealand, and territory rulesets Korean, Japanese, etc would be even with the runs of rulesets within those also distributed equally, however, there will often be games played using one ruleset that would be legal in another, and in those cases the results of the winning margin should included in the data for those rulesets too, or better yet, whenever a move is played that is legal in some rulesets but not others, split the run into multiple branches that comply with different rulesets. I had the thought of making dual area-territory rules the standard in lentitear, so the final score is half the difference in number of stones on the board plus the difference in territory, and since fair komi for territory is 6.5 and fair komi for area is 7, fair komi in dual area-territory scoring with each weighted at a half would be 6.75! |
|
| Page 1 of 1 | All times are UTC - 8 hours [ DST ] |
| Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |
|