AI komi

[xela]

OK, that'll teach me to take shortcuts with late-night data analysis, hoping to knock it off before midnight! I'll see if I can fix it in 15 minutes before going to work. What could go wrong?

----

Adding back in the resignations, but removing games with unequal ranks, gives 23,222 games.

OGS_result_list2.csv

(752.35 KiB) Downloaded 390 times

All 23,222 games: median score W+2.5 (after adjusting for komi, which varies between -99 and +30, but the vast majority are sensible numbers)

Restricting to 6.5 komi: 21,965 games, median score W+3.5, i.e. black is 4 points ahead on the board. Still seems to support Bill's argument.

Excluding resignations, allowing any komi: 7,866 games, median score W+1.5, mean score W+0.45.

Excluding resignations, 6.5 komi only: 7,378 games, median score W+1.5, mean score W=0.77.

Sorry I don't have time to format it into a nice table, the day job beckons.

Overall, white resigned 7,609 games and black resigned 7,747.

----

Medians! Yes, Bill, I've been paying attention to you :-)

[Bill Spight]

OK, that'll teach me to take shortcuts with late-night data analysis, hoping to knock it off before midnight!

I sympathize. How many times have I resolved not to post late at night, because of misstatements, omissions, and goofs!

[jlt]

@xela: it would be interesting to calculate the median for several categories of players: dan, SDK, 10-19k, 20-50k. The reason is that about 75% of the games are between players 20k or weaker.

[ez4u]

...Just a quick comment on the LZ set: ...
The interesting part of that dataset would be what happens inside the first 1M games (but LZ may have learned too fast to have enough data at nearly random and DDK levels, and there were also huge bugs in it around then which may distort results)...

here is a closer look at the first 2 million games. Not because I think it will explain anything, rather because it looks cool. Note that LZ continued on the 5x64 net until around 3 million games, at which point it moved up to 6x128.

Winning rates for LZ self-play games 1st 2M games.jpg (78.35 KiB) Viewed 8971 times

[lightvector]

You should be aware that some of the Leela Zero's "beginner mistakes" were utterly unlike anything a human would mistakenly do, and in specific ways that really would mess with the result.

For example, there was an early time when black would accidentally pass, and then white would pass and win. I think there were also periods where white learned that passing was "good" early in the game, as well as any time white got ahead in the number of stones (such as by captures), because sometimes black could pass in response and lose. Once black caught on to that trick though, black would win a ton of games because white would pass away the advantage every time after getting an advantage, until eventually white caught on to that it shouldn't be passing. And it was only white that could learn this "bad" strategy because it was only white that started with a positive komi such that passing might possibly seem good in some cases.

So, stupid stuff like that. The above story might not quite be right, but things "like that" were occurring sometimes I think. Careful about using early LZ to generalize about komi.

[Bill Spight]

For example, there was an early time when black would accidentally pass, and then white would pass and win. I think there were also periods where white learned that passing was "good" early in the game, as well as any time white got ahead in the number of stones (such as by captures), because sometimes black could pass in response and lose. Once black caught on to that trick though, black would win a ton of games because white would pass away the advantage every time after getting an advantage, until eventually white caught on to that it shouldn't be passing. And it was only white that could learn this "bad" strategy because it was only white that started with a positive komi such that passing might possibly seem good in some cases.

Just curious. How were those games scored on the board (befoce applying komi)? By area at the time? IMHO, proximity scoring may be an improvement.

[lightvector]

Yeah. Tromp-Taylor rules, of course. LZ was meant to replicate AlphaGoZero reasonably closely, so any change to the rules for scoring would also have not been appropriate.

[Bill Spight]

Yeah. Tromp-Taylor rules, of course. LZ was meant to replicate AlphaGoZero reasonably closely, so any change to the rules for scoring would also have not been appropriate.

Right. Thanks.

Ing adopted proximity scoring in his last set of rules, although the English translations I have seen assume regular area scoring. The only case I know of where the two differ for games that end normally is one seki shape.

[xela]

@xela: it would be interesting to calculate the median for several categories of players: dan, SDK, 10-19k, 20-50k. The reason is that about 75% of the games are between players 20k or weaker.

OK. It's interesting and baffling. In fact most of the games are players 30k or below, including one game between two players who'd achieved a 51k rank. I think this is very noisy data.

For each rank band, I've calculated four medians. In order: all games; games with 6.5 komi only; all games with resignations excluded; 6.5 komi with resignations excluded. Numbers in brackets are the number of games.

rank	all games	6.5 komi	no resign	6.5 komi, no resign
30k and below	W+14.5 (13562)	W+18.5 (13030)	W+1.5 (3891)	W+1.5 (3690)
20-29k	B+4.5 (3464)	B+4.5 (3173)	W+0.5 (1630)	W+0.5 (1499)
10-19k	W+0.5 (2718)	W+0.5 (2546)	W+1.5 (1246)	W+1.5 (1167)
SDK	B+2.5 (2926)	B+3.5 (2783)	W+2.5 (936)	W+2.5 (895)
dan	B+3 (511)	B+3.5 (421)	W+1.5 (147)	W+1.5 (121)

(This tool is handy for generating tables, although L19 doesn't seem to recognise the [th] tags, need to manually change those to [td], and I've no idea why it's inserting a massive space before the table.)