Yes, exploring atypical scenarios is not be very useful to evaluate rating systems performance. For statistical models, scenarios are not very useful when it's anecdotal (or highly unlikely).

You need a lot of real world data to create or evaluate rating systems.

You may analyse the data to extract its typical characteristics.
This can be used to create a statistical model of the data.
Preferrably, you want to find the simplest data model that still captures the most important characteristics and statistical behaviour of the data.

You're looking for probability distributions that model typical human players by only a few parameters. The most important parameter being the rating.
Other parameters may include the standard deviation of the player's rating or even a full history of previous game results, nationality, etcetera.
You can also incorporate some other stuff and some theoretical consideration in your models. In the case of go, there are declared ranks, komi, handicap, time settings, hypothetical perfect play, improving players, passage of time, etcetera.

A good data model helps to create robust algorithms for the experimental rating system (in particular, how it updates the ratings when it processes game results).
You may then feed the data into your experimental rating system as a simulation of the real world and evaluate how the rating system performs over time (accuracy of predictions, stability, rating drift).

You can iterate this process to improve your experimental system until you're satisfied.

I did go into some scenarios in this thread, like the discussion about the 6k player playing against opponents which were some Elo distance away from him.

I said that the predicted winrates of the AGA system are much to high and that a match between a 6k and a 7k would bring the AGA ratings of these players closer together (the 6k player's rating would go down).

That is true, but it's not really a good scenario. Considering only 2 players is not enough to analyse the system. It ignores too many things that happen when there are more players in the system.
For example: If you add a 5k players and have these 3 players play many games with too high expectations of winrates, the 6k player's rating will not deflate. He will gain rating points from his games with the 5k player and he will lose rating points from his games with the 7k player. But overall, his rating will stay the same if he plays the same number of games against both opponents. What does happens is that the 5k's rating goes up and the 7k's rating goes down. So the rating system's high winrate expectations contracts the ratings toward the middle.

If you extrapolate this to a system with many players, it will contract the ratings towards the median rating. The median rating depends on the demographics of the player polulation in the system. In the EGF system, 5k is the median rank in tournament games.

So it's not really true that all players' ratings get deflated by high winrate expectations. A more accurate statement would be that only ratings above 5k will deflate over time in the EGF system. But the system is pretty much anchored at the top, so over time, the deflation above 5k will push everybody below downwards as well.

I knew this when I discussed the 6k scenario, but I didn't mention it then, because I tried to keep it simple. But perhaps it's better to not oversimplify things.

Many thanks, jann. I think I understand your ideas better.

Many thanks, xela. You have shed light on the questions involved.

And thanks to others, as well. I don't want to leave anybody out who has contributed to this discussion, but I am only responding with jann and xela's latest notes in mind.

OK, nothing really specific. I may say more mañana, but I actually have a life. I think.

----

A rating system is, in a way, a fool's errand. Why is that? Because it pretends that we can represent a player's strength with a single number. We can't.

Everybody is familiar with the situation, even if we don't personally know of one, in which Player A can usually beat Player B, who can usually beat Player C, who can usually beat Player A. If we could represent the strength of each of these players by a single number, the player's rating, then Player A's rating would be greater than Player B's rating, which would be greater than Player C's rating, which would be greater than Player A's rating. Tilt! No puedo, señor.

Why are such situations possible? One reason is that there are several skills and other factors that combine to form skill at go. For instance, there is skill at reading, but actually, there are at least three skills which produce that one. There is skill at invading, skill at sabaki, skill at utilizing thickness, etc. And each of these skills are probably composed of other skills, as well. In addition there are factors such as memory, emotional control, discipline, physical fitness, alertness, etc. If we can represent each of these skills and factors with a single number, we still cannot reduce all of those numbers to a single number. That is why I said that skill at go is a vector, with a number for each factor that makes up a person's go skill.

However, as xela alludes to, other players also matter. For instance, if Player A never played Player C in the situation above, we would happily think that Player A's general go skill was better than Player C's. So really, we should probably think of a player's go strength as a matrix.

Now, the situation with Players A, B, and C, is not so unusual if they are all amateur shodans. But it would be quite unusual, perhaps almost impossible, if Player A were a shodan, Player B were a one kyu, and Player C were a two kyu. There may be a certain two kyu who can regularly beat a certain shodan, playing even, but I have never heard of such a case. In any event, I am willing to say that there is some difference in go skills such that the weaker player will never beat the stronger player on a regular basis. What this means is that we can order some players on the basis of general go strength, but not all players. We say that go strength is partially ordered.

Ratings, OC, are numbers, and are this completely ordered. Since they are taken to represent go strength, which is only partially ordered, nobody has a precise rating. It is not that there is some uncertainty about a player's rating, which could be reduced with more games played. There is an irreducible uncertainty, such that nobody has a precise rating. Any attempt to assign precise ratings is doomed to failure. As xela points out, "better" is ambiguous.

----

What about ranks? Player's ranks are still not completely ordered. It is not unusual for two players of different ranks to play even with each other. It is unusual for an amateur of a lower rank to regularly beat a player of a higher rank, however. So ranks are pretty good indicators of general go skill. They are not nearly as precise as ratings, but that is a good thing, IMO. Ranks better reflects reality than ratings.

The earliest numerical ranking system I am aware of goes back centuries. A one rank difference meant that the higher ranked player could normally take White to make the game even. A two rank difference meant that the higher ranked player alternated between taking White and giving two stones. A three rank difference meant that the higher ranked player gave two stones. Etc. These were pro ranks, OC. Over time pro ranks got closer together. Today the idea that a pro 8 dan would give 4 stones to a pro shodan is absurd. It was absurd 100 years ago, as well. 3 stones was not. Today there is no guarantee that a pro 8 dan can give a pro shodan 2 stones. Times change.

But going back to ancient times, two pros might agree to a lengthy match of several games, often 10 or more. It was normal in these matches for the handicap to change after four wins in a row, regardless of the official handicap. For instance, a player might go from playing Black all the time to playing Black in two games out of three and White in the other.

When I was learning go it was usual for amateurs to change handicaps after a three game winning streak. For instance, the players might be playing even by alternating between taking White and Black, and then, after one player won three games straight, to that player playing White. And then if White won three games straight, she would give two stones. That change was a bigger leap, OC, but we were not worried about niceties.

Note that we usually played even by alternating between taking White and Black, not by komi. Some players used komi, but most of us did not bother with it.
----

Against that background, the idea of one player playing Black and the other playing White for 10 games without changing the handicap after Black won 3 or 4 games in a row is bizarre. Komi or no komi. It is a made up scenario to produce a problem that makes no practical sense. I won't say that the problem does not exist, because jann sees one, but I don't.

Now, with some people playing many games a day online with a rating system that, for some reason or other, does not recalculate ratings immediately, something like that scenario could occur. In fact, with thousands of games played on different servers every day, such a scenario may not be all that unusual. But first, it is ephemeral, as modern rating systems are self-correcting. And second, since precise ratings are a pipe dream, anyway, who cares?

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

My two main guides in life:
My mother and my wife.

Everything with love. Stay safe.

Did you mean the opposite? Given that your next sentence was "So the rating system's high winrate expectations contracts the ratings toward the middle."



I don't understand this sentence. What do you mean by "anchored at the top"? Why does everyone get deflated?

As I wrote several times, if there are a huge number of samples, with ratings adjusted afer every game, a rating system is fine in practice even ignoring the issue with "H1" games. But there are scenarios where this is not the case (eg. new player with few games), and I also found the fact that many of these games have dual winners interesting (esp. compared to chess). Apparently I'm the only one.



For a random thought experiment I could even imagine a system that does not throw away most of this data, ie. a server that ignores komi, records each game as board results B+3, B+8, W+1 etc, and manages player ratings using the whole result from each game (instead of truncating to 1-1 bits, which is dubious for H1). It may even be more successful in managing player matchups and handicaps (esp. if it doesn't even allow komi to be set, though I think weak amateurs would not play much differently because of 6 pts, until late endgame at least).

Back to what started this conversation --



If the AGA can put resources behind this, it strikes me that this would be a great topic for a Kaggle competition. Can you publish a big file with the (anonymised/de-identified?) results of all games played in 2019 plus some metadata? Metadata might include handicap, komi, time settings, AGA ratings of the players where known, exact date and time when the game was played, which country/region people were logged in from if known.

I suspect the correlation between AGA and KGS rank would be weak, as some people play better online than over the board, some people the other way round, and different people take online games more or less seriously.

I also wonder whether you'd find cliques in the KGS players. For instance, group A plays mostly fast games, group B plays mostly slow games, they hardly ever play against each other, and 2k in group A is a different strength from 2k in group B. Or it could split up by time of day, country, or something else.

The goals of the Kaggle competition could be to design a better ranking system from scratch, or to suggest how to improve the current system without radical change, or to explain the various factors making ranks (appear to be) unstable or inconsistent.

I confess that I do not understand much of what has been written under this topic. In hopes of getting a little closer to the topic of KGS' ranks, and adjustments thereto, I offer the following data.

The three Ayabot00X bots (ayabot001, ayabot002, and ayabot003) have been playing steadily since 2014 on KGS. I did the following:

1. Downloaded the cvs files for these three from KGS analytics.
2. Deleted all free games and games against unrated players, leaving some 624K games in total.
3. Replaced all positive handicap with an equal negative number where the ayabot played White (= giving handicap)
4. For all handicap = 0 games where the ayabot was one level higher and playing W, substituted handicap = -1 (i.e. assumed that ayabot received komi = 0.5; the csv file does not list komi, just handicap)
5. For all handicap = 0 games where the ayabot was one level lower and playing B, substituted handicap = 1 (i.e. assumed that ayabot gave a komi = 0.5)
6. Totaled the games and wins for each handicap and for Black versus White.
7. Calculated the winning rates and compared the winning percentage at each handicap for Black versus White.

We can see in the results table that the bots won significantly more games as White than as Black at all handicap levels. This was surprising to me. I expected that the bias that favors White in assigning traditional handicaps would favor White at all levels except even games (in the list handicap = 0). Therefore I expected that White would win approximately 50% of the handi = 0 games and a higher percentage of the remainder. This did not turn out to be the case. The ayabots won 56% of their games as White with handi = zero and 58% as White with handi = 5 (stones). Note that we have to be careful with the handi = 6 figures; the ayabots played with a maximum handicap of 6 stones so these results may be "noisier" than the rest.

I honestly do not know what to make of this data. On the other hand it only took me about an hour and a half last night, start to finish, to download it an produce the calculations. I would think that this is a vast, easily accessible treasure trove of information. Recall that when Remi published his paper on Whole History Ratings, he used 10.8 million KGS games in his work! I think that for discussion of rating/ranking systems to be used in real life, such data is a better basis than hypothetical Andrew-Bart-Chuck round robins!

These bots played at slightly different levels but always as sdk's. This table shows the breakdown of games by kyu level.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

The robot players cannot handle handicap games 'correctly'. It's a bad set of data to pick.
It would be nice if KGS could simply block handicap games with bots.

_________________
North Lecale

Well, it should not be. For a half stone rank difference the proper handicap uses 0.5 komi or 0 komi. Because the traditional handicap for a one amateur rank difference was for the lower ranked player to take Black without komi, 50 years ago most amateurs did not know that that was the wrong handicap. Now they should know that White should give komi for a proper handicap. If that is not general knowledge I blame those who run go servers and tournament organizers.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

My two main guides in life:
My mother and my wife.

Everything with love. Stay safe.

I am curious. Which part of the results do you think indicates this lack of ability?

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Now I think I know why you think that there is something special about H1. You have been misled by those who say that the proper handicap for a one amateur rank difference is for the lower ranked player to take Black with ½ pt. komi. In fact, the proper handicap is for White to give komi. That is so for all handicaps, not just H1. Given incorrect knowledge of proper handicaps, there is something special about H1 with Black giving a 6½ or 7½ pt. komi.

Should there be a rating system based upon gradations of komi? Such a system is certainly possible. It is unclear how much that buys you, however, given the irreducible uncertainty of ratings, especially amateur ratings.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

My two main guides in life:
My mother and my wife.

Everything with love. Stay safe.

That's not true in any respect.

(a) In traditional play there were no separate amateur ranks. They used the same grade systems as pros.

(b) They had non-komi handicaps using colour alternation in series of 2,3 or 4 games. That's why they had grades. The system is well known here from Japan, but it was common in China. Indeed they must have invented it. Grades are mentioned as early as the Wei dynasty (3rd century), and as for the mention of series, we have this explained from the hand of Ming Dynasty author Xu Zhongye (Ming was 1368~1644), e.g. "A grade 1 [=9d] gives him [a grade 2 = 8d] first move in two out of three games."

(c) After WWII separate amateurs emerged as a result of the 1949 Japanese rules and their first tournaments used 4.5 komi.

(d) Both amateurs and professionals in Japan knew early on that a komi handicap was needed to make a single game properly equal. The earliest known komi game is 1751 involving what we may call amateurs (i.e. unofficial players), though go may have been their professionals gamblers. And in 1759 we have an example of White giving komi. Shusaku and Shuwa also famously discussed komi and the earliest komi game (1832) involving pros includes Shuwa.

(e) Western amateurs knew about komi at the very least in the 1960s (first international amateur tournaments).

(f) What no-one really knew was simply how much komi should be. Setting it at around 5 points was probably out of deference to the two Shus, but smaller komis were tried, especially in the 30s, though sometimes in conjunction with the series-type handicap.

No AI bots were harmed in the writing of this post.

Not at all, I'm perfectly aware of the half stone (komi) error of traditional handicaps. And as you wrote, this equally affects all handicaps (and is easy for rating math to account for).

What's special about H1 games is that both komi and no-komi is very common for half rank difference, open for the players to choose, and in fact I have seen several cases where one player were even unaware if the game he plays has komi or not (set automatically). The players simply play the same game on the board, but the recorded 1-bit result (which is all a typical rating system sees) is determined by this (external) setting for the B+1 - B+6 range.

Well then there is your problem, right? That's why you say that the choice of komi is arbitrary.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

My two main guides in life:
My mother and my wife.

Everything with love. Stay safe.

Then we are talking about different traditions. I'm talking about the one adopted later by amateurs.

IMX, amateurs in both Japan and the US typically used Black first as the correct handicap for a one stone difference in strength. (Edit: At no time in my experience did any amateur, whether dan or kyu, whether in Tokyo, Kyoto, Los Angeles, New York, San Francisco, or Santa Fe, whether at the Nihon Kiin or Kiin associated clubs, or informally, at home, tell me that taking Black with no komi was not the correct handicap for a one stone amateur rank difference. No pro told me that, either.) Finally, in the late 1970s I introduced White giving komi in New Mexico. Even as recently as the late 1990s on rec.games.go I had to argue that simply letting Black play first was the wrong handicap for a one amateur rank difference. Don't tell me there was no such amateur tradition.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

My two main guides in life:
My mother and my wife.

Everything with love. Stay safe.

Well, I find it spooky to talk about something that happened in my lifetime (and yours) as a tradition, especially in a field that is already very ancient. There is something similar perhaps going in the antiques field. We have tv programmes here in which experts bemoan the trend to class objects 50 (or even 25) years old as antiques, instead of using the 100-year criterion that has long been used in their field. By that modern trend both you and I would be classed as antiques. I don't feel like one, and I very strongly suspect you don't either (maybe for simplification we'll leave getting out of bed in the morning out of it )

But that was just scene setting. My main focus was

That would have been around the time I started go. Early on then, I came across the German system of counting grades with simple numbers, 1 down to whatever, splitting each Japanese grade in two and making no distinction between dan and kyu. I think 1-dan was 18/19 from memory.

Japanese visitors found this (and other novelties) very peculiar and would write articles in Kido and so on, describing western practices in the style of "dogs there can walk on their hindlegs." The thought police hadn't reached there yet. But even so they were a bit misguided, because they didn't take enough account of the fact that we had relatively few opponents, and so had to play lots of handicap games. They also overlooked the fact that we had few ways of getting a reliable rating, and in particular few ways of changing a rating quickly. The German system, sporadically adopted elsewhere, was an attempt to provide a workable ratings framework for Europe.

It didn't really work - for several reasons, I suppose. It was a creature of mathematicians, and not all go players were mathematicians. Most other players seemed in thrall to the Japanese system and just mimicked it, without properly realising it was not fit for purpose here perhaps, or, much more likely, if they did realise, they didn't care - "we are just weak amateurs, really" (and we really were then!) and "it's a storm in a teacup." I certainly was in the latter don't-care camp.

Yet I believe most of us - even someone as bored by decimal points as me - still registered what the mathematicians were saying: that Black as a one-stone handicap is skewgee, and, in addition, that the rest of the Japanese handicap scale wasn't anywhere near linear. There was even a translated book - wasn't there? - in which Ishida Yoshio played other pros at all the various handicaps, and showed the lack of linearity. We knew we were surrounded by imperfections.

So, my contention is that "50 years ago most amateurs did not know that that was the wrong handicap" should have read "50 years ago most amateurs did not CARE that that was the wrong handicap."

I would also go further and say that is even more the case nowadays, since we can now find opponents easily and don't need to rely on a handicap system to get a rating.

That might not seem to marry with the representation of the go world currently given by L19, but I still do believe most go players in the real world are still not mathematicians and computer scientists.

Watch it, Sonny!



If you fall asleep in your chair you don't have to go to bed.



There is evidence from extant handicap game records that, like the German system except for the naming of ranks, the Japanese system in Dosaku's time used half-stone rank differences.



Good point.



Right. I think that it was unfortunate that so many Western go players in the mid-20th century were mathematicians. It sort of put people off until the hippies came along and thought that go was some kind of mystical right brain game. (At least, in the US. ) Still, in the US I have found that go players are rather cultured by comparison with the rest of the population. I could drop a Shakespeare quote at the bridge table and people would go huh? While over supper go players would discuss the interesting properties of the verb, cleave.

Edit: I doubt if the amateur "tradition" started as late as after WWII. I read in a book by Segoe his opinion that an amateur could feel satisfaction as a shodan because that meant that he would take only four stones from a pro. I read that at a time when pros were giving amateur shodans seven stones. Taking amateur rank inflation into account, I think that pushes the time when amateur ranks separated from pro ranks and adopted one handicap stone per rank as the standard into the late 19th century.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

My two main guides in life:
My mother and my wife.

Everything with love. Stay safe.

I read what their developer's wrote.

_________________
North Lecale

Yes, I did mean the opposite .


I'm sorry, I wasn't very clear. Anchoring, contraction and downward pressure are all different aspects of the issue.

I'll try to clarify what I mean with an example:

7d EGF is pretty much fixed in practise (the expected winrates in the EGF system are basically correct for high dans and high dan ratings have a large "mass" in the system. The rating system only updates their rating by a tiny amount per game). So once he is at 7d, his rating is not affected much by some quirks that the EGF system may have.

But from 5k to mid dan, you would have this "rating contraction effect" from exaggerated winrate expectations. The low dan ratings slowly go down (being pulled towards 5k). Suppose a 2d's rating dropped below 2100 at some point (2d EGF is supposed to be 2200 and 2100 is supposed to be 1d). His rating didn't drop because he got weaker, but just because the rating system is slowly contracting over many years. Now the 2d may eventually feel forced by his low rating to demote himself to 1d (many of his peers already preceded him).

Then there is this 5k player in his club whose skill didn't change and his rating didn't change either (even though he also plays tournaments, a lot actually, but those are mostly even games against other 5k players). His rating is not affected by the general contraction to 5k.

Since the skill gap between him and the (former) 2d remained the same in reality, the (former) 2d can still give the 5k a handicap of 6-7 stones when they play. And they play each other regularly.
So the 2d's demotion can have a domino effect in his club. The 5k knows he still needs 6-7 stones handicap and he may start to doubt his own rank. After all, the dan rank of his opponent has more weight than his own kyu rank. He may even feel some pressure to demote himself. This is what I meant with "pushing everybody below downward".

You may think this is all speculation, but I really believe such effects are really happening (very slowly, I think this process has been going on for at least a decade).

You might ask what happens if the (former) 2d plays a match against a 7d with 6-7 stones handicap (the correct handicap for a 1d against a 7d). Won't that make it clear that the 2d's demotion wasn't really called for? Perhaps, but there aren't many 7d around in Europe, so this doesn't happen nearly as much as club games between 2d and 5k. So the demographics work against such "self-correcting" mechanisms.

In the Netherlands there are still some places where that German system is used. We call those numbers "classes". I know they still use it for the ongoing blitz competition in Tilburg and I use it to calculate handicap tables for my own club. The last Dutch dan diploma that I recieved in 2005 also stated the class equivalent of my promotion (4d, class 13). It was actually once common to have half rank dan promotions in the Netherlands, but I think it fell in disuse in the last decade.