I have similar opinions about the AGA system, largely based on running the WHR rating algorithm on American Yunguseng Dojang games (there are over 5000 serious even games by now) and seeing how the resulting ratings compare with AGA ones.

There are a few problems that I believe I observe about the rating system. One is that the emphasis on handicap games means that it's hard for the system to be incredibly surprised at any particular result. In a chess tournament, you might be (the equivalent of) 10k and beat a couple of 5ks, and rocket up. If those are 5-stone handicap games, the fact that you won a couple of coin-flips isn't so impressive. Add to this a large amount of inertia in the rating system (and apparently the fact that it imposes unreasonable demands on winrate) and you get a relatively stagnant rating pool.

There is thankfully a pressure relief valve in the form of a rule that if you enter a tournament at least three ranks up from your established rating, and win at least one game, your rating resets entirely. The bad news is that this seems to be by far the most efficient way to get even a modest increase in skill recognized. If you are rated 4k and think you are 3k, I believe that the best way to achieve it is not by regular play but by entering tournaments at 1k until you win one game. That doesn't feel like a healthy system.

It goes against the grain, I suppose, but I have long believed that ranking in terms of handicap stones, despite the fact that skill in giving or taking handicap stones is variable, it better than ELO ratings or score differences in differentiating skill levels. The main reason for my belief is that, over a large range of skill levels, handicap stones have a roughly linear relationship. E. g., if player A is three ranks stronger than player B, that means that a handicap of 3 stones with White giving komi normally gives each player around a 50-50 chance of winning the game. If player B is four ranks stronger than player C, then if A gives C a handicap of 7 stones, White giving komi, that will also normally give each player around a 50-50 chance of winning. Elo ratings, OTOH, will tell us that A will crush C in an even game.

I think go ranks have no meaning without handicap. Handicap defines ranks. If go had no handicap system and all games would be even games, we would just use an Elo rating system.

But the go rating systems all try to align winrates to handicaps.
{snip}

This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run. I do think the go rating systems are reasonably effective in this respect, but there is clearly room for improvement.

This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run.

so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi

There is another problem with "handicap" in this case (faux H1). Suppose a player plays 100 games like this and loses all by 3-4 pts (no komi). This would correspond to winning all by 3-4 pts (with komi). I doubt there is a completely correct handling of such results.

OC a typical system calls this 0-100 and drops his ratings. But in another universe he decides to reject "handi" games and play even (same opponents, same move sequences). There these (the same performance) are called 100-0 and huge rating gain.

There is another problem with "handicap" in this case (faux H1). Suppose a player plays 100 games like this and loses all by 3-4 pts (no komi). This would correspond to winning all by 3-4 pts (with komi). I doubt there is a completely correct handling of such results.

OC a typical system calls this 0-100 and drops his ratings. But in another universe he decides to reject "handi" games and play even (same opponents, same move sequences). There these (the same performance) are called 100-0 and huge rating gain.

I divided ranks into two, so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi, etc. Instead of complicating the calculation of ratings, I had each game count the same number of points, and increased the point range of each successive rank by 5%, going upwards. Since most of our players were in their 20s I added inflation points to the winners' scores.

My guess is that any simple ranking system like that will work if most of the games are played with the proper handicap. But the question is how many points you should give to the winner and to loser in the case of an even game between players of different ranks.

I think results like that are highly unlikely.

A simplified model for this would be to have a coin flipping match where the coin is fair or slightly biased, so that the heads/tails odds are 50/50 (match on josen = fair handicap) or 56/44 (match on even = odds favouring the strong shodan). It's possible to toss heads 100 times in a row with such coins

Why do you think that the result you would get (100 losses) would be incorrect? (OC, the failure to promote or demote the players with such a streak is not correct. You seem to be assuming that.)

IOW, what is the problem?

This is just an extreme example, but the delay of promotion can be explained by the system updating only once a day, for example.

The problem is that the player's new rating (9k or 9d) will depend solely on whether he accepted "handi" or played even. And not on his strength or skill (games, opponents and move sequences being identical).

I never really understood why the weight of handicap games wasn't reduced. Knocking off 10% of the normal contribution per handicap stone shouldn't be too radical an approach to acknowledging the additional uncertainty introduced by handicap go. KGS blocks anything greater than 6 stones

I didn't mean the example specific for that system, but as a general problem in interpreting the results of "h1" games.

This is not a simple math/rating problem but a more general one (and OC if you adjust after every game the problem is less significant).

Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contest