KGS Ranking adjustment?

[Bill Spight]

This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run.

so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi

There is another problem with "handicap" in this case (faux H1). Suppose a player plays 100 games like this and loses all by 3-4 pts (no komi). This would correspond to winning all by 3-4 pts (with komi). I doubt there is a completely correct handling of such results.

Why do you think that the result you would get (100 losses) would be incorrect? (OC, the failure to promote or demote the players with such a streak is not correct. You seem to be assuming that.)

OC a typical system calls this 0-100 and drops his ratings. But in another universe he decides to reject "handi" games and play even (same opponents, same move sequences). There these (the same performance) are called 100-0 and huge rating gain.

Why would the handicap not change during the run in the alternate universe, as well?

IOW, what is the problem?

[gennan]

I just realised that I misread something on https://wismuth.com/elo/calculator.html#elo_diff=250. The 50 Elo in the table at the bottom of the page was for IGS (PandaNet), not KGS. The winrates/Elo values for KGS are similar to EGF around 3k and 5d according to that table. Sorry about that.

[gennan]

This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run.

so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi

There is another problem with "handicap" in this case (faux H1). Suppose a player plays 100 games like this and loses all by 3-4 pts (no komi). This would correspond to winning all by 3-4 pts (with komi). I doubt there is a completely correct handling of such results.

OC a typical system calls this 0-100 and drops his ratings. But in another universe he decides to reject "handi" games and play even (same opponents, same move sequences). There these (the same performance) are called 100-0 and huge rating gain.

I think results like that are highly unlikely. There is allways statistical variation / ramdomness in playing quality. That can be expressed as a standard deviation of ranks. I estimate the standard deviation around shodan is roughly 1 rank. So a strong shodan playing a 100 game match against a weak shodan on josen will almost never end 100-0.

A simplified model for this would be to have a coin flipping match where the coin is fair or slightly biased, so that the heads/tails odds are 50/50 (match on josen = fair handicap) or 56/44 (match on even = odds favouring the strong shodan). It's possible to toss heads 100 times in a row with such coins, but it's highly unlikely. With the fair coin (50/50 odds), it will happen about one time in a million matches of 100 games. Playing 100,000,000 games of go would take about a 100,000 years when they play 3 games a day and take a day off on Sundays, so both players will probably have died from old age before it happens. With the biased coin (56/44 odds), it will happen more often, but still I think it's not going to happen within their lifetimes.

These are all back-of-the-envelope calculations (in fact, I didn't even use that) and go matches may not be exactly the same as a coin flipping matches, but it should be a decent first approximation for some ballpark estimations.

[jlt]

I divided ranks into two, so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi, etc. Instead of complicating the calculation of ratings, I had each game count the same number of points, and increased the point range of each successive rank by 5%, going upwards. Since most of our players were in their 20s I added inflation points to the winners' scores.

My guess is that any simple ranking system like that will work if most of the games are played with the proper handicap. But the question is how many points you should give to the winner and to loser in the case of an even game between players of different ranks.

[Javaness2]

I never really understood why the weight of handicap games wasn't reduced. Knocking off 10% of the normal contribution per handicap stone shouldn't be too radical an approach to acknowledging the additional uncertainty introduced by handicap go. KGS blocks anything greater than 6 stones

[Bill Spight]

I divided ranks into two, so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi, etc. Instead of complicating the calculation of ratings, I had each game count the same number of points, and increased the point range of each successive rank by 5%, going upwards. Since most of our players were in their 20s I added inflation points to the winners' scores.

My guess is that any simple ranking system like that will work if most of the games are played with the proper handicap. But the question is how many points you should give to the winner and to loser in the case of an even game between players of different ranks.

Actually, I did work that out for mismatches up to 3 ranks, after which it became ridiculous, based upon different plausible distributions. But nobody played any rating games with rating mismatches. OC, that could be a problem if one player is known to be stronger than his or her rank. Why play a rated game against that player if, say, you could win only 5 pts. or lose 200 pts.?

Edit: I didn't mind, myself. For two reasons. 1) I didn't mind. 2) Because of the the greater point range for the upper dan levels, it was easy to absorb a larger ratings loss than usual without affecting the rank.

[jann]

I think results like that are highly unlikely.

Sure, that is an overexaggerated example, but the underlying problem seems valid to me.

A simplified model for this would be to have a coin flipping match where the coin is fair or slightly biased, so that the heads/tails odds are 50/50 (match on josen = fair handicap) or 56/44 (match on even = odds favouring the strong shodan). It's possible to toss heads 100 times in a row with such coins

IMO the odds are not this low. Any board result in the -1 - -6 range would suffice.

Why do you think that the result you would get (100 losses) would be incorrect? (OC, the failure to promote or demote the players with such a streak is not correct. You seem to be assuming that.)

This is just an extreme example, but the delay of promotion can be explained by the system updating only once a day, for example.

IOW, what is the problem?

The problem is that the player's new rating (9k or 9d) will depend solely on whether he accepted "handi" or played even. And not on his strength or skill (games, opponents and move sequences being identical).

[Bill Spight]

Why do you think that the result you would get (100 losses) would be incorrect? (OC, the failure to promote or demote the players with such a streak is not correct. You seem to be assuming that.)

This is just an extreme example, but the delay of promotion can be explained by the system updating only once a day, for example.

One reason for making the calculation of the new rating so simple was that the players could do it themselves, after each game.

IOW, what is the problem?

The problem is that the player's new rating (9k or 9d) will depend solely on whether he accepted "handi" or played even. And not on his strength or skill (games, opponents and move sequences being identical).

As I mentioned above, I worked out tables for mismatches, but the players preferred to play rating games with the prescribed handicaps. Especially since it was always possible for them to know their ratings instantaneously.

There was no tradition of playing games with the wrong handicaps.

[jann]

I didn't mean the example specific for that system, but as a general problem in interpreting the results of "h1" games.

[dfan]

I never really understood why the weight of handicap games wasn't reduced. Knocking off 10% of the normal contribution per handicap stone shouldn't be too radical an approach to acknowledging the additional uncertainty introduced by handicap go. KGS blocks anything greater than 6 stones

The AGA system does treat larger handicap games as being less informative.

[Bill Spight]

I didn't mean the example specific for that system, but as a general problem in interpreting the results of "h1" games.

Excuse me, you've got me confused. First, what is the problem except for not updating often enough? Which is a problem for any system that does not do so. (The New Mexico system updated after every game. Something you did not know.) Second, what do handicaps have to do with anything? You can get the same thing with even games.

You have two players who are close in strength who play a long series of games in which the weaker player just barely loses every time, and the rating system does not update their ratings until afterwards. As a result, their ratings are widely apart and do not reflect their true difference in skill.

Hmmmm. Now suppose that their ratings are actually updated after every game. With even games their ratings still diverge quite a bit. With a handicap system, the handicap changes, which will change who wins. The weaker player will get a larger handicap and start winning. Then their ratings will start to return to their previous values. At some point they will be back to their original handicap, and their ratings will start to diverge again. And so on, and so on. The handicaps could flip back and forth a number of times during the run, but their ratings will not diverge widely, as will happen with only even games.

[jann]

I think we talk about different things.

With higher handicaps the games themselves differ, so there is no real problem in blindly accepting the win/loss result as THE result, and update the ratings for that.

But with "H1" against the same opponents, all the games move by move can be completely the same whether the games were played as no-komi games or as even games (OC the player may play differently but not necessarily so, especially not in the ama dan range).

So if board results in the range of -1 and -6 are somehow over-represented (like the extreme example above), the assigned rating will mostly depend on something that has absolutely nothing to do with the player's strength (whether he accepted "h1" or they played "even"). This is not a simple math/rating problem but a more general one (and OC if you adjust after every game the problem is less significant).

[Bill Spight]

This is not a simple math/rating problem but a more general one (and OC if you adjust after every game the problem is less significant).

OK, let me generalize. Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contest, results with no variability will defeat the method.

[jann]

Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contest

The percentage of correct answers is an exact, factual data (just like the percentage of various board scores). The percentage of wins (given those board scores) depends on an arbitrary parameter "komi".

[jlt]

@jann: normally, if a strong 1 dan plays against a weak 1 dan, then his winrate with komi 0.5 will be just a bit lower than his winrate with komi 7.5, so for the rating system to be fair, he should be awarded a little more points for a victory with komi 0.5 than for a victory with komi 7.5. How much is "a little more points" is not easy to determine, this has to be calculated using experimental data (see the link in the first post by gennan).