KGS ranking revisited

[HermanHiddema]

Mef, I resent being used for that kind of analysis. Couldn`t you at least have chosen some other player and WITHHELD their name? Doing what you did has a definite flavour of ad hominem about it, as did another poster`s allusion to "illusory superiority".

Well, that other poster would be me then.

And I'm sorry you feel that way. My post was in reply to one by Kaya.gs about the psychology of rating systems. It was not intended to apply to anyone personally, and it also was not meant to apply specifically to the current KGS rating system, or to people's experiences with that. It was just an explanation of, and link to a Wikipedia article about, a well known psychological phenomenon that affects all people in general, and is therefore relevant to all rating systems. It never even crossed my mind that you or anyone else would consider it a personal attack.

I have no opinion, in fact, on whether or not the current parameters for the KGS rating system are optimal. I hardly ever play online anyway. I'm just remarking that it is impossible to please everyone, and that there is, IMO, no possible set of parameters for the KGS rating system that will not result in complaints.

[RobertJasiek]

it is impossible to please everyone

It is, however, possible to please a much greater percentage of players.

[Mef]

You act like there are no "system anomalies". Even if a system like KGS is the best available (or reasonably possible) doesn't mean that it's not crap in a lot of ways. For example fake short-lived accounts are a reality. Also the assumption "a constant amount of games played over time" is often not realistic (christmas, other holidays). And "heavyness" of accounts even if a lot of the time is only frustration/bias actually admittedly exists and we're only arguing about how bad it actually is (fast improving people / general population).

On the contrary, I know there are anomalies, I know every rating system will have its quirks, and I've frequently said on both this forum and other places that individual player strength fluctuations due to natural variation will outweigh most of the quirks of any somewhat reasonable rating system.

What I dislike is the notion that we need a process of "I am frustrated and think there is problem with X we need to make modification Z" -> "Ok! Let's immediately change the system and add Z!"

What I prefer is "I am frustrated and think there is problem with X" -> "Ok, let's verify phenomenon X actually occurs. If there is an issue with X, let's identify the extent to which it is a problem and who is affected by it. Once we know who is affected and how they are affected, let's come up with a reasonable mitigation measure that returns us to the expected system behavior."

KGS's rating system is a mathematical model being applied to large dataset, if there are issues they should manifest themselves in data. At least a problem should be demonstrable with an example or case study, if not with a large amount of data from a group. As mentioned up thread by others, there are natural biases that will pop up and any and all of our perceptions and in order to get an honest assessment of what's happening vs. what should be happening, we should be able to come up with objectively measurable criteria we can use to go out and test in order to confirm or disprove our suspicions.

[Mef]

In any case, the graph Mef made could even support my points: between September to November I only managed a win rate of 40% or worse, yet my graph went UP.

May 2011:
61% but the rating dropped.

This is perhaps another misconception about how rating systems should behave -- both of these cases are correct behavior for this system. Most go rating systems lump people into pools (ranks, grades, etc). The result is that you are not typically playing someone with the same rating as you, but instead playing someone who is average for your pool. What this means is that if you are on the either tail of the pool you are not expect to win 50% of your games.

If you have just recently promoted to 1d, you may have a rating of 1.0, but your average opponent will be rated 1.5 -- this means that you are playing your average game under-handicapped by about half a stone. The expected number of games you will win is a bit lower than half (using the EGF numbers, a 50 point rating difference near 1 dan is about a 37-39 % expected win rate...I think KGS is a little wider maybe 33-35% for a half stone difference).

This means that if you have just promoted, and you are winning 40% of your games, you are actually doing a little bit better than the rating system was predicting, which means your rating will go up accordingly (maybe to 1.2d or something like that). If you were to maintain a 50% win rate you would be expected to have your rating move up to 1.5.

On the other side of the pool you have the reverse issue, if you are a 1.9d who is on average playing 1.5d players, you are expected to win 60 to 66% of your even games, because you are playing people the system expects are weaker than you. If you are a 1.9d there's a very real chance you will get paired with that person who is a 1.2d and you are playing a game that under slightly different circumstances would be played at a 1 stone handicap. Maintaining a 60% win-rate against an average 1d is merely doing what is expected and will not on its own cause your rating to rise.

Any system that assigns players a rating (as opposed to just a rank) will suffer from this. This solution to this "problem" is simple - narrow the pools until you are satisfied with the bandwidth. On KGS the easiest way to do it (if you are worried about being on the edge of a band) is to offer a fractional komi.

[hyperpape]

I see that Mef and I were writing to make the same point at the same time. Oh well.

Speaking for myself, the KGS system does not deliver a 50-50 ratio.

This is one of the most unintuitive things about ranks. If you just crossed the 1 kyu threshold (or any rank threshold), you need to win less than 50% of your games to maintain or even improve your rank. Why? Well, everyone between 1 kyu and 1 dan is labeled "1 kyu" but the system thinks that some of them are stronger--they're almost 1 dan--and others are weaker. Take the average, who's a ".5 kyu". When you're paired against that player, you are expected to win less than 50%, because that player is a bit stronger than you. So if you win 50% your rank will increase. If you win somewhat less than 50%, your rank may stay stable. And of course it's the opposite as you approach the threshold from below--you have to win more than 50% of your games.

Note: this point isn't really specific to KGS--it's true of any system that uses dan/kyu ranks but estimates how likely you are to win based on your precise rank (i.e. 1.4325 kyu) and updates your rank accordingly. But it is an argument that EGD style ranks where you see two extra significant digits are psychologically better. It's intuitive that a 2200 who beats a 2250 has just had "good win". But tradition is a hard thing to change.

[Tami]

On the other side of the pool you have the reverse issue, if you are a 1.9d who is on average playing 1.5d players, you are expected to win 60 to 66% of your even games, because you are playing people the system expects are weaker than you. If you are a 1.9d there's a very real chance you will get paired with that person who is a 1.2d and you are playing a game that under slightly different circumstances would be played at a 1 stone handicap. Maintaining a 60% win-rate against an average 1d is merely doing what is expected and will not on its own cause your rating to rise.Any system that assigns players a rating (as opposed to just a rank) will suffer from this. This solution to this "problem" is simple - narrow the pools until you are satisfied with the bandwidth. On KGS the easiest way to do it (if you are worried about being on the edge of a band) is to offer a fractional komi.

Thanks for explaining this in easy-to-understand terms. It seems a bit clearer to me why it can sometimes feel that there is no benefit in winning, and a lot of loss in losing.

I think it would be an improvement if the KGS algorithm would offer the option of modifying komi so that a rated game would carry the same weight for both players.

[RobertJasiek]

On KGS the easiest way to do it (if you are worried about being on the edge of a band) is to offer a fractional komi.

No. The easiest solution is: Do not use any pools! Players do not alter their strengths in pools but alter them continuously. Therefore pools are a bad model for a rating system.

This is perhaps another misconception about how rating systems should behave

Not a misconception but a different preference, see above.

-- both of these cases are correct behavior for this system.

Correct only under the assumption that the system design criteria (such as using pools at all) were any good.

[RobertJasiek]

What I prefer is "I am frustrated and think there is problem with X" -> "Ok, let's verify phenomenon X actually occurs. If there is an issue with X, let's identify the extent to which it is a problem and who is affected by it. Once we know who is affected and how they are affected, let's come up with a reasonable mitigation measure that returns us to the expected system behavior."

I prefer to make changes BEFORE generating any frustration. Specify criteria which a rating system must fulfil for each player. Then design a rating system that will fulfil the criteria for almost all players (98% rather than only 60%).

[averell]

On KGS the easiest way to do it (if you are worried about being on the edge of a band) is to offer a fractional komi.

No. The easiest solution is: Do not use any pools! Players do not alter their strengths in pools but alter them continuously. Therefore pools are a bad model for a rating system.

A "pool" here is just a word he used to describe the range of opponents that are eligible to play an even game with you. If you want to play even games, you need such a set. The only thing you could criticize is how large it should be (at the moment, all people with a rating that gives them the same "rank").

[RobertJasiek]

Also to help players understand how much a win or loss is worth i would like to see an addition in the games tab list where its stated how much that game made your rank shift up or down

I want to see more: I want to see BEFORE agreeing on an opponent 1) which winning percentage the system assumes and 2) how many rating points a win / tie / loss will affect my rating.

[RobertJasiek]

If you want to play even games, you need such a set.

Of course not.

What is needed is 1) the option of the players to agree on playing an even game and 2) the potential opponent's rating number.

[averell]

If you want to play even games, you need such a set.

Of course not.

What is needed is 1) the option of the players to agree on playing an even game and 2) the potential opponent's rating number.

I'll try once more. If you want to play even games with any players that are not exactly as strong as you are, there will be a rating difference. The larger this difference is, the bigger the winning chances for one player. The fractional komi was suggested as an option to get this percentage back to 0.5. The only "pooling" going on, is the default suggestion of 6.5 komi for players which are within one rank, and is not actually a feature of the ranking system.

On the other hand i wholeheartedly support the plea for detailed rank info, and i also like that prediction info idea.

[hyperpape]

Specify criteria which a rating system must fulfil for each player. Then design a rating system that will fulfil the criteria for almost all players (98% rather than only 60%).

Specify criteria which a rating system must fulfil for each player. Then prove that such a system is possible. Then implement it.

You of all people must know that step 2 is essential.

[mitsun]

If a 1.1-dan plays a 1.9-dan, there are at least two reasonable options:

The pairing system can say both are 1-dan, so the game is even (komi 6.5)
The pairing system can say rating difference = 0.8, so the game is uneven (komi 0.5)

I gather KGS uses the first option. I play mostly on IGS, where I believe the second option is implemented. IGS also has a command to predict win/loss probability against any opponent, based on exact ratings and handicap and komi, but the calculation has bugs and probably will never be fixed. Every system has problems

[RobertJasiek]

Specify criteria which a rating system must fulfil for each player. Then prove that such a system is possible. Then implement it.

You of all people must know that step 2 is essential.

Right. Pretty much this procedure is what I'd like to see. Where formal proofs are not found, there must at least be thorough simulations for players with assumed different behaviours of when and whom they play etc.