KGS ranking system

[hibbs]

Also I would have to do some searching but I know wms has mentioned several places that every rating system he has tried with asymmetric ultimately becomes unstable (example: A, B, and C are rated 1k and all play each other and no one else, and on average win 50% of the time against each other....Instead of A, B, and C all staying the same rank, they all either spiral up to 9d or drop to 30k).

Thanks, that is very useful information. If you find these references, I would be quite interested in it.

[speedchase]

nonsense, the rating system should increase in accuracy if you add more data to it (obviously). The rating system should fit to my play schedule, I shouldn't have to play in such a way that gives me a good rating.

You're in luck, it does. Snorri's ideas won't help at all (see Mef's point).

So play 2-7 rated games a week, and you won't feel stuck, your rank will be where it should, and you'll be happy.

These two statements seem strangely at odds.
either way, you seem to be misrepresenting the arguments of the other side. I would be shocked if anyone was complaining about not getting promoted after 4 wins, but I once had an 18 game win streak, and only went up half a stone.

[hibbs]

The only difference is that if you play games slowly, you will not create deceptive streaks. That is, in a long string of games, there should be common times when you win 4 or 5 games in a row, and we humans are incapable of perceiving such sequences as random (sequences judged random by participants in studies are typically biased, while random sequences are judged to be non-random).

So if you play too many games too quickly, you will eventually fool yourself into thinking you've had a jump in rank and KGS is "stuck". In fact, you will not have had a jump, and KGS will be right more often than not. So play 2-7 rated games a week, and you won't feel stuck, your rank will be where it should, and you'll be happy.

If you assume that your current rank is correct, (which would mean a 50% win ratio), then a streak of five won games in a row should indeed be common (the probability for this is 3%, so roughly this should happen once in 30 games). A streak of 10 won games in a row would have a chance of 0.1%, which could still happen once a year if you play a lot.... A streak of 15 won games would be around 0.003 %. In such a case I would assume it is much more likely that the person in question has actually improved. So I do see why people get frustrated if their likely improvement is not reflected by the rating system.

Also, if you play games slowly, you will also create deceptive streaks. The difference is that if you play slowly you are much more likely to be promoted on a deceptive streak than if you play a lot of games. But you are much more likely to be demoted as well...

Last one: There have been repeated reports that people have created new accounts that quickly got a higher rank than the original one. This would be a clear indication that there was a real improvement. On the other hand, I have no idea how often this happens in reality. It could also be that these people become quiet if the new account stabilizes at the same rank as the old one...

[Mef]

Last one: There have been repeated reports that people have created new accounts that quickly got a higher rank than the original one. This would be a clear indication that there was a real improvement. On the other hand, I have no idea how often this happens in reality. It could also be that these people become quiet if the new account stabilizes at the same rank as the old one...

As someone who occasionally looks into/tracks this sort of thing...while there are some cases of this happening, there are also many cases of people who make a new account, play stronger opponents, get a solid rating 2 stones higher....then over the course of the next month see their rating slowly return to what it once was (the value they were complaining the old account was "stuck" at).

Edit later because this is in response to the poster that asked for it:

Wms talked a little bit about weighting systems here

[Mef]

If you want a rating system to be more responsive to improvement, then maybe you'd have to reward streaks disproportionately, like WBaduk does. For example, if someone goes on a winning streak, you might lower the weightings of any games that precede that streak. I don't know if that would destabilize the system.

The points you make are more or less sound, it really comes down to perception and what you want out of a rating system. For instance when you say "destabilize" the rating system, this might be a little ambiguous...It would not necessarily destabilize the system in the way mentioned up thread (where ranks will spiral one direction or another)...but it will make the ranks more fluid, and likely decrease the predictive capacity of the rating system. It's worth noting however that many critics don't consider predictive capacity to be a desirable characteristic of a rating system, they would prefer a descriptive system (that reflects what has already happened, and not necessarily what will happen).

Probably it would not be popular because you can't just reward winning streaks. You have to punish losing streaks as well, which might not be as popular.

This is very true as well...While it has been known to happen, it is much less common to hear people complain their rank is "stuck" when the 8 game losing streak does not cause demotion.

[schawipp]

[...]
Assume that someone has played 20 rated games in that period and won 50% (that is 10, so 10 games lost, 10 won). The persons attends a workshop, reads a book, or for whatever else reason suddendly improves considerably. A promotion to the next higher rank now requires 14 straight wins (which would make it 24 won, 10 lost or 70% win rate, required for promotion).
If someone has played 200 rated games and won 50%, then he would require a win streak of 140 straight wins. (Not in reality, because some of the older games would disappear from the calculation. But he still would need considerably more wins than someone who does not play that often).

A possible solution would be to create a weighting function for both player's kyu, which depends not on time but on the number of games played afterwards. If e. g. the "half-value number" of games is defined as 20 let's have following example:

Player A (7k) wins against Player B (8k)

Two weeks later, player A has played 20 games, player B played 40 games.

For player A this game will be weighted by 50% for player B by 25%.

I would take the kyu values fixed at the time of the game for the rank evaluation. Currently it seems, that my KGS rank increases when the ranks of the players, which I played before, increase. Maybe player B promoted from 8k to 6k within two weeks after the game mentioned above, however at the time of the game he was in fact 8k, so his later improvement should not affect the rank of player A (unless they are playing a new game).

That's my 2ct, OTOH the current ranking system on KGS works also fine for me.

[Mef]

A possible solution would be to create a weighting function for both player's kyu, which depends not on time but on the number of games played afterwards. If e. g. the "half-value number" of games is defined as 20 let's have following example:

As mentioned earlier, this is an asymmetric rating system, which leads to instability in the system as a whole

[hibbs]

It's worth noting however that many critics don't consider predictive capacity to be a desirable characteristic of a rating system, they would prefer a descriptive system (that reflects what has already happened, and not necessarily what will happen).

I think that predictive capacity should be most desirable characteristic of a rating system. That is why I was surprised that the math of the KGS system seems to be compellingly correct, but on the other hand a lot of people seem to not like it.

Wms talked a little bit about weighting systems here

Thanks for that.. I must have missed that statement by WMS. If I had seen this before, I would not have started this thread. I do not doubt that WMS'observation is correct, which means if you want to have good predictive system you have to live with the perceived shortcomings...

[schawipp]

As mentioned earlier, this is an asymmetric rating system, which leads to instability in the system as a whole

Yes, I see the point. A possible workaround would be to use the geometric average of both player's weightings wA, wB:

W = sqrt(wA * wB)

(wA = weighting of player A, wB = weighting of player B, both calculated according to their number of games played)
and to use W as common weighting factor for both players instead of wA, wB. If both players played 20 more games afterwards, W would be 50%. If one player is faster / slower than the other, the decay of W would speed up / slow down accordingly.

Edit: I see that this has been already intensively discussed, so never mind... :

[hyperpape]

What happens is that bunches of games played closely together at higher than your average play rate have a tendency to stabilize the rating if the win/loss rate is near 50%. So later, it takes a lot of games to move that number unless they've mostly aged out. Is this a problem? From one point of view, it isn't. If a player decides to invest a lot of games in a short period and have that anchor the rating for a few months, then so be it. That's when the player played the most games, and that's where most of the data is. Fine. So where's the problem? I think the thing that nags some players is that when they think they have improved, they may resent the idea that they might have to spend months (or months worth of games) to prove it. By playing rated games at a consistent rate, you accept the moving average for what it is. If you think you have improved, you have the option of waiting for adjustment or you can invest more. That is why I suggested 1 rated game a week. Then when you think you've improved, you can ramp it up to, say 1-2 rated games a day and have things move faster if you are right. Now if one is going to suggest it is bad to deny "the system" rated games, that's another discussion.

I agree with one part of this. If you play a ton of games at one point when you have a chance, at a rate that you can't commit to later, you can become temporarily "stuck". So you might not want to do that. It's not that playing fewer games is better (which is how I misread your original post), but that there can be a downside to binging on games.

[hyperpape]

nonsense, the rating system should increase in accuracy if you add more data to it (obviously). The rating system should fit to my play schedule, I shouldn't have to play in such a way that gives me a good rating.

You're in luck, it does. Snorri's ideas won't help at all (see Mef's point).

So play 2-7 rated games a week, and you won't feel stuck, your rank will be where it should, and you'll be happy.

These two statements seem strangely at odds.
either way, you seem to be misrepresenting the arguments of the other side. I would be shocked if anyone was complaining about not getting promoted after 4 wins, but I once had an 18 game win streak, and only went up half a stone.

My comment about feeling stuck was not about the ranking system, but about your subjective perceptions. So there is no contradiction.

As for your 18 game streak, it would be interesting to see the records, to see what held you back. An 18 game streak without a rating gain would have to be quite unusual circumstances.

[Mike Novack]

I suspect the subjective problem is unfamiliarity with statistics and the "scientific method".

a) Suppose in a prior period you won 50% of N games. You attend a workshop, study a book, etc. and presumably have improved. You now play a sequence of M games winning them all. Should your rank be upped to reflect that? (based upon M)

b) a) Suppose in a prior period you won 50% of N games ............................ .........................................You now play a sequence of M games winning them all. Should your rank be upped to reflect that? (based upon M)

"b" is the so called "null hypothesis" that the outcome was purely by random chance. Notice that if your rank is upped in case "b" that was the wrong thing to do.

The point I am trying to make here is that the lay person tends to grossly underestimate the size of M required to have it be unlikely that the observed outcome was pure chance. For example, suppose this class is attended by 32 people. More likely than not one of them would come home and win their next five games. That class really helped, didn't it. Nah, it was a class on baking.

[speedchase]

@Mike any length of wins COULD be explained by probability, but that doesn't mean they should. There are much more reasonable explinations.
edit: removed a mistake.

[ez4u]

... I would be shocked if anyone was complaining about not getting promoted after 4 wins, but I once had an 18 game win streak, and only went up half a stone.

My comment about feeling stuck was not about the ranking system, but about your subjective perceptions. So there is no contradiction.

As for your 18 game streak, it would be interesting to see the records, to see what held you back. An 18 game streak without a rating gain would have to be quite unusual circumstances.

Download the CSV file for speedchase from the bottom of the KGS Analytics page and look at August 2011. It would be interesting to have some informed commentary on his promotion record throughout the summer and into the fall.

[speedchase]

Download the CSV file for speedchase from the bottom of the KGS Analytics page and look at August 2011. It would be interesting to have some informed commentary on his promotion record throughout the summer and into the fall.

Thanks, I was having trouble finding it. If I am reading the file correctly, it says I was 11k the entire time.