KGS ranking revisited

[HermanHiddema]

Many players seem to have this notion of their "real strength", which is usually how strong they would be if they removed all the parts of their play they don't like. And it is nonsense.

While we're coming at this from similar points of view, I differ with this point. It is not entirely meaningless. It's just impossible to systematically test (this may be my favorite philosophical distinction, fwiw).

Sometimes we see a player who is almost perfect in most regards, but also has a glaring flaw. We recognize that he's better than his performance in the sense that if he fixes that flaw he will be exceptional. And it does matter in one sense. Suppose A is exceptionally talented, but is also an alcoholic, while B is less talented, but is completely disciplined. A wins when he's been sober, but loses more often because of his personal problems. But if A ever gets his act together, B will never match him, no matter how hard he works.

Less extreme examples are common, and while they're prone to bias and wishful thinking, that doesn't mean they're not sometimes real. They're just not the business of a ratings system.

Can't say I really think much of the concept of "unmeasurable but real". And anyway, the example you give is easily measurable. A player's performance could be approximated, statistically, as a distribution. In the example, player A and B have the same mean performance, but player A has a higher standard deviation.

But, until a player fixes that "flaw", I do not think that he is stronger. Flaws are part of the deal. You could say he has a higher peak performance, but that is it.

[hyperpape]

Can't say I really think much of the concept of "unmeasurable but real".

Well, a flip answer would be to introduce you to my friend Gödel...A less flip answer would be that there are obvious cases of meaningful statements that can't be tested in any practical way. There is a fact of the matter of how many times the letter 't' has been printed in books since time began. There is obviously no practical way to test this.

And anyway, the example you give is easily measurable. A player's performance could be approximated, statistically, as a distribution. In the example, player A and B have the same mean performance, but player A has a higher standard deviation.

Is this right? Couldn't he have a smaller standard deviation but a larger range--perhaps he is hung-over 95% of the time and plays terribly all of that time?

But, until a player fixes that "flaw", I do not think that he is stronger. Flaws are part of the deal. You could say he has a higher peak performance, but that is it.

I do not know what to make of the semantics of "stronger". I do agree a rating system should not try to gauge these facts. Nonetheless, they are important dimensions of evaluating a player. That is the claim that I would like to insist upon.

Edit: removed a dangling [/quote] tag.

[jts]

Can't say I really think much of the concept of "unmeasurable but real".

Well, a flip answer would be to introduce you to my friend Gödel...A less flip answer would be that there are obvious cases of meaningful statements that can't be tested in any practical way. There is a fact of the matter of how many times the letter 't' has been printed in books since time began. There is obviously no practical way to test this.

Would you also say that there is just a counter-factual fact about the world, to wit, the number of times 't' would have been printed in books since time began if Cleopatra had had a snub nose? Being a realist about unmeasurable facts is one thing, but being a realist about unmeasurable counterfactuals is quite another.

But, until a player fixes that "flaw", I do not think that he is stronger. Flaws are part of the deal. You could say he has a higher peak performance, but that is it.

I do not know what to make of the semantics of "stronger". I do agree a rating system should not try to gauge these facts. Nonetheless, they are important dimensions of evaluating a player. That is the claim that I would like to insist upon.

Yes, this may make sense.
"He has a thoughtless fuseki but he finds incredible tesuji in the middlegame."
"Even when he loses all the fighting, he can usually count on making up twenty points in the endgame."
"He memorized all of Dosaku's games and always has beautiful shape, but I wish he would stop to think more at critical junctures; he often makes careless blunders."
"Somehow he has an encyclopedic knowledge of joseki, but when he plays he's usually drunk, or hung over, or both."

These are all ways to qualitatively describe someone's playing strength, but I think what Herman is saying that whenever he hears someone say "KGS can't measure my real strength because _______" they somehow think that a problem like erratic play or poor use of time or alcoholism can be fixed by an accurate rating system. It would be just as sensible (i.e., not very) to ask for a rating system, or a concept of "real strength", that abstracted away from weak reading, or sloppy endgame, or poor joseki choice.

[hyperpape]

Can't say I really think much of the concept of "unmeasurable but real".

Well, a flip answer would be to introduce you to my friend Gödel...A less flip answer would be that there are obvious cases of meaningful statements that can't be tested in any practical way. There is a fact of the matter of how many times the letter 't' has been printed in books since time began. There is obviously no practical way to test this.

Would you also say that there is just a counter-factual fact about the world, to wit, the number of times 't' would have been printed in books since time began if Cleopatra had had a snub nose? Being a realist about unmeasurable facts is one thing, but being a realist about unmeasurable counterfactuals is quite another.

Ehhh...I'm inclined to hem and haw. I think your example is a clear one that shouldn't have an answer, but I don't know whether "unmeasurable counterfactual" is really the category that explains why. I don't really have a theory of any of this stuff.

I think what Herman is saying that whenever he hears someone say "KGS can't measure my real strength because _______" they somehow think that a problem like erratic play or poor use of time or alcoholism can be fixed by an accurate rating system. It would be just as sensible (i.e., not very) to ask for a rating system, or a concept of "real strength", that abstracted away from weak reading, or sloppy endgame, or poor joseki choice.

I think this is common ground to the three of us. My point was really about meaninglessness of those questions. It may have been a distraction, but my thought was that by overreaching, and saying it's meaningless, you may confuse the issue. The real point is that meaningless or not, these questions aren't the proper ones for a rating system.

[Mef]

I think that hits the nail on the head.

The KGS may be accurate for most players most of the time, but it seems to be based on the assumption that nobody ever improves. Once you have a stable rank, then it becomes extremely hard to change it, no matter how much you win or lose. And, I kind of agree with Robert Jasiek here, it is much easier to play worse than your mark than up to it because on more than one occasion I have been close to a promotion, lost a crucial game and then gone on to lose a string on games out of sheer frustration. I`m sure that experience is not unique. If only it wasn`t quite so like climbing a greasy pole, maybe not so many players would go on tilt so often.

The latest adjustment, the downward one that prompted this thread, came as a nasty surprise - I had been nursing my main account, the heavy one, toward 1d by steadfastly resisting tilty emotions whenever I did lose, and the adjustment undid all that. It also brought my 1d account temporarily back to 1k.

And, yes, rank and ratings graph are important to me. I have been putting effort into improving my go, and I was using these things to measure my progress. Maybe I have little talent for the game and I am only improving in small steps, but I still like to see my graph go upwards over the passing months.

For sure, I totally get it that the system is not intended for providing feedback on players` progress, but only for making a roughly 50-50 win/lose balance. Could it not be though that the 50-50 balance is merely an illusion of a mirage? If, in fact, there are many, many players of different strengths crammed into a small ratings band because of heaviness, then might not their mutual scores tend to even out over time, thereby giving the false impression of accuracy? (Strong 3k beats weak 3k, but weak 3k wins against weak 1k, who then narrowly beats strong 2k, who beats strong 3k, who goes on tilt and loses to weak 3k).

Still, if it`s never going to change, then that`s just too bad. At least it's still fun to play free games and watch broadcasts.

When I get the chance, I like to clear up misconceptions in threads like these. In an earlier thread I've already addressed some of Robert's concerns of heaviness.

Cliff notes:

if you actually look at the data you would see that even as a 4d playing more than 3000 games per year there was not any "heaviness" in his rank. It was still capable of moving 1/3 to 1/2 a stone in less than a week, provided he continued playing games at a similar rate as he had been. The lack of movement was not related to any heaviness in his account, but actually due to the rarity of any streaks where he had a significant deviation from a 45-55% win rate).

I don't have the time to do that sort of thing again (I figure that if someone playing 10 games/day for a year in one of the most stable rank ranges on KGS has no issue moving their rank when it is appropriate, the majority of us should be ok). What I have done though is attach an image, assuming that your Universal go server handle is the one you use on KGS:

Rank-Case-Study-2.JPG (52.12 KiB) Viewed 10672 times

It shows your rank graph as well as your winning percentage in rated games and total games played. It is broken down month by month with cumulative totals at the bottom. For reference, I added a horizontal line at 1.5k. I used August 1, 2011 as the start of the dataset and April 30, 2012 as the end. I'll let the L19 community decide if they feel the KGS server is doing a good job of handling these results or if there is an issue with "heaviness", or a bias of losses being weight more than wins.


Regarding the idea of rank "crunching" where you have several ranks in one band. This type of phenomenon is actually something the study linked yoyoma posted would have detected (unless all rank systems tested suffer from it). If you have KGS grouping people together they shouldn't be, then a superior system would be able to identify these mismatches and easily predict the winner in an even game (a full 1 stone difference should represent something like a 75-80% chance of winning for the stronger player in an even game). In the aggregate, the superior system would be able to correctly predict a much larger percentage of even game winners than KGS.

Another way to check for this "crunching" would be to look at winning percentages in handicap games. If players within a band actually represent a several stone span, then when you play someone outside the band there is a good chance there is a strength difference that is not being compensated for by the handicap. The result is that the player with black should have a severe disadvantage in these games (note: handicap games on KGS are already a half stone under-handicapped by default, but now they would be even more so.). If 3k was really "2k-3k-4k" and 4k was really "4k-5k-6k", and 5k was really "6k-7k-8k", it would mean your average 2 stone game should really be a 4 stone game, and black would be in trouble. In the end, you'd expect something like less than 20% of handicap games being won by black.

[RobertJasiek]

In an earlier thread I've already addressed some of Robert's concerns of heaviness.

On which I have aleady answered:
http://www.lifein19x19.com/forum/viewto ... 558#p85558

if you actually look at the data you would see that even as a 4d playing more than 3000 games per year there was not any "heaviness" in his rank. It was still capable of moving 1/3 to 1/2 a stone in less than a week, provided he continued playing games at a similar rate as he had been. The lack of movement was not related to any heaviness in his account, but actually due to the rarity of any streaks where he had a significant deviation from a 45-55% win rate).

Which player does this refer to, when where those 1/3 to 1/2 jumps in less than a week, and are you sure that they were not manual KGS rating shifts?

Rank-Case-Study-2.JPG [...]
if there is an issue with "heaviness"

That KGS player is hardly interesting as a study case for heaviness because she played rather few games in comparison to those players really suffering from heaviness.

[averell]

[...] (a full 1 stone difference should represent something like a 75-80% chance of winning for the stronger player in an even game). [...]

Is that an assumption, or does that actually hold statistically? Because for example in EGF ranks 1k-1d is still a 40% win chance, and to get below 25 you have to be 6d. Now it could be the KGS ranks are further apart, but i seriously doubt it.

[...] In the end, you'd expect something like less than 20% of handicap games being won by black.

Same thing here basically.

Another thing that keeps happening is people making new accounts, because it's apparently easy to get a solidly ranked account up to 4 stones higher than your "actual strength" with just a lucky (time-)win and then lose to make it solid. Quite a few of "fake" 8/9ds keep cropping up.

Overall i would agree that the KGS rating system does a "good enough" job however, because individual playing strength will vary over 2 stones easily for 80% of the KGS population from one day to the next, and the current very inaccurate average prediction is the best you can do based on game results only.

[stalkor]

I guess like most things "haters gotta hate"

I've been following this thread and it got from a discussion right into kicking something as much and as hard as you can but the thing i'm missing is probably the most important thing, a solution!

Instead of repeating what has been stated in several other threads, come up with ideas and mathematical backup for it.

One of my ideas (of which i dont have any facts to back up) is for example to shorten the history taken into account from 6 to 5 or 4 months, this can possibly result in a less "rigid" rank.

Also to help players understand how much a win or loss is worth i would like to see an addition in the games tab list where its stated how much that game made your rank shift up or down (if im not mistaken a rank is a number so it could be done).

For example +0.20 or -0.33

This will create the opportunity to see the weight of each game, not the increment after a day.

[Mef]

In an earlier thread I've already addressed some of Robert's concerns of heaviness.

On which I have aleady answered:
http://www.lifein19x19.com/forum/viewto ... 558#p85558

Yes, as I understand it, your rank only gets heavy when no one else is watching.

That KGS player is hardly interesting as a study case for heaviness because she played rather few games in comparison to those players really suffering from heaviness.

I will save myself a thousand words.

Note: Cumulative totals exclude the months of May, June, and July when games were played as a 4d

Rank-Case-Study-3.JPG (50.12 KiB) Viewed 10596 times

[Mef]

Is that an assumption, or does that actually hold statistically? Because for example in EGF ranks 1k-1d is still a 40% win chance, and to get below 25 you have to be 6d. Now it could be the KGS ranks are further apart, but i seriously doubt it.

Another thing that keeps happening is people making new accounts, because it's apparently easy to get a solidly ranked account up to 4 stones higher than your "actual strength" with just a lucky (time-)win and then lose to make it solid. Quite a few of "fake" 8/9ds keep cropping up.

Overall i would agree that the KGS rating system does a "good enough" job however, because individual playing strength will vary over 2 stones easily for 80% of the KGS population from one day to the next, and the current very inaccurate average prediction is the best you can do based on game results only.

It's been a long time since I've looked into calculating the exact values (or even known exactly what constants KGS uses). From my interpretation of the EGF ratings page a 1d vs. a 1k in an even game is between 70% and 75% expected win for the 1d (depending one what you call 1d...Table II lists GoR 2000 as 27.8% chance of beating a 2100 ). At any rate, I'm not too worried about quibbling over exact percentages for each system, I imagine it would depend on how well you wish your system to scale for handicaps and a variety of other factors.

The point I was trying to make is that if there are the system anomalies that some suspect, there should be some easily measurable side effects we could predict and identify. I guess at the end of the day, when possible I'd much prefer hard facts to sneaking suspicions and speculation...but then again, I am from Missouri...

[speedchase]

Note: Cumulative totals exclude the months of May, June, and July when games were played as a 4d

Rank-Case-Study-3.JPG

This is very good evidence of how overly heavy it gets.

[averell]

Code: Select all

Statistics of Even Games - all players

 SeMin  SeMax    AdR     Nw     Ng     Pw    ASe   Pw-ASe  SE(AdR)   ARGD     dPw  
 -----  -----  ------  -----  -----  -----  -----  ------  -------  ------  ------
   0.0    2.5   731.9   1357  27306    5.0    1.0     3.9      1.3   -36.3     3.7 
   2.5    5.0   408.8   2258  17924   12.6    3.7     8.9      4.7   -56.0     7.9 
   5.0    7.5   334.0   3087  17274   17.9    6.2    11.6      6.1   -63.9    11.7 
   7.5   10.0   293.9   3966  18319   21.6    8.7    12.9      8.0   -64.9    13.6 
  10.0   12.5   260.3   4752  18248   26.0   11.3    14.8      9.7   -66.2    16.3 
  12.5   15.0   230.0   5142  18190   28.3   13.7    14.5     10.7   -66.8    17.6 
  15.0   17.5   208.4   6042  19263   31.4   16.2    15.1     12.5   -64.2    18.9 
  17.5   20.0   189.0   6554  20207   32.4   18.7    13.7     14.8   -61.6    17.6 
  20.0   22.5   168.2   7098  20378   34.8   21.3    13.6     16.1   -58.7    18.8 
  22.5   25.0   151.7   7708  21138   36.5   23.8    12.7     18.4   -56.0    18.1 
  25.0   27.5   138.8   8428  22659   37.2   26.3    10.9     21.6   -52.4    15.5 
  27.5   30.0   117.6   8219  20817   39.5   28.7    10.7     22.1   -54.2    17.4 
  30.0   32.5   102.4   8795  21870   40.2   31.3     8.9     24.8   -48.5    15.4 
  32.5   35.0    89.2   9698  23469   41.3   33.8     7.5     27.8   -44.5    13.6 
  35.0   37.5    76.8  10582  24572   43.1   36.3     6.8     30.8   -40.8    12.3 
  37.5   40.0    61.8   9978  23050   43.3   38.7     4.6     32.7   -42.8    10.6 
  40.0   42.5    46.3   9933  21930   45.3   41.3     4.0     35.1   -40.2    10.2 
  42.5   45.0    33.0  10222  22000   46.5   43.8     2.7     37.6   -39.0     8.8 
  45.0   47.5    19.7  10566  22237   47.5   46.3     1.3     40.7   -36.0     6.8 
  47.5   50.0     6.6  10908  22585   48.3   48.7    -0.5     43.8   -33.5     4.5 

The actual data, from that same site (2000-Jan-01 to today)
In this table, AdR is average rating difference, and Pw is percentage of wins.
While the actual percentages don't matter to me either, the fact is that a single rank or 100 elo points doesn't make all that much difference w.r.t. winning chances. Of course this is EGD tournament data, but i doubt KGS blitz ranks fare any better.

The point I was trying to make is that if there are the system anomalies that some suspect, there should be some easily measurable side effects we could predict and identify. I guess at the end of the day, when possible I'd much prefer hard facts to sneaking suspicions and speculation...but then again, I am from Missouri...

You act like there are no "system anomalies". Even if a system like KGS is the best available (or reasonably possible) doesn't mean that it's not crap in a lot of ways. For example fake short-lived accounts are a reality. Also the assumption "a constant amount of games played over time" is often not realistic (christmas, other holidays). And "heavyness" of accounts even if a lot of the time is only frustration/bias actually admittedly exists and we're only arguing about how bad it actually is (fast improving people / general population).

That said, the only thing i would like to see changed with the KGS ranks is to make the data more open (actual accessible numbers), because people might just come up with a better system. Another hypotethical weakness is rank anchors, which have to be kept secret, but i don't see why they would be that hard to find. I might try this as a pet project when i get a few days of free time.

[jts]

Note: Cumulative totals exclude the months of May, June, and July when games were played as a 4d

Rank-Case-Study-3.JPG

This is very good evidence of how overly heavy it gets.

What are you talking about? What percentage of games you won in even games against 4dans doesn't tell us too much about whether you should be promoted to 6dan several months later. Maybe you have something else in mind, but you need to be more specific rather than sniping at Mef, who did an excellent job on his analysis of supposedly "heavy" rankings.

[lemmata]

Irrelevant musing: Given how long this discussion on ranks is, I wonder if server popularity could permanently be increased by radical rank inflation.

Some on-topic musing: Does it really matter how close the KGS ranking system is to someone's ideal? If you can maintain, say, an 80% winning rate against people of equal rank over a month or two, you will rank up. Sure that might not be ideal, but it just means that the system is conservative in rejecting the null hypothesis that the parameter governing a series of random trials has not changed.

More on-topic musing: In the grand scheme of things, rank accuracy is not that important in comparison to other issues. Consider an extreme scenario in which the server designer only cares about rank accuracy. The designer may force everyone to play ONLY randomized automatch games without the option of choosing a rank range (the designer will choose the "optimal" range). After all, we can't have pools of people who only play each other and skew the system (national rooms, players who only play others +/- 1 rank). Freedom to choose is quite valuable to go players, but that freedom may adversely affect the accuracy of a server's ranking system. Designing a system that is robust while allowing those freedoms may result in a system that unbiased (accurate on average), but is slower to recognize changes (but eventually does).

KGS-specific musing: I wonder if wms would get fewer complaints if he opted to display that a player is 9.892 kyu rather than 10 kyu. A player's ranking actually moves all the time. We just don't see it because the KGS displays integer-valued ranks. We can look at the rank graph, but that is not quite as visceral as seeing an actual number.

Meta-thread musing: This discussion really seems to be more about what would be an ideal ranking system. The KGS-specific issues seem more like a distraction to the discussion the participants really want to have. Perhaps a new, more focused thread is in order?

Zen musing: A Tygem 4 dan once told me that he was 4 gup/kyu according to tests at the Hanguk Giwon (Korean Baduk Association). I don't think he complained too much about being under-ranked according to the Hanguk Giwon. Unless you are a pro, these numbers are just for fun. Enjoy the game.

[Tami]

Mef, I resent being used for that kind of analysis. Couldn`t you at least have chosen some other player and WITHHELD their name? Doing what you did has a definite flavour of ad hominem about it, as did another poster`s allusion to "illusory superiority".

I have not been claiming MY rank was wrong, rather that I would only like to see the system made more fluid - in both directions, up and down. I think 1k is currently about right, although I'd rather be labelled a 2k temporarily or a 1d temporarily than told that I never change.

In any case, the graph Mef made could even support my points: between September to November I only managed a win rate of 40% or worse, yet my graph went UP. I won 57% of my games since March and my line is back where it started, thanks to the recent adjustment. My rank never changes - even in periods when I make the mistake of playing a lot of games while upset or distracted.

I am in favour of a system that demotes you when you are doing badly, and promotes you when you are doing well, instead of using many months worth of data to keep you the same. I`m also very much in favour of not making large adjustments at seemingly random times. It would be easier for us lay people to understand.

I would honestly rather be demoted if I lost the bulk of my games in, say, a one month period, if it meant that I could be promoted as easily for winning the bulk of my games in a similar period. Using a long history only makes it hard to understand why your graph and rating behave as they do. If you ARE improving quickly, then your history only holds you back.

Stability is not such a great thing, anyway: if a player is off-form, then allowing their rank to drop down would give them the chance to get back on-form by having easier opponents to play against. Again, if they were on hot form, then a promotion would give them tougher partners, and result in their form cooling off. That is, a fluid system would absorb temporary changes better than a rigid one, like a martial artist flowing with a punch. In contrast, with KGS`s largely unchanging ranks, you get players winning or losing a lot without promoting or demoting, and even if their win-loss scores even out to 50-50 over a 300-year period, the impression they usually get is that the system is just heavy.

Anyway, just so that it's clear: I have never claimed that my rank, personally speaking, was inaccurate; only that I would like to see a system that was more responsive to form, and assigned ranks in a more fluid way. That does not mean going to the opposite extreme, only that it would desirable to have a system that did not feel quite so like swimming through concrete.