Can't say I really think much of the concept of "unmeasurable but real". And anyway, the example you give is easily measurable. A player's performance could be approximated, statistically, as a distribution. In the example, player A and B have the same mean performance, but player A has a higher standard deviation.

But, until a player fixes that "flaw", I do not think that he is stronger. Flaws are part of the deal. You could say he has a higher peak performance, but that is it.

Well, a flip answer would be to introduce you to my friend Gödel...A less flip answer would be that there are obvious cases of meaningful statements that can't be tested in any practical way. There is a fact of the matter of how many times the letter 't' has been printed in books since time began. There is obviously no practical way to test this.

Is this right? Couldn't he have a smaller standard deviation but a larger range--perhaps he is hung-over 95% of the time and plays terribly all of that time?

I do not know what to make of the semantics of "stronger". I do agree a rating system should not try to gauge these facts. Nonetheless, they are important dimensions of evaluating a player. That is the claim that I would like to insist upon.

Edit: removed a dangling [/quote] tag.

_________________
Occupy Babel!

Would you also say that there is just a counter-factual fact about the world, to wit, the number of times 't' would have been printed in books since time began if Cleopatra had had a snub nose? Being a realist about unmeasurable facts is one thing, but being a realist about unmeasurable counterfactuals is quite another.



Yes, this may make sense.
"He has a thoughtless fuseki but he finds incredible tesuji in the middlegame."
"Even when he loses all the fighting, he can usually count on making up twenty points in the endgame."
"He memorized all of Dosaku's games and always has beautiful shape, but I wish he would stop to think more at critical junctures; he often makes careless blunders."
"Somehow he has an encyclopedic knowledge of joseki, but when he plays he's usually drunk, or hung over, or both."

These are all ways to qualitatively describe someone's playing strength, but I think what Herman is saying that whenever he hears someone say "KGS can't measure my real strength because _______" they somehow think that a problem like erratic play or poor use of time or alcoholism can be fixed by an accurate rating system. It would be just as sensible (i.e., not very) to ask for a rating system, or a concept of "real strength", that abstracted away from weak reading, or sloppy endgame, or poor joseki choice.

Ehhh...I'm inclined to hem and haw. I think your example is a clear one that shouldn't have an answer, but I don't know whether "unmeasurable counterfactual" is really the category that explains why. I don't really have a theory of any of this stuff.

I think this is common ground to the three of us. My point was really about meaninglessness of those questions. It may have been a distraction, but my thought was that by overreaching, and saying it's meaningless, you may confuse the issue. The real point is that meaningless or not, these questions aren't the proper ones for a rating system.

_________________
Occupy Babel!

When I get the chance, I like to clear up misconceptions in threads like these. In an earlier thread I've already addressed some of Robert's concerns of heaviness.

Cliff notes:

I don't have the time to do that sort of thing again (I figure that if someone playing 10 games/day for a year in one of the most stable rank ranges on KGS has no issue moving their rank when it is appropriate, the majority of us should be ok). What I have done though is attach an image, assuming that your Universal go server handle is the one you use on KGS:



It shows your rank graph as well as your winning percentage in rated games and total games played. It is broken down month by month with cumulative totals at the bottom. For reference, I added a horizontal line at 1.5k. I used August 1, 2011 as the start of the dataset and April 30, 2012 as the end. I'll let the L19 community decide if they feel the KGS server is doing a good job of handling these results or if there is an issue with "heaviness", or a bias of losses being weight more than wins.


Regarding the idea of rank "crunching" where you have several ranks in one band. This type of phenomenon is actually something the study linked yoyoma posted would have detected (unless all rank systems tested suffer from it). If you have KGS grouping people together they shouldn't be, then a superior system would be able to identify these mismatches and easily predict the winner in an even game (a full 1 stone difference should represent something like a 75-80% chance of winning for the stronger player in an even game). In the aggregate, the superior system would be able to correctly predict a much larger percentage of even game winners than KGS.

Another way to check for this "crunching" would be to look at winning percentages in handicap games. If players within a band actually represent a several stone span, then when you play someone outside the band there is a good chance there is a strength difference that is not being compensated for by the handicap. The result is that the player with black should have a severe disadvantage in these games (note: handicap games on KGS are already a half stone under-handicapped by default, but now they would be even more so.). If 3k was really "2k-3k-4k" and 4k was really "4k-5k-6k", and 5k was really "6k-7k-8k", it would mean your average 2 stone game should really be a 4 stone game, and black would be in trouble. In the end, you'd expect something like less than 20% of handicap games being won by black.

On which I have aleady answered:
viewtopic.php?p=85558#p85558



Which player does this refer to, when where those 1/3 to 1/2 jumps in less than a week, and are you sure that they were not manual KGS rating shifts?



That KGS player is hardly interesting as a study case for heaviness because she played rather few games in comparison to those players really suffering from heaviness.

Is that an assumption, or does that actually hold statistically? Because for example in EGF ranks 1k-1d is still a 40% win chance, and to get below 25 you have to be 6d. Now it could be the KGS ranks are further apart, but i seriously doubt it.

Same thing here basically.

Another thing that keeps happening is people making new accounts, because it's apparently easy to get a solidly ranked account up to 4 stones higher than your "actual strength" with just a lucky (time-)win and then lose to make it solid. Quite a few of "fake" 8/9ds keep cropping up.

Overall i would agree that the KGS rating system does a "good enough" job however, because individual playing strength will vary over 2 stones easily for 80% of the KGS population from one day to the next, and the current very inaccurate average prediction is the best you can do based on game results only.

I guess like most things "haters gotta hate"

I've been following this thread and it got from a discussion right into kicking something as much and as hard as you can but the thing i'm missing is probably the most important thing, a solution!

Instead of repeating what has been stated in several other threads, come up with ideas and mathematical backup for it.

One of my ideas (of which i dont have any facts to back up) is for example to shorten the history taken into account from 6 to 5 or 4 months, this can possibly result in a less "rigid" rank.

Also to help players understand how much a win or loss is worth i would like to see an addition in the games tab list where its stated how much that game made your rank shift up or down (if im not mistaken a rank is a number so it could be done).

For example +0.20 or -0.33

This will create the opportunity to see the weight of each game, not the increment after a day.

_________________
admin of the ASR league and KGS admin

Yes, as I understand it, your rank only gets heavy when no one else is watching.



I will save myself a thousand words.

It's been a long time since I've looked into calculating the exact values (or even known exactly what constants KGS uses). From my interpretation of the EGF ratings page a 1d vs. a 1k in an even game is between 70% and 75% expected win for the 1d (depending one what you call 1d...Table II lists GoR 2000 as 27.8% chance of beating a 2100 ). At any rate, I'm not too worried about quibbling over exact percentages for each system, I imagine it would depend on how well you wish your system to scale for handicaps and a variety of other factors.

The point I was trying to make is that if there are the system anomalies that some suspect, there should be some easily measurable side effects we could predict and identify. I guess at the end of the day, when possible I'd much prefer hard facts to sneaking suspicions and speculation...but then again, I am from Missouri...

This is very good evidence of how overly heavy it gets.

The actual data, from that same site (2000-Jan-01 to today)
In this table, AdR is average rating difference, and Pw is percentage of wins.
While the actual percentages don't matter to me either, the fact is that a single rank or 100 elo points doesn't make all that much difference w.r.t. winning chances. Of course this is EGD tournament data, but i doubt KGS blitz ranks fare any better.


You act like there are no "system anomalies". Even if a system like KGS is the best available (or reasonably possible) doesn't mean that it's not crap in a lot of ways. For example fake short-lived accounts are a reality. Also the assumption "a constant amount of games played over time" is often not realistic (christmas, other holidays). And "heavyness" of accounts even if a lot of the time is only frustration/bias actually admittedly exists and we're only arguing about how bad it actually is (fast improving people / general population).

That said, the only thing i would like to see changed with the KGS ranks is to make the data more open (actual accessible numbers), because people might just come up with a better system. Another hypotethical weakness is rank anchors, which have to be kept secret, but i don't see why they would be that hard to find. I might try this as a pet project when i get a few days of free time.

What are you talking about? What percentage of games you won in even games against 4dans doesn't tell us too much about whether you should be promoted to 6dan several months later. Maybe you have something else in mind, but you need to be more specific rather than sniping at Mef, who did an excellent job on his analysis of supposedly "heavy" rankings.

Irrelevant musing: Given how long this discussion on ranks is, I wonder if server popularity could permanently be increased by radical rank inflation.

Some on-topic musing: Does it really matter how close the KGS ranking system is to someone's ideal? If you can maintain, say, an 80% winning rate against people of equal rank over a month or two, you will rank up. Sure that might not be ideal, but it just means that the system is conservative in rejecting the null hypothesis that the parameter governing a series of random trials has not changed.

More on-topic musing: In the grand scheme of things, rank accuracy is not that important in comparison to other issues. Consider an extreme scenario in which the server designer only cares about rank accuracy. The designer may force everyone to play ONLY randomized automatch games without the option of choosing a rank range (the designer will choose the "optimal" range). After all, we can't have pools of people who only play each other and skew the system (national rooms, players who only play others +/- 1 rank). Freedom to choose is quite valuable to go players, but that freedom may adversely affect the accuracy of a server's ranking system. Designing a system that is robust while allowing those freedoms may result in a system that unbiased (accurate on average), but is slower to recognize changes (but eventually does).

KGS-specific musing: I wonder if wms would get fewer complaints if he opted to display that a player is 9.892 kyu rather than 10 kyu. A player's ranking actually moves all the time. We just don't see it because the KGS displays integer-valued ranks. We can look at the rank graph, but that is not quite as visceral as seeing an actual number.

Meta-thread musing: This discussion really seems to be more about what would be an ideal ranking system. The KGS-specific issues seem more like a distraction to the discussion the participants really want to have. Perhaps a new, more focused thread is in order?

Zen musing: A Tygem 4 dan once told me that he was 4 gup/kyu according to tests at the Hanguk Giwon (Korean Baduk Association). I don't think he complained too much about being under-ranked according to the Hanguk Giwon. Unless you are a pro, these numbers are just for fun. Enjoy the game.

Mef, I resent being used for that kind of analysis. Couldn`t you at least have chosen some other player and WITHHELD their name? Doing what you did has a definite flavour of ad hominem about it, as did another poster`s allusion to "illusory superiority".

I have not been claiming MY rank was wrong, rather that I would only like to see the system made more fluid - in both directions, up and down. I think 1k is currently about right, although I'd rather be labelled a 2k temporarily or a 1d temporarily than told that I never change.

In any case, the graph Mef made could even support my points: between September to November I only managed a win rate of 40% or worse, yet my graph went UP. I won 57% of my games since March and my line is back where it started, thanks to the recent adjustment. My rank never changes - even in periods when I make the mistake of playing a lot of games while upset or distracted.

I am in favour of a system that demotes you when you are doing badly, and promotes you when you are doing well, instead of using many months worth of data to keep you the same. I`m also very much in favour of not making large adjustments at seemingly random times. It would be easier for us lay people to understand.

I would honestly rather be demoted if I lost the bulk of my games in, say, a one month period, if it meant that I could be promoted as easily for winning the bulk of my games in a similar period. Using a long history only makes it hard to understand why your graph and rating behave as they do. If you ARE improving quickly, then your history only holds you back.

Stability is not such a great thing, anyway: if a player is off-form, then allowing their rank to drop down would give them the chance to get back on-form by having easier opponents to play against. Again, if they were on hot form, then a promotion would give them tougher partners, and result in their form cooling off. That is, a fluid system would absorb temporary changes better than a rigid one, like a martial artist flowing with a punch. In contrast, with KGS`s largely unchanging ranks, you get players winning or losing a lot without promoting or demoting, and even if their win-loss scores even out to 50-50 over a 300-year period, the impression they usually get is that the system is just heavy.

Anyway, just so that it's clear: I have never claimed that my rank, personally speaking, was inaccurate; only that I would like to see a system that was more responsive to form, and assigned ranks in a more fluid way. That does not mean going to the opposite extreme, only that it would desirable to have a system that did not feel quite so like swimming through concrete.

_________________
Pasta of Muppets, pulled into strings
Twisted fusilli, macaroni with cream,
Filled up with me you can`t taste a thing,
Just call the waiter, he`ll hear you scream.
Pasta, pasta, eat up your pasta!

First of all, i didn't "snip" at Mef, just pointed out that he brought data that supports the position opposite that of his. Second of all, who said anything about wins against 4dans getting you promoted to 6dan. I was pointing out that in a period of large variance, his graph moves less that 1/4th of a stone in either direction.

Well, During this period, rj's wins as a 5dan varied between 44% and %49 - a strong performance but hardly erratic swings justifying huge rank changes. If anything, it's surprising he swung by even a half stone with such a steady win rate - perhaps we could blame this on heaviness, but that would be a little facile.

It's true that in a few months where he played very few games his win rate was much higher or lower ... 2 games at 0%, 17 at 75%... but surely you agree that given how many hundreds of games he plays, it would be cruel to bust him down to 1k on the basis of two games, or raise him to 7d on the basis of seventeen.

You are using the logic of the current system to argue against the logic of a new system. it would only be "cruel" if you assume that he would have to stay here. in a more fluid system his rank could move up and down alot and it would be neither cruel or illogical, but rather then your rank would reflect how well you were playing

Question: do you think there is any connection between "how well you were playing" and "how well you will play in the future"? Because there are many ways to look at "how well you were playing". You could look at yesterday, you could look at last week, you could look at the last week, you could look at the last three years, you could look at all the games you played sober, you could only look at Tuesdays because that's your lucky day.

My contention is that the only sensible way to quantify how well you've been playing is to choose the measure that best predicts how well you'll play in the future.

Just out of curiosity, if you flipped a coin 600 times and, during those 600 flips, got a sequence of 12 consecutive flips of which (in any order) 9 were heads and 3 were tails, would you believe that the odds of flipping heads had changed during those 12 flips?

The fault with this analogy is that people change, coins don`t.

_________________
Pasta of Muppets, pulled into strings
Twisted fusilli, macaroni with cream,
Filled up with me you can`t taste a thing,
Just call the waiter, he`ll hear you scream.
Pasta, pasta, eat up your pasta!