Oddities in KGS ranking system

[shapenaji]

Herman: You know, I've often wondered if the best approach to rank is to track a player's distribution and mean. And then use bayes theorem to update the distribution based on the distribution of their defeated and victorious opponents.

Of course it is... why wouldn't it be?

Well, to my knowledge, no one has ever implemented a bayesian-style updating distribution as rating. Most folks seem to be content tracking 1 number.

A player's distribution should tell you a lot, if they're strong against lower ranked players, it would be skewed one way, strong against stronger players, it would be skewed another.

But seeing as it's never been done, maybe there's a reason...

[Kaya.gs]

...
Besides accounts being heavy and such, there is an impressive psychological aspect of the system that does not feel to affect point-based systems like in Wbaduk or Tygem.

Back then when playing with danigabi[5d] account i have played certain 2ds giving them 3 handicap stones. I would win & lose, and i think i won a tad more than lost (say 60%). The impressive happens later. Right after losing a game, i would log back in with Rakuen[7d], and play the very same player with 6H. Suddently, i would win almost 80%.

How is it possible that increasing many stones , my chances to win go up. My current account, DexMorgan, has been brought up to 7d with a similar effect.

I think this is a specific anomaly of this history-based rating system, where the psychology of the palyers deeply affect the end results and hence its accuracy.
...

Call me anal, but I can't see a claim like this without wanting to check the facts. (That is also why I like GoGoD so much!) Happily we have the KGS Archives. Memory is a tricky little beast. I am sure that we have all had the experience of retelling moments of remembered glory over a beer only to find out afterwards that things weren't quite like that. So I was not too surprised that an examination of the archives for Rakuen and danigabi did not immediately turn up a lot of examples that fit the situation described above. Maybe kaya.gs could point out which games he was referring to?

Rakuen has little games, but check out dex morgan.It has 80 games in the last 2 months.

I believe that i could defend a 7d on good days, which i rarely get, and im 6d on average. I always felt danigabi was under-rated, and i think dexmorgan is slightly over-rated.
Buth both accounts have almost 1.5 stone difference.

I believe this is a major psychological effect on my opponents rather than myself. And this effect i can only see on KGS, specially on high handicap.

Flashback did something similar, and so did other strong players, that giving extra handi is a rating/probability of winning success.

However the opposite is true. playing only proper handi games feels like a total drag, as losses are very hard-felt. This is from a user stand-point and the evidence is anecdotical. But i have no reason to believe opposite either.

[palapiku]

Herman: You know, I've often wondered if the best approach to rank is to track a player's distribution and mean. And then use bayes theorem to update the distribution based on the distribution of their defeated and victorious opponents.

Of course it is... why wouldn't it be?

Well, to my knowledge, no one has ever implemented a bayesian-style updating distribution as rating. Most folks seem to be content tracking 1 number.

A player's distribution should tell you a lot, if they're strong against lower ranked players, it would be skewed one way, strong against stronger players, it would be skewed another.

But seeing as it's never been done, maybe there's a reason...

The reason is almost certainly that it's too much work and trouble for a very small benefit.

Of course, if we were really good proper Bayesians, we'd never assume something as preposterous (and obviously false) as the idea that strength can be reduced to a single metric (which is still the core assumption even when you're tracking a changing distribution). Instead, we'd track individual players' results against one another and then try to extrapolate stuff from that, in some much more complicated fashion. There wouldn't be numeric rankings at all, instead the system would be able to tell you your chance of winning against a specific player.

[uPWarrior]

I believe that some of the problems raised could be solved using a "volatility" factor.
Please check this rating system as a possible example:
http://apps.topcoder.com/wiki/display/t ... ing+System

It would obviously need to be adapted as we have 1on1 games, but games should get more important (or "heavier") the farther from the expected outcome one gets. The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.

[hyperpape]

Of course, if we were really good proper Bayesians, we'd never assume something as preposterous (and obviously false) as the idea that strength can be reduced to a single metric (which is still the core assumption even when you're tracking a changing distribution). Instead, we'd track individual players' results against one another and then try to extrapolate stuff from that, in some much more complicated fashion. There wouldn't be numeric rankings at all, instead the system would be able to tell you your chance of winning against a specific player.

But if we're good decision theorists and/or programmers, we'll think about the cost of various implementations both in terms of human and computer effort. A pure bayesian approach probably fails on both counts, while Shapenaji's solution might work. And if we're pragmatists, we'll say that since people will continue to fuss about rating systems until the end of time, we might as well turn that sunk cost of time into new rating system prototypes since that's still better than arguing about ratings on forums (economists will disagree).

[hyperpape]

So I was not too surprised that an examination of the archives for Rakuen and danigabi did not immediately turn up a lot of examples that fit the situation described above. Maybe kaya.gs could point out which games he was referring to?

Rakuen has little games, but check out dex morgan.It has 80 games in the last 2 months.

So you gave a story, got challenged to provide specific games, and then responded by ...not providing specific games?

[xed_over]

The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.

really?!

so if I flip a coin 10 times and it comes up heads all ten times, should I expect the probability of a heads result on the next toss to be greater than 50/50?

[judicata]

so if I flip a coin 10 times and it comes up heads all ten times, should I expect the probability of a heads result on the next toss to be greater than 50/50?

This reminds me of a conversation in which I tried to convince someone that, if a roulette ball lands on red 5 times in a row, the chance of it landing on red the 6th time is still slightly less than 50/50 (because of the green slot).

But, this may be slightly different than a coin toss or roulette wheel, because a streak may indicate recent significant improvement (or maybe not?).

[Redundant]

The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.

really?!

so if I flip a coin 10 times and it comes up heads all ten times, should I expect the probability of a heads result on the next toss to be greater than 50/50?

Yes! 10 heads in a row is starting to be good evidence that you aren't dealing with a fair coin.

[daal]

The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.

really?!

so if I flip a coin 10 times and it comes up heads all ten times, should I expect the probability of a heads result on the next toss to be greater than 50/50?

There is a difference between flipping a coin and playing a game of go. In go, the possibility exists that greater skill is causing the winning streak, and the longer the streak, the more probable that this explanation is plausible, no? If a flipped coin turned up heads ten times in a row, I'd also suspect that skill is involved.

[danielm]

The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.

really?!

so if I flip a coin 10 times and it comes up heads all ten times, should I expect the probability of a heads result on the next toss to be greater than 50/50?

There is a difference between flipping a coin and playing a game of go. In go, the possibility exists that greater skill is causing the winning streak, and the longer the streak, the more probable that this explanation is plausible, no? If a flipped coin turned up heads ten times in a row, I'd also suspect that skill is involved.

That's why I (like HermanHiddema) suggested that after a number of wins, the player is matched up like a higher rank. Because of the ranking difference, he will automatically gain more rating points if he wins.

Insisting on proper handicaps in every game seems misguided to me, when the goal is to rapidly track a player's current strength. Someone might have a long winning streak against even opponents by chance, but if he keeps winning against stronger opponents, that data is much more conclusive.

[shapenaji]

The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.

really?!

so if I flip a coin 10 times and it comes up heads all ten times, should I expect the probability of a heads result on the next toss to be greater than 50/50?

Who let the frequentist in here? *shoos xed_over away*

BAYESIANS ONLY

[wms]

The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.

really?!

so if I flip a coin 10 times and it comes up heads all ten times, should I expect the probability of a heads result on the next toss to be greater than 50/50?

I'm trying to stay out of this discussion since it is about ideas for a differenc rank system from KGS, but I can't resist adding a comment on "the 5th win in a 5win streak..." business. xed_over is right, it's nonsense for the the 5th win to mean more than the 1st win after a loss. I think the idea of each win in a streak mattering more is because, "win-win-win-win-win" should definitely increase your rank more than "win-lose-win-lose-win," so it feels like the 5th win should count for more. But of course almost any rank system will give you a bigger boost from the 5 wins than from the 3 wins and the 2 losses, so the situation is already handled without needing to give the 5th win more significance.

[jts]

If a flipped coin turned up heads ten times in a row, I'd also suspect that skill is involved.

Yes! 10 heads in a row is starting to be good evidence that you aren't dealing with a fair coin.

If the ten flips were HTHTHTHTHT, would you consider that evidence that it was a trick coin?

[Redundant]

Yes! 10 heads in a row is starting to be good evidence that you aren't dealing with a fair coin.

If the ten flips were HTHTHTHTHT, would you consider that evidence that it was a trick coin?

That is some small amount of evidence that the coin alternates heads and tails. If you can't tell, I'm very much a bayesian.