Oddities in KGS ranking system

Comments, questions, rants, etc, that are specifically about KGS go here.
User avatar
daniel_the_smith
Gosei
Posts: 2116
Joined: Wed Apr 21, 2010 8:51 am
Rank: 2d AGA
GD Posts: 1193
KGS: lavalamp
Tygem: imapenguin
IGS: lavalamp
OGS: daniel_the_smith
Location: Silicon Valley
Has thanked: 152 times
Been thanked: 330 times
Contact:

Re: Oddities in KGS ranking system

Post by daniel_the_smith »

uPWarrior wrote:... games should get more important (or "heavier") the farther from the expected outcome one gets. The 5th win in a 5win streak should increase one's rating more than a win after a win-lose-win-lose scenario.


Bayes' Theorem describes exactly how much you should change a probability in response to new evidence, which is why the word "Bayesian" keeps showing up in these rating discussions. if you want to join the Bayesian Conspiracy with shapenaji, Redundant, and me, read this: http://yudkowsky.net/rational/bayes

If you search for the phrase "In front of you is a bookbag", you'll see an example relevant to this discussion, although it won't make much sense unless you've read the prior examples.

In the terms used there, every game you win is a few decibels of evidence that you're under-ranked, and every loss is an equal amount of evidence that you're over-ranked (the amount of evidence should be scaled for the strength difference, of course).

People find a lot more patterns in random data than is actually there. The whole point of the rating system is to figure out if your 5 recent wins are enough evidence of improvement to counteract your average prior performance. I'd expect WHR (which is a Bayesian system) to do much better at making sense of such runs than systems that just add on an arbitrary bonus for runs (is WLWWWWLW really worse than LLWWWWWW? I think it depends a lot on time elapsed between the games). WHR assumes your strength changes over time. I'm not aware of another system that does.
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com
User avatar
jts
Oza
Posts: 2662
Joined: Sat Sep 18, 2010 4:17 pm
Rank: kgs 6k
GD Posts: 0
Has thanked: 310 times
Been thanked: 632 times

Re: Oddities in KGS ranking system

Post by jts »

Redundant wrote:
jts wrote:
Redundant wrote:Yes! 10 heads in a row is starting to be good evidence that you aren't dealing with a fair coin.


If the ten flips were HTHTHTHTHT, would you consider that evidence that it was a trick coin?


That is some small amount of evidence that the coin alternates heads and tails. If you can't tell, I'm very much a bayesian.

And if the ten flips were HTTHHHTTTT, would that be some small evidence that the next five flips would be HHHHH?
User avatar
Redundant
Lives in sente
Posts: 924
Joined: Thu Apr 22, 2010 3:00 pm
Rank: lazy
GD Posts: 0
KGS: redundant/silchas
Tygem: redundant
Wbaduk: redundant
DGS: redundant
OGS: redundant
Location: Pittsburgh
Has thanked: 45 times
Been thanked: 103 times

Re: Oddities in KGS ranking system

Post by Redundant »

jts wrote:And if the ten flips were HTTHHHTTTT, would that be some small evidence that the next five flips would be HHHHH?


Can you get to your point here? Any seeming structure in the result of a trial is evidence for nonrandomness. The strength of the evidence relies on the improbability of the result.
User avatar
jts
Oza
Posts: 2662
Joined: Sat Sep 18, 2010 4:17 pm
Rank: kgs 6k
GD Posts: 0
Has thanked: 310 times
Been thanked: 632 times

Re: Oddities in KGS ranking system

Post by jts »

Redundant wrote:
jts wrote:And if the ten flips were HTTHHHTTTT, would that be some small evidence that the next five flips would be HHHHH?


Can you get to your point here? Any seeming structure in the result of a trial is evidence for nonrandomness. The strength of the evidence relies on the improbability of the result.

My point is, that after ten penny flips, you are going to have some highly improbably sequence of heads and tails, on which you can happily impose some ridiculous structure. If you start with a set of priors that is significantly different from the data-generating process, Bayes' rule will give you silly results.
User avatar
daniel_the_smith
Gosei
Posts: 2116
Joined: Wed Apr 21, 2010 8:51 am
Rank: 2d AGA
GD Posts: 1193
KGS: lavalamp
Tygem: imapenguin
IGS: lavalamp
OGS: daniel_the_smith
Location: Silicon Valley
Has thanked: 152 times
Been thanked: 330 times
Contact:

Re: Oddities in KGS ranking system

Post by daniel_the_smith »

I heard the Rev. Bayes' ghost will torment you in your sleep if you blame him for your horrible priors...
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com
User avatar
shapenaji
Lives in sente
Posts: 1103
Joined: Tue Apr 20, 2010 10:58 pm
Rank: EGF 4d
GD Posts: 952
Location: Netherlands
Has thanked: 407 times
Been thanked: 422 times

Re: Oddities in KGS ranking system

Post by shapenaji »

jts wrote:My point is, that after ten penny flips, you are going to have some highly improbably sequence of heads and tails, on which you can happily impose some ridiculous structure. If you start with a set of priors that is significantly different from the data-generating process, Bayes' rule will give you silly results.


The sequence doesn't really matter for Bayes,

But you're right, It really comes down to the strength of your prior.

The only reason why people believe that after 10 heads in a row, that the next flip is 50-50 is because they have strong confidence in the prior of the coin. They don't believe in an unfair coin, so therefore it doesn't matter how much evidence arrives to support that conclucsion.

Tweaking your prior is important, if you don't think 5 heads in a row is enough to change the bias of the coin much, then by-all-means, make the prior strong enough to absorb 5 heads in a row.

There's nothing silly about the results, the only silly part is your choice of prior.
Tactics yes, Tact no...
hyperpape
Tengen
Posts: 4382
Joined: Thu May 06, 2010 3:24 pm
Rank: AGA 3k
GD Posts: 65
OGS: Hyperpape 4k
Location: Caldas da Rainha, Portugal
Has thanked: 499 times
Been thanked: 727 times

Re: Oddities in KGS ranking system

Post by hyperpape »

Just to throw this out there: Bayes' theorem is just a fact of probability. It is the particular ideas about assigning priors and updating via conditionalization that are philosophically/mathematically controversial.
uPWarrior
Lives with ko
Posts: 199
Joined: Mon Jan 17, 2011 1:59 pm
Rank: KGS 3 kyu
GD Posts: 0
Has thanked: 6 times
Been thanked: 55 times

Re: Oddities in KGS ranking system

Post by uPWarrior »

It seems like my comment on a possible alternative with a volatility factor was completely ignored and the discussion quickly focused on the oversimplistic 5win streak.

The example I provided is not related to Bayes' theorem nor to patterns.
Last edited by uPWarrior on Fri Sep 09, 2011 5:32 am, edited 1 time in total.
xed_over
Oza
Posts: 2264
Joined: Mon Apr 19, 2010 11:51 am
Has thanked: 1179 times
Been thanked: 553 times

Re: Oddities in KGS ranking system

Post by xed_over »

uPWarrior wrote:The example I provided is not related to Bayes' theorem nor to patterns.

yes it is, actually
Post Reply