Life In 19x19 http://lifein19x19.com/ |
|
AlphaZero paper published in journal Science http://lifein19x19.com/viewtopic.php?f=18&t=16270 |
Page 2 of 2 |
Author: | lightvector [ Sun Dec 09, 2018 3:36 pm ] |
Post subject: | Re: AlphaZero paper published in journal Science |
seberle wrote: If Elo can't be converted to handicap at these high ranks, how do you determine handicap from Elo? Or can you? At what rank does the rule "100 Elo points = 1 rank" begin to break down? If you're equating ranks with stones, I'd say it breaks down all over the place, since 100 Elo points = 1 rank is not so great a rule of thumb to begin with. You might be confused due to the fact certain rating models used by various Go organizations or servers that alter the very definition of what an "Elo point" is to try to make it so that under those systems 100 "Elo points" = 1 rank by definition. But of course those altered "Elo points" have little to do with the traditional Elo points that presumably you're asking about, i.e. the ones that underlie FIDE chess ratings, goratings.org, BayesElo, WHR, and the ones that academic publications will usually use when reporting ratings differences. With traditional Elo points, a fixed Elo difference corresponds to a fixed modeled winning chance rather than a fixed rank difference, scaled so that 400 Elo ~= 10:1 winning odds. The correspondence between traditional Elo differences (i.e. winning chance) and rank difference is not simply a fixed ratio and it becomes highly nonlinear once you get into even amateur dan ranks, much less pro level or beyond. If you're interested in some actual data, I know there are some studies on OGS and/or KGS out there that have been done, or if you like, here's some old real data from EGF tournaments: http://gemma.ujf.cas.cz/~cieply/GO/statev.html That data is just among humans of course. If you want to add bots into the mix, any computer chess programmer will tell you that Elo differences between bots (particularly ones measured with self-play) don't necessarily translate into the same Elo differences against humans, and the same appears to be true for Go. And in Go it appears that without clever tricks like "dynamic komi" (or even with such tricks), strong Go bots also scale quite differently than humans in handicap games versus even games. Hope that helps. Basically rating and rankings are a pretty complex mess and you can't really boil it down into any simple rule. |
Author: | seberle [ Sun Dec 09, 2018 9:47 pm ] |
Post subject: | Re: AlphaZero paper published in journal Science |
lightvector wrote: seberle wrote: 100 Elo points = 1 rank is not so great a rule of thumb to begin with. Thanks, that was very helpful. Ok, see if I'm understanding better. The EGF rating system, for example, has modified the Elo system so as to force 100 rating points to be equivalent to one rank. If the table here (https://senseis.xmp.net/?EGFRatingSystem) is any indication, it looks like the EGF system wanders far from the Elo win rate of about 36% for one rank difference in the SDK ranks, but is reasonably close for DDK. Am I interpreting this correctly? |
Author: | jlt [ Mon Dec 10, 2018 3:07 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
My understanding is the same, but I think that winning percentages are calculated from a theoretical formula, and not really observed. To get observed winning percentages, go to the website http://www.europeangodatabase.eu/EGD/winning_stats.php Between 2003 and 2018, we get the table where the grades are the "declared grades". Assuming that declared grades reflect real strength accurately, we can see that At the 15k rank, 100 EGF points = 57 real Elo At the 5-10k ranks, 100 EGF points = 50 real Elo At the 2k rank, 100 EGF points = 72 real Elo At the 1d rank, 100 EGF points = 102 real Elo At the 3d rank, 100 EGF points = 117 real Elo At the 6d rank, 100 EGF points = 220 real Elo It seems however difficult to convert Elo points into handicap stones. We can read on the same website |
Author: | seberle [ Mon Dec 10, 2018 4:48 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
jlt wrote: It seems however difficult to convert Elo points into handicap stones. We can read on the same website The table says for instance that a 3d wins 19% of his H1 games against a 4d, which is very strange since he wins 33.8% of his even games against a 4d. So maybe there is a bias (people choose to play handicap games when they think that their real strength difference is larger than their official rank difference), so it's not easy to determine how many stones represent a difference of 1 EGF rank. Are you sure that "The table says for instance that a 3d wins 19% of his H1 games against a 4d"? I'm new to this, but I thought the table was saying the weaker player (any rank) wins 19% of their games against a player 3 ranks stronger when given a handicap of one stone. I'm not surprised that handicap stones don't even things out smoothly since the first "handicap stone" is just komi, which is only half the value of the first move. Two handicap stones are actually only worth 1 1/2 ranks, and so forth. Or at least, that is what I have understood. Correct me if I've gotten this wrong! |
Author: | jlt [ Mon Dec 10, 2018 5:06 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
Yes, I misread the table and you are right. Anyway the statistics of handicap games are not precise enough, so I don't have enough data to determine how many stones is worth one EGF rank difference. |
Author: | Bill Spight [ Mon Dec 10, 2018 9:46 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
seberle wrote: jlt wrote: It seems however difficult to convert Elo points into handicap stones. {snip} The table says for instance that a 3d wins 19% of his H1 games against a 4d, which is very strange since he wins 33.8% of his even games against a 4d. So maybe there is a bias (people choose to play handicap games when they think that their real strength difference is larger than their official rank difference), so it's not easy to determine how many stones represent a difference of 1 EGF rank. I'm not surprised that handicap stones don't even things out smoothly since the first "handicap stone" is just komi, which is only half the value of the first move. Two handicap stones are actually only worth 1 1/2 ranks, and so forth. Or at least, that is what I have understood. Correct me if I've gotten this wrong! Traditionally, rank differences were determined by handicap differences. In theory, one stone difference was equivalent to one rank difference. But handicap differences (at least for amateurs) gave an advantage to White, an advantage equivalent to komi (i.e., ½ stone). So a player two ranks stronger gave only a two stone handicap, with no komi, instead of giving three stones with Black giving komi or giving two stones with White giving komi. Modern tournament ranks and online ranks are based upon even games, and do not necessarily tell us the proper handicap. |
Author: | seberle [ Wed Dec 12, 2018 12:38 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
Quote: Traditionally, rank differences were determined by handicap differences. In theory, one stone difference was equivalent to one rank difference. But handicap differences (at least for amateurs) gave an advantage to White, an advantage equivalent to komi (i.e., ½ stone). So a player two ranks stronger gave only a two stone handicap, with no komi, instead of giving three stones with Black giving komi or giving two stones with White giving komi. Modern tournament ranks and online ranks are based upon even games, and do not necessarily tell us the proper handicap. This is interesting. First of all, how were handicap differences handled "traditionally" (do you mean before komi?). If we don't change komi, then what is the difference between a one-stone handicap and simply letting black go first? Or was going first considered being one rank stronger traditionally? Secondly, does either system work out precisely (without doing fine adjustments to komi)? I mean, if a two-stone handicap (any system) means a 7k can play an even game against a 5k and a 5k can play an even game against a 3k, does it necessarily mean that a four-stone handicap for the 7k will get an even game against the 3k? I suppose this question is even more important for the one-stone, two-stone question: if one stone means one rank, does two stones really mean two ranks? I know I saw a debate on Sensei's Library about this once, but I didn't understand it very well and I don't remember exactly where I saw it. |
Author: | John Fairbairn [ Wed Dec 12, 2018 4:01 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
Quote: This is interesting. First of all, how were handicap differences handled "traditionally" (do you mean before komi?). If we don't change komi, then what is the difference between a one-stone handicap and simply letting black go first? Or was going first considered being one rank stronger traditionally? To get your head round this you need to understand that the ranks we now use are a relatively modern construct. Traditionally (Edo times) ranks were limited to pro-level dans. They didn't even use komi. Honinbo Shuho introduced some lower grades for amateurs but his system was soon abandoned (for political reasons) and players reverted to the old dan-only/pro-only system, essentially until the democratisation after World War II. The amateurs started using their own dan scale then, and the first amateur 6-dan was Hirata Hironori in 1955 (for winning the 1st Amateur Honinbo - the prize nowadays is 8-dan). Since then amateurs in Japan were able to use kyus - and did, but the lower ranks have been used with much more gusto in the west. Indeed, a number-only system was introduced by amateurs in Germany even before the war, and was either used or copied by other western amateur associations. We have seen western amateurs - very many with a mathematical background like those early Germans - obsessively try to apply rules and numbers to many aspects of go. But handicaps existed well before ranking systems and so it follows they can have no real correlation. They were used to a very limited extent between pros but mostly were (and still are, in Japan) nothing more than a teaching tool. No doubt for that pedagogic reason, too, the stone placings were fixed - the idea of free handicap placement is another modern idea, inspired first by mathematical amateurs in Japan (and even giving rise to a book on them by a pro!). The use of these for rankings, and of komi, likewise has no tradition (or even theory) behind it. The use of komi (mainly in trying to determine what an even game means - and that's varied a lot in the last 100 years) is likewise mainly an Japanese amateur idea, from 1751. Pros tried it a few times from the early 19th century, starting at 5 points and gradually reducing over the decades until it even reached 2 points. It only started to rise after World War II. So you can see that trying to tie ranks to handicaps is like climbing up a greasy pole. Historical grades and handicaps both differed for pros and amateurs from modern ones, modern grades and handicaps differ between amateurs and pros (and by country). Komi has been messed up for 300 years. The philosophical drive behind rankings differs in the west and the Far East. People have different ideas on how to implement handicaps. Etc, etc. Life is too short to worry about such things. Of course we all would like a way of quantifying how much stronger A is than B, but it seems sensible to accept it's always going to be a wild guesstimate - at least until we get a DeepRatingsZeroPixie algorithm. |
Author: | jlt [ Wed Dec 12, 2018 4:34 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
Statistics on handicap games on the EGD website lack precision but seem to show that handicaps are approximately additive (if a rank A is 2 stones stronger than B, which is 3 stones stronger than C, then A is 5 stones stronger than C). However some players are particularly good at playing with or against handicap, and vice-versa, so the proper handicap between two given people cannot be predicted by subtracting their ranks. Bots are an extreme example (very bad at handicap go). |
Author: | moha [ Wed Dec 12, 2018 7:06 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
It is true that using the number of (extra) black stones as handicap - without black giving komi - is mathematically incorrect. It also makes it harder to maintain a good "feeling" about who is ahead (since in some games W gets komi which must be factored into the human intuition, but in other games he gets none - same problem as for a value net). So the same board position can be roughly even AND W significantly ahead depending on this. But there are problems even without the doubtful human systems. If player A wins 50% against player B even after passing his first move, he is clearly one stone stronger. His winrate against B in even games cannot be correctly guessed from this, as that also depends on how close they are to perfect play. IMO there are two ways to define one rank class: a certain (like 70%) winrate in even games (ELO-like), OR 50% winrate in N-stone games. For the latter approach, the scale ends with perfect play being a few stones above top pro level. For the former, the scale continues to amplify smaller and smaller strength differences and has many more steps after top pros (but is probably still bounded somewhere). |
Author: | Bill Spight [ Sun Dec 16, 2018 1:33 pm ] |
Post subject: | Re: AlphaZero paper published in journal Science |
seberle wrote: Quote: Traditionally, rank differences were determined by handicap differences. In theory, one stone difference was equivalent to one rank difference. But handicap differences (at least for amateurs) gave an advantage to White, an advantage equivalent to komi (i.e., ½ stone). So a player two ranks stronger gave only a two stone handicap, with no komi, instead of giving three stones with Black giving komi or giving two stones with White giving komi. Modern tournament ranks and online ranks are based upon even games, and do not necessarily tell us the proper handicap. This is interesting. First of all, how were handicap differences handled "traditionally" (do you mean before komi?). If we don't change komi, then what is the difference between a one-stone handicap and simply letting black go first? Or was going first considered being one rank stronger traditionally? Without komi, evenly matched players alternated between taking White and taking Black. Traditionally, an amateur player one rank weaker simply took Black, a player two ranks weaker took two stones, etc. This system favors the stronger player, given that one rank difference is roughly equivalent to one handicap stone difference. A player one rank weaker than his opponent should alternate between taking Black and taking two stones. Long ago, pro ranks followed a similar system, with one rank difference roughly equivalent to a ½ handicap stone difference. So against a 9 dan pro an 8 dan took Black, a 7 dan alternated between Black and two stones, a 6 dan took two stones, etc. Over time, pro ranks got closer together, so they used a different handicapping system. These days, pros do not give handicaps to other pros, with perhaps rare exceptions. Quote: Secondly, does either system work out precisely (without doing fine adjustments to komi)? I mean, if a two-stone handicap (any system) means a 7k can play an even game against a 5k and a 5k can play an even game against a 3k, does it necessarily mean that a four-stone handicap for the 7k will get an even game against the 3k? I suppose this question is even more important for the one-stone, two-stone question: if one stone means one rank, does two stones really mean two ranks? I know I saw a debate on Sensei's Library about this once, but I didn't understand it very well and I don't remember exactly where I saw it. Go requires many different skills, so no ranking system will be precise. However, handicap stones are surprisingly additive. I once gave a 40 stone handicap and won by 10 pts. Surprisingly close. |
Author: | hyperpape [ Sun Dec 16, 2018 1:51 pm ] |
Post subject: | Re: AlphaZero paper published in journal Science |
At one go congress, I did a simul with a professional and the player next to me asked how many stones to place. His response was "9 stones if you want to win, 7 if you want to learn". It is arguably a feature, not a bug, that traditional handicap schemes favor the stronger player. |
Author: | Kirby [ Sun Dec 16, 2018 6:06 pm ] |
Post subject: | Re: AlphaZero paper published in journal Science |
hyperpape wrote: At one go congress, I did a simul with a professional and the player next to me asked how many stones to place. His response was "9 stones if you want to win, 7 if you want to learn". It is arguably a feature, not a bug, that traditional handicap schemes favor the stronger player. There's somewhat of a psychological factor as well. If you believe that the opponent is stronger, it's necessary to fight the urge to, e.g., play conservatively when the situation is unclear. It may not be true in the extreme, but to some effect, if you think your opponent can beat you at X-stones, (s)he has a decent chance of doing so. |
Author: | seberle [ Sat Dec 22, 2018 12:23 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
John Fairbairn wrote: The use of komi (mainly in trying to determine what an even game means - and that's varied a lot in the last 100 years) is likewise mainly an Japanese amateur idea, from 1751. Pros tried it a few times from the early 19th century, starting at 5 points and gradually reducing over the decades until it even reached 2 points. It only started to rise after World War II. I had no idea komi was such an old idea. Apparently Wikipedia doesn't know this either! (https://en.wikipedia.org/wiki/Rules_of_Go#Komi) Wikipedia wrote: Before the 20th century, there was no komi system. I might be interesting to include a bit more historical information about komi in the Wikipedia article. Is there a reference for 18th and 19th century use of komi? |
Author: | seberle [ Sat Dec 22, 2018 12:36 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
Furthermore, the Wikipedia article on the history of komidashi (https://en.wikipedia.org/wiki/Komidashi#History) specifically says: Wikipedia wrote: The compensation (komi) system was introduced into professional Go in Japan as a gradual process of innovation, beginning in the 1930s. What reference could be used to add more correct details to these articles? |
Author: | seberle [ Sat Dec 22, 2018 12:41 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
seberle wrote: Furthermore, the Wikipedia article on the history of komidashi (https://en.wikipedia.org/wiki/Komidashi#History) specifically says: Wikipedia wrote: The compensation (komi) system was introduced into professional Go in Japan as a gradual process of innovation, beginning in the 1930s. What reference could be used to add more correct details to these articles? The same article cites John Fairbairn as a source of komi history (but no specific citation). The only reference (at the bottom of the article) is the Sensei's Library page, which just says "there were some games played with compensation in the 19th century." |
Author: | John Fairbairn [ Sat Dec 22, 2018 3:36 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
Quote: The same article cites John Fairbairn as a source of komi history (but no specific citation). seberle: The original source was New In Go, a copy of which comes with the GoGoD database. I vastly expanded this into a longish chapter in The Go Companion, a book published by Slate & Shell. But S&S have ceased their own publication of paper books. I plan to re-issue an on-demand version, much updated, but that will have to wait until I finish my current mega-project. Giving these references does not, of course, imply that I am happy to see my research migrate to Wikipedia or elsewhere |
Author: | seberle [ Sun Dec 23, 2018 8:20 am ] |
Post subject: | Re: AlphaZero paper published in journal Science |
John Fairbairn wrote: Quote: The same article cites John Fairbairn as a source of komi history (but no specific citation). I vastly expanded this into a longish chapter in The Go Companion, a book published by Slate & Shell. But S&S have ceased their own publication of paper books. I plan to re-issue an on-demand version, much updated, but that will have to wait until I finish my current mega-project. John Fairbairn: I'd been wanting to buy The Go Companion for a long time. I hope your "mega-project" is finished soon! |
Page 2 of 2 | All times are UTC - 8 hours [ DST ] |
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |