Life In 19x19

shapenaji wrote:

Mef wrote:

fentonaop wrote:
I mentioned 'inadequate' because the first winner only plays with four other players and the second winner only plays with five (or four) other players. They are not well exposed to many other players. This system equivalents to single elimination albeit the two winners have to defeat some of their opponents twice.

The problem with your argument is this: If you failed to play either of the final qualifiers in direct competition, that means there were at least two other competitors in the field you have lost to. If you have already lost to two other competitors in the field before challenging the two players who have (thus far) proved to be strongest, it is very hard to make a legitimate claim you were one of the top two competitors in the tournament. One could perhaps reasonably argue that there is not enough information to distinguish the "1st place" qualifier (winner of the winners bracket] from the "2nd place" qualifier (winner of the loser's bracket), because there is a chance the 1 loss could occur to an opponent not faced by the champion...but that's not really the goal of this tournament. This tournament is meant to divide 1&2 from 3-16...and that's what it does.

Here's a thought experiment:

You have 16 players of identical strength, On any given day, one of them will beat another one with a 50-50 shot.

The double elimination tournament MUST produce a first and second place (by design).

But that doesn't mean that it's a particularly good test of relative strength.

In this case, a proper test should find all of the players tied. And this test is not sensitive to that.

shapenaji, if it was your job to select 2 players to become pros from this hypothetical field of 16 players, how would you do it?

tapir wrote:The thought experiment has a major flaw: With identical strength every result is justified. (And even the most elaborate tournament system can't produce a better pick than k.o., double k.o. or even a coin toss competition.)

Or in other words, the qualifier isn't intended to produce an evaluation of relative strength, but to select two players.

The point of the thought experiment was to illustrate that the knock out tournament says nothing about how the winners would have fared against the players they didn't play.

The most elaborate tournament system certainly CAN produce a better pick than ko or dko.
Simply, round robin, you may have to break ties, but the round robin gives you a very strong understanding of the field. (Technically, I think a double round robin, where each player plays each other player as both white and black, is the strongest though)

I don't think the best of 3 is particularly useful. There are considerations of playing style. Beating one person in 2 games may be considerably easier than beating 2 people in 2 games.

yoyoma wrote:shapenaji, if it was your job to select 2 players to become pros from this hypothetical field of 16 players, how would you do it?

We have 11 games right now, I think we can find a way to fit in 4 more. 2 games a day for a week?

Basically, I think this tournament should be a big deal, and we shouldn't jump to the format that guarantees a winner in a shorter time.

hyperpape wrote: I'm gonna guess that there's no tournament that you can feasibly hold in a week that gives you reliable information about relative strength in a case where players are extremely close to 50-50. That's just the nature of coin flips--someone will get lucky.

See above post, Round Robin, 2 games a day. The tournament would be finished in 8 days.

If you're willing to cram 3 games into each day (making the test into a real endurance challenge) you could do it in 5 days.

shapenaji wrote:The point of the thought experiment was to illustrate that the knock out tournament says nothing about how the winners would have fared against the players they didn't play.

My point wasn't that you are necessarily wrong, but that your thought experiment simply doesn't work as you need no data at all to make a good selection, if the players are of the same strength.

I often see this kind of argument about tournaments when people seem to assume that the aim is the best measurement of relative strength, but usually this isn't the aim (take a look at the rating if you need this measurement). You want to determine a winner, based on performance (in the games played) and as long as not obviously inferior players get selected everyone is happy with the result. And of course nobody takes players serious, who complain, that they lost their games because they were paired against Peter (6d) instead of Hans (6d), although they technically could be right. Multiple rounds of an elimination tournament should be enough to eliminate weaker players, but whether someone eliminated would be a tenth of a stone stronger than the actual winner on a good day with sunshine doesn't matter at all. In fact, it adds drama to the whole thing.

shapenaji wrote:

hyperpape wrote: I'm gonna guess that there's no tournament that you can feasibly hold in a week that gives you reliable information about relative strength in a case where players are extremely close to 50-50. That's just the nature of coin flips--someone will get lucky.

See above post, Round Robin, 2 games a day. The tournament would be finished in 8 days.

If you're willing to cram 3 games into each day (making the test into a real endurance challenge) you could do it in 5 days.

Huh? Hyperpape's point is that anyone with no prior information about the strength of the players would be forced to draw the conclusion that the player who went 10-5 was significantly stronger than the player who went 5-10, but we know ex hypothesi that these two players are equally strong; their records in this tournament are statistical noise. Thus: "There's no tournament that you can feasibly hold in a week that gives you reliable information on relative strength..."

jts wrote:

shapenaji wrote:
See above post, Round Robin, 2 games a day. The tournament would be finished in 8 days.

If you're willing to cram 3 games into each day (making the test into a real endurance challenge) you could do it in 5 days.

Huh? Hyperpape's point is that anyone with no prior information about the strength of the players would be forced to draw the conclusion that the player who went 10-5 was significantly stronger than the player who went 5-10, but we know ex hypothesi that these two players are equally strong; their records in this tournament are statistical noise. Thus: "There's no tournament that you can feasibly hold in a week that gives you reliable information on relative strength..."

The point is that the Round Robin has the capacity of showing that the players are very close in strength, whereas a KO does not. The point wasn't that the Round Robin works perfectly, just better.

tapir wrote:

shapenaji wrote:The point of the thought experiment was to illustrate that the knock out tournament says nothing about how the winners would have fared against the players they didn't play.

My point wasn't that you are necessarily wrong, but that your thought experiment simply doesn't work as you need no data at all to make a good selection, if the players are of the same strength.

Here's the thing though, the thought experiment still holds as we move the ranks away from 50% gradually. Sure, if they're all exactly equal, we can take whatever player we like. But when the difference between players is small, we need as many games as we possibly can to test that.

I often see this kind of argument about tournaments when people seem to assume that the aim is the best measurement of relative strength, but usually this isn't the aim (take a look at the rating if you need this measurement). You want to determine a winner, based on performance (in the games played) and as long as not obviously inferior players get selected everyone is happy with the result. And of course nobody takes players serious, who complain, that they lost their games because they were paired against Peter (6d) instead of Hans (6d), although they technically could be right. Multiple rounds of an elimination tournament should be enough to eliminate weaker players, but whether someone eliminated would be a tenth of a stone stronger than the actual winner on a good day with sunshine doesn't matter at all. In fact, it adds drama to the whole thing.

The aim here IS the best measurement of relative strength though. We're after someone to represent us in international competition. I think we can fairly say that it's not the pro qualifier that should be the source of our excitement, but the professional career. The pro qualifier should be difficult, and accurate.

shapenaji wrote: I think we can fairly say that it's not the pro qualifier that should be the source of our excitement, but the professional career. The pro qualifier should be difficult, and accurate.

I think that a tournament system that was more difficult and accurate, would be more exciting. I would have enjoyed seeing a round robin tournament.

shapenaji wrote:

tapir wrote:

shapenaji wrote:The point of the thought experiment was to illustrate that the knock out tournament says nothing about how the winners would have fared against the players they didn't play.

My point wasn't that you are necessarily wrong, but that your thought experiment simply doesn't work as you need no data at all to make a good selection, if the players are of the same strength.

Here's the thing though, the thought experiment still holds as we move the ranks away from 50% gradually. Sure, if they're all exactly equal, we can take whatever player we like. But when the difference between players is small, we need as many games as we possibly can to test that.

Well, you are the one who suggested "recognizes when all players are equally strong" as a test of a tournament set up. But in fact (I think you now agree?), this is not something that distinguishes various tournament formats. In all cases, the only way to increase our chances of getting the strongest player in a tight field is to play more games.

As to tapir's comment; I think you've dodged his main point, which is "If you are going to be an extremist about accuracy, why not use official lifetime AGA rating as the tool for selecting pros?"

jts wrote:
Well, you are the one who suggested "recognizes when all players are equally strong" as a test of a tournament set up.

Right, the point of the test was to illustrate that when players were closely rated, a knockout tournament is a poor judge of who is the best player.

How are we arguing about the fact that Round Robin is a better test of relative strength??

But in fact (I think you now agree?), this is not something that distinguishes various tournament formats. In all cases, the only way to increase our chances of getting the strongest player in a tight field is to play more games.

But this does distinguish formats. It is pathological in both cases, but the round robin is capable of allowing ties... So obviously, if the players are closely rated, it will do a better job of establishing this.

The Knockout tournament simply lacks depth perception.

As to tapir's comment; I think you've dodged his main point, which is "If you are going to be an extremist about accuracy, why not use official lifetime AGA rating as the tool for selecting pros?"

I don't think I'm being an extremist, I'm suggesting we follow the same pro format as Japan...

Because lifetime AGA rating is a test of past performance.


EDIT:

More generally, if someone is going to become a pro, they should need to fight against all the other opponents in the qualifier. Everybody gets a chance at the pinata.

The round robin is capable of allowing ties, but it's vanishingly unlikely that sixteen people would tie, to the point of being merely symbolic... meanwhile, the point stands that a tie between any two players for the second pro spot is not a desideratum of a qualifying tournament.

I agree with you about why we don't want to use lifetime performance as our benchmark (I would add that we want to see how people handle themselves under the pressure of such high stakes), but once you add these factors (we want current performance of all contenders, we want performance against each other rather than against random other people, we want all contenders to have an equal chance to train and compete...), you are left with the problem of scheduling a finite number of games, and once you've decided how many games you have time for, you need to decide how much time you have to play. Gangshen Shi played 12 games and went 8-4, while Andy played 8 and went 7-1. Scheduling an extra three games for each of them might have been nice, but I don't see any evidence that *given* the number of games they played, the elimination format was worse than some sort of Swiss.

By the way, are you sure you're right about Japanese? I was under the impression that players there are removed from the qualification tournament once they're out of contention, which is effectively a form of elimination rather than a round robin, but I'll take your word for it if you say otherwise. If you have a long, grueling tournament in which people compete past the point where they have any chance to win, it can create problems.

jts wrote:The round robin is capable of allowing ties, but it's vanishingly unlikely that sixteen people would tie, to the point of being merely symbolic... meanwhile, the point stands that a tie between any two players for the second pro spot is not a desideratum of a qualifying tournament.

True, but if you hold these tournaments repeatedly, the round robin will show all the players to be tied more significantly than a knockout run the same number of times.

I agree with you about why we don't want to use lifetime performance as our benchmark (I would add that we want to see how people handle themselves under the pressure of such high stakes), but once you add these factors (we want current performance of all contenders, we want performance against each other rather than against random other people, we want all contenders to have an equal chance to train and compete...), you are left with the problem of scheduling a finite number of games, and once you've decided how many games you have time for, you need to decide how much time you have to play. Gangshen Shi played 12 games and went 8-4, while Andy played 8 and went 7-1. Scheduling an extra three games for each of them might have been nice, but I don't see any evidence that *given* the number of games they played, the elimination format was worse than some sort of Swiss.

It's true that in this case, when the top players are clear outliers to the rest of the field, it doesn't really matter which format you use. But in future, the difference may not be as stark. I think we need to prepare for a time when the field is deeper.

I also think that, because style is such a factor in go (there are people at my same rating that I'm pretty convinced I'd never lose against, and people who I just can't beat), having 3 games against another player is not likely to give us more useful information than having those 3 games be against 3 opponents.

EDIT: And hence an 7-1 or 8-4 score seems somewhat less impressive. It looks more to me like a 4-0 and a 3-1

By the way, are you sure you're right about Japanese? I was under the impression that players there are removed from the qualification tournament once they're out of contention, which is effectively a form of elimination rather than a round robin, but I'll take your word for it if you say otherwise. If you have a long, grueling tournament in which people compete past the point where they have any chance to win, it can create problems.

They may remove players that are completely out of contention (I'm not sure), but that's a very different form of elimination, which takes into account how the field is doing. If there's a clear outlier, people will be eliminated quickly, while if everyone is close, it could go for the full round robin.

I'm not positive I agree with that approach, since the top players could have scored a number of their victories against those weaker players, and other players will then have to work harder to get the same score (because those weaker players won't be around to score wins off).
The only way that would make sense to me would be to eliminate their existence in the tournament when they're eliminated. Basically you subtract wins against them from everyone else's score.

Alternately, you run the entire round-robin tournament, but with some kind of prize for everyone down to next-to-last place, so that people are still fighting for something.

jts wrote:By the way, are you sure you're right about Japanese? I was under the impression that players there are removed from the qualification tournament once they're out of contention, which is effectively a form of elimination rather than a round robin, but I'll take your word for it if you say otherwise. If you have a long, grueling tournament in which people compete past the point where they have any chance to win, it can create problems.

I guess you are mixing the preliminary to the exam (as portrayed in HnG, which was a very unique format there - 3 wins to proceed / 3 losses to disqualify) with the actual exam, which is (in reality) a round robin league. And of course a round robin league might have the major drawback that the selection might be based on games of players that have nothing at stake anymore. And you don't want to set up a tournament system that potentially gives bad incentives (see Badminton, Sumo, Ice Hockey etc.).

shapenaji wrote:It's true that in this case, when the top players are clear outliers to the rest of the field, it doesn't really matter which format you use. But in future, the difference may not be as stark. I think we need to prepare for a time when the field is deeper.

Right, I take it what you are saying is that if someday we have a tournament where 15 players have a .50 chance against each other, and the 16th has a .52 chance against all of them, what tournament system would be most likely to identify #16 as the best player? And my position is, under any tournament, #16's odds are pretty poor, so we shouldn't change the tournament on that basis.

EDIT: And hence an 7-1 or 8-4 score seems somewhat less impressive. It looks more to me like a 4-0 and a 3-1

Well, you can't have it both ways. Either the best source of information about the relative likelihood of one player to beat another player is more games played between those two players (which is what I thought we had agreed on), in which case more games played between the top contenders is much more informative than a handful of games played between top contenders and the bottom half of the field; or Go has magical stylistic properties that make ranking players meaningless, and thus (as Tapir comes close to saying) the pro test is basically an entertaining lottery.

(I'm not saying that it has to be 100% ordinal ranking or 100% entertaining lottery, but as we move towards the "entertaining lottery" end of the spectrum, that doesn't mean that we need a different tournament design to find the best player, it means that no best player exists.)

I'm not positive I agree with that approach, since the top players could have scored a number of their victories against those weaker players, and other players will then have to work harder to get the same score (because those weaker players won't be around to score wins off).

The only way that would make sense to me would be to eliminate their existence in the tournament when they're eliminated. Basically you subtract wins against them from everyone else's score.

I agree that it is neither fair to deny people freebie victories against the (dan-ranked!) noobs at the end of the tournament, nor to give people freebie victories against people who don't have skin in the game anymore; no tournament system is perfect. But if victories and losses against the freebies stop counting once a player is eliminated, that means that ultimately we get the system that I wanted: we decide that games between Andy and Gangshen are more informative than the first-round games. So at that point what we want is basically a multiple-elimination tournament, with more games per round as we go on.

Andy Liu is one of the strongest player in the NA. No question about it.
How about the second pro selected by this interesting pro-qualifier tournament?
Can anyone post his scores in the US Open and NAIM? Does he have a convincing result?