Life In 19x19
http://lifein19x19.com/

“Decision: case of using computer assistance in League A”
http://lifein19x19.com/viewtopic.php?f=10&t=15538		 Page 32 of 36
Author:  Bojanic [ Mon Jun 18, 2018 10:32 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Bill Spight wrote:
Actually, focusing on the top choice is the most sensitive measure of agreement with Leela. Adding other choices makes the statistics look impressive, but also makes them less sensitive.

Edit: Yes, I know that Frejlak is not making a statistical argument. :)

This sensitivity might be a problem if we compare it to nothing. If we compare it to other game in same criteria, it is not.

Also, focusing just on A and numbers has another drawback - A is most common forced move - in one analyzed game there was more than 20% of such moves.

Edit: I also did not make statistical argument.
Author:  Gobang [ Mon Jun 18, 2018 10:37 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A
RobertJasiek wrote:
Gobang wrote:
the difference between 4d and 6d+ must be as clear as day and night


As I have explained in https://lifein19x19.com/forum/viewtopic ... 62#p232862 , it is not clear at all for only one game or a few games. It is only clear for many played games.


I don't agree, and I am old enough to know that something is not necessarily true, just because someone thinks he has "explained" it.
Author:  jlt [ Mon Jun 18, 2018 10:55 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Winning statistics are available on the website http://www.europeangodatabase.eu/EGD/winning_stats.php

Between Jan. 2003 and June 2018,

between a 1d and a 5d, the 1d won 6.5 % of the time.
between a 2d and a 6d, the 2d won 3.7 % of the time.
between a 3d and a 7d, the 3d won 1.5 % of the time.
Author:  Uberdude [ Mon Jun 18, 2018 10:56 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A
There is an easy way to test the claim of people who say it is easy to distinguish a 4d and 6d. Someone picks some 4/6d games, removes identifying information and shows them to these people. They then have to say which players were 4d and which 6d. How many correct will they get. This thread could do with less talking and more testing.

maf's site https://kyudan.net is a good way to demonstrate your skill (or lack thereof) of judging player ranks from play. I look forward to seeing some familiar usernames at the top of the leaderboard with average rank guess error less than 2.

Edit: I made this test at viewtopic.php?f=10&t=15832
Author:  Bill Spight [ Mon Jun 18, 2018 11:30 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Bojanic wrote:
Bill Spight wrote:
Actually, focusing on the top choice is the most sensitive measure of agreement with Leela. Adding other choices makes the statistics look impressive, but also makes them less sensitive.

Edit: Yes, I know that Frejlak is not making a statistical argument. :)

This sensitivity might be a problem if we compare it to nothing. If we compare it to other game in same criteria, it is not.

Also, focusing just on A and numbers has another drawback - A is most common forced move - in one analyzed game there was more than 20% of such moves.


Actually, that's a plus, not a drawback. If the move is forced, the player and Leela should almost always agree. Counting B or C as a match when the move is forced is problematic.
Author:  Bojanic [ Mon Jun 18, 2018 11:45 pm ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Bill,
I think that in Metta-Kim game is a B forced move. Player can answer on two similar places.
Author:  Charlie [ Tue Jun 19, 2018 12:42 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
jlt wrote:
Winning statistics are available on the website http://www.europeangodatabase.eu/EGD/winning_stats.php

Between Jan. 2003 and June 2018...


To save everyone, here, the effort of the long query-time over at the EGD:

Out of 4505 even games in the same time period, a weaker 4-dan player defeated their stronger 6-dan opponent 14.4% of the time.

Anyone who thinks that 14.4% is "rare" or argues that a sample of 4505 data is anything but statistically significant needs to encounter a clue at high velocity. I suppose one could argue that the data are skewed or biased or downright erroneous but doing so would constitute an own-goal: it would render the ranks meaningless and the whole question of whether a 4-dan victory against a 6-dan opponent is exceptional or not would become moot.
Author:  Uberdude [ Tue Jun 19, 2018 1:00 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Charlie, I think it is fair to point out that if a 4d plays a 6d in a McMahon tournament then (with some dependence on the level of the bar) there is a bias in that the 4d was more likely in good form and the 6d poor form for them to be drawn together. So maybe an average 4d beats an average 6d only 10% not 14% but I agree neither of those are rare.
Author:  Charlie [ Tue Jun 19, 2018 1:24 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Uberdude wrote:
Charlie, I think it is fair to point out that if a 4d plays a 6d in a McMahon tournament then (with some dependence on the level of the bar) there is a bias in that the 4d was more likely in good form and the 6d poor form for them to be drawn together. So maybe an average 4d beats an average 6d only 10% not 14% but I agree neither of those are rare.


You're right about the bias induced by McMahon but, without digressing too far down this tangent, I'd still argue that 14.4% is correct because our definitions of "4-dan" and "6-dan" are based on the same data -- mostly from McMahon tournaments, surely. In fact, the McMahon draws would also be based on that data.

Perhaps it is true that a "true 4-dan" defeats a "true 6-dan" less than 14.4% of the time but to test that hypothesis, we would have to record decades of data from tournaments that use a mode that does not induce a bias. Within such an "ideal" system, however, both Metta and Ben-David may end up with different ranks.
Author:  Uberdude [ Tue Jun 19, 2018 2:20 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
On the "It's easy to tell a 4 dan from a 6 dan" issue, I made a little test here, can you figure out the players' ranks? viewtopic.php?f=10&t=15832.
Author:  Javaness2 [ Tue Jun 19, 2018 2:40 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Lukan wrote:

Javaness (#638):
Yes, this discussion is running in circles a lot, I have already expressed that like 400 posts ago. Especially many useless stats and numbers are being presented here again and again. Shouldn't we keep this mainly as a Go-case and listen more to strong players' Go-feelings? I feel like that is being mostly ignored here.


I don't see much new in this thread for the last few days. I guess everyone made their mind up already. So now either:
  • stage 3 of the appeals process is invoked by Israel - via EGF Appeals(?) Committee, we already had stages 1 and 2.
  • UK team, or Chris Bryant directly, launches a new appeal based on suspicions of cheating with LeelaZero.
  • talk endlessly here until a revolutionary system to detect cheating appears. I don't think it will appear quickly.
Author:  Uberdude [ Tue Jun 19, 2018 2:43 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Lukan wrote:
Especially many useless stats and numbers are being presented here again and again. Shouldn't we keep this mainly as a Go-case and listen more to strong players' Go-feelings? I feel like that is being mostly ignored here.

I don't think stats are necessarily useless, they have some advantages (objective, can be automated and fast for many games), but they are difficult to develop and use well. I agree that analysis by strong players is a useful tool, particularly given our go stats analysis is just beginning and a baby compared to chess, but I think it's very important to make the test fair and blind: i.e. don't just show a strong player like yourself 3 of Carlo's online games and 3 of his offline and asks "Is there a big difference in the play? Do you think he cheated", but show you 3 games online and 3 offline each of players A, B, C and D (where one is Carlo and the others are random other people of similar strength with games from similar conditions) and then ask if you think any of them have the suspicious big difference. If you and say 2 other strong players all picked out the same blinded player as Carlo being the suspicious one then that is much stronger evidence. A problem is a lot of top European players probably followed the case so can recognise these games of Carlo, maybe we could ask some pros in Asia to be the judges.
Author:  Bojanic [ Tue Jun 19, 2018 2:50 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
I have read (again, this time in details) Italian appeal.
Google found it here:
http://frmor.net/downloads/statistical_ ... _David.pdf

My comments:
- as other statistical analysis, it is focused mainly on, ehrr, statistics, and less on go game.
- question of spent time and time for analysis in Leela:
I already wrote here on about it. Spent time in the game cannot be same as used in program. First it is necessary to manually input moves into Leela, and later to client. Player could be out of place for some reason, or distracted, and some time could be wasted. Also, sometimes spent time on analysis can be even longer than recorded, because analysis could be done in opponent's time.
- high percentage similarities of other player's games is not relevant. Those player's games should be compared to their live games.
- it is completely omitted that some moves could be forced, some expected, others could be important, and to analyze longer sequences of related moves - basically player's side of story
- Only positive thing is that it showed some of the lacking of original analysis, but it suffers from same lacking.
- two more recent live games were not analyzed, but hey were not played at the time.

Edit, most important: in this paper are listed two Metta's games from EGC 2017. I would be very interested to take a look at them.
Author:  AlesCieply [ Tue Jun 19, 2018 3:05 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Bojanic wrote:
I have read (again, this time in details) Italian appeal.
Google found it here:
http://frmor.net/downloads/statistical_ ... _David.pdf

This is only a part of the Italian appeal. The document submitted to the PGETC appeals committee contained more of it. Specifically, a section about the outstanding ethical profile of Carlo Metta is missing here and there are some other parts missing as well. I do not think it would be proper if I shared the complete document on this forum but would welcome if the Italians did, so anyone can make a judgement for himself.
Author:  AlesCieply [ Tue Jun 19, 2018 3:09 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Uberdude wrote:
On the "It's easy to tell a 4 dan from a 6 dan" issue, I made a little test here, can you figure out the players' ranks? https://lifein19x19.com/viewtopic.php?f=10&t=15832.

The problem with this kind of test is that it is not the same as playing a game and than judging the strength of your opponent. For me, it is also hard to tell whether the game was played by a 4d or 6d. I am a 1d player and I think I could more realistically estimate whether the player is 2k or 3d. Going away from my rank I guess my estimates become much less reliable.
Author:  AlesCieply [ Tue Jun 19, 2018 3:17 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Charlie wrote:
jlt wrote:
Out of 4505 even games in the same time period, a weaker 4-dan player defeated their stronger 6-dan opponent 14.4% of the time.

Is the statistics based on grades declared by players or on ratings? Is a player with GoR=2449 counted as 4d? How is the statistics affected by games when strong 4d beats weak 6d (this relates to the point raised by Uberdude)? It would be more proper to see statistics on how often the players with ratings between 2350 and 2450 beat opponents about 200 rating points higher.
Author:  Bill Spight [ Tue Jun 19, 2018 4:12 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Lukan wrote:
Shouldn't we keep this mainly as a Go-case and listen more to strong players' Go-feelings? I feel like that is being mostly ignored here.


I, for one, would be glad to do so if we had any evidence that strong go players were good at detecting cheating by their feelings. We have no such evidence. Unfortunately.
Author:  Uberdude [ Tue Jun 19, 2018 4:15 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
<moderator>
As several have noted this is a rather long/popular/confusing/messy/repetitive thread. To try to make it easier to follow I will split off the anonymous accuser discussion to another thread, and possibly other sub-threads too if they can be nicely extracted. And if you want to start a new thread on a sub-topic instead of cluttering this one please feel free!
</moderator>
Author:  Bill Spight [ Tue Jun 19, 2018 4:26 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Bojanic wrote:
Bill,
I think that in Metta-Kim game is a B forced move. Player can answer on two similar places.


As I said, there should almost always be a match with A.

But your point about forced moves vs. unforced moves gets us back to the point about which we agree, and which Regan also gets at. We need to distinguish the difficulty level of plays.
Author:  bugsti [ Tue Jun 19, 2018 5:27 am ]
Post subject:  Re: “Decision: case of using computer assistance in League A
Lukan wrote:
Shouldn't we keep this mainly as a Go-case and listen more to strong players' Go-feelings? I feel like that is being mostly ignored here.


Of course you were referring to Ke Jie's and Park Junghwan's Go feelings and not to those of laughably weak European "strong" players, right?
Page 32 of 36 All times are UTC - 8 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/