Life In 19x19 http://lifein19x19.com/ 

A little experiment with 400 games http://lifein19x19.com/viewtopic.php?f=18&t=16327 
Page 1 of 1 
Author:  Vargo [ Wed Dec 26, 2018 4:49 am ] 
Post subject:  A little experiment with 400 games 
I was very surprised to see that two 400 game matches could have a 6% difference in winrates (the winrates were 33% and 39%, with the same parameters) To be sure that the results of these two 400 game matches were within reasonable probabilistic limits, I set up a 100000 throws match of "Heads or Tails". The result comes in seconds, so, this "heads or tails" experiment can be run several times. Bottom line is : For a 100000 throws game, After 400 throws the winrate is anywhere between ~44% and ~56% After 1000 throws the winrate is usually between ~46% ~54% After 10000 throws, it's within 12 % of 50% and at 100000 throws, it's usually very close to 50% It's really a surprise to me that 400 games gives so little information ! Below, two graphs, in each , the 4 pictures describe the same game, after 400, 1000, 10000 and 100000 throws 
Author:  dfan [ Wed Dec 26, 2018 5:10 am ] 
Post subject:  Re: A little experiment with 400 games 
The variance of a binomial distribution (summing up the results of n trials where each one has a probability of success of p) is np(1p), and the standard deviation is the square root of that. 68% of results lie within one standard deviation of the mean and 95% of results lie within two. For this case, that means that if we assume that the engines are exactly equal in strength, the standard deviation of a 400game experiment is sqrt(400 * .5 * .5) = 10. So we expect 68% of our experiments to end with a result in the range of 200 ± 10 wins for the first engine (47.5% to 52.5% win rate) and 95% to end with 200 ± 20 wins (45% to 55%). 
Author:  lightvector [ Wed Dec 26, 2018 6:42 am ] 
Post subject:  Re: A little experiment with 400 games 
Yep, the fact that you need so many games to reliably test strength differences (unless they're extreme) becomes painfully apparent if you're a bot developer. 
Author:  Vargo [ Wed Dec 26, 2018 7:00 am ] 
Post subject:  Re: A little experiment with 400 games 
dfan wrote: So we expect 68% of our experiments to end with a result in the range of 200 ± 10 wins for the first engine (47.5% to 52.5% win rate) You're right, and it also means that when you run 400 game matches between evenly matched contestants, you'll get false winrates by 3% or more (...46%, 47% or 53%, 54%...) in ~30% of the matches ! That's a lot... 
Author:  Mike Novack [ Wed Dec 26, 2018 7:44 am ] 
Post subject:  Re: A little experiment with 400 games 
But back to Vargo's original question. What has been shown for the binomial is for the special case where the p is 0.5 (the engines are actually of equal strength) BUT  the whole purpose of the experiment was to determine if one engine was in fact stronger than the other (p NOT 0.5). Here the results were 0.39 and 0.33 and the question being asked was the difference between these meaningful. Try doing the expansion again with a value for p something like p = 0.36 and then see if results of carrying out the experiment of 0.39 and 0.33 are unlikely or not. 
Author:  dfan [ Wed Dec 26, 2018 10:21 am ] 
Post subject:  Re: A little experiment with 400 games 
Yes, I was just providing the analysis to the experiment described in this thread, partially because the full parameters to that experiment were known (the coin was fair) and the parameters to the original experiment are not (we don't know what the "real" strength difference between the two engines is). That said, if we assume a coin that comes up heads 36%, of the time, the standard deviation of a 400game result is now 9.6 instead of 10, which means that ~68% of the time the result will lie between .36 * 400  9.6 and .36 * 400 + 9.6 wins, which comes out to a winning percentage range of 33.6% to 38.4%. 95% of the time the observed result of a 400game match will lie within 31.2% and 40.8%. 
Author:  mhlepore [ Wed Dec 26, 2018 6:43 pm ] 
Post subject:  Re: A little experiment with 400 games 
I think in the case of two runs of 400 flips, with one giving 33% heads and one giving 39% heads, it makes sense to to a straightforward test of proportions. The variance would get pooled. 
Author:  moha [ Thu Dec 27, 2018 10:43 am ] 
Post subject:  Re: A little experiment with 400 games 
One should be very careful when calculating expected deviation, particularly because these simple figures assume sample independence, which is not necessarily true. For example the first few moves and choices of openings / first josekis can have large effect on the outcome, and these are often identical within the sample set. So you may not even getting 400 effective games worth of samples, which further increases variance. 
Page 1 of 1  All times are UTC  8 hours [ DST ] 
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ 