It is currently Tue May 11, 2021 9:35 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 118 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #81 Posted: Tue Feb 09, 2021 4:01 am 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
@akigo many thanks!!!

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #82 Posted: Wed Feb 10, 2021 5:09 am 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
BadukAI v1.4 40b(original version s604, maxPlayouts 1, numSearchThreads 1) - Zenith 7 9d, komi 0, 2:2
BadukAI v1.4 40b(optimized version s604, maxPlayouts 1, numSearchThreads 1) - Zenith 7 9d, komi 0, 1:3
all games BadukAI plays white

BadukAI (optimized version) - Zenith:


Attachments:
BadukAI - Zenith.sgf [1.38 KiB]
Downloaded 736 times


Last edited by And on Wed Feb 10, 2021 10:49 am, edited 2 times in total.

This post by And was liked by: go4thewin
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #83 Posted: Wed Feb 10, 2021 9:55 am 
Beginner

Posts: 2
Liked others: 0
Was liked: 2
Rank: KGS 3 dan
Katago s616 70 visits, 1 playout, NumberOfSearchThread: 1 vs Golaxy Lion two star(3700 elo rating): 1-0.
Incredible power of the strongest network.


Attachments:
自由对弈_星阵VSBaoTran_2021-02-10.sgf [732 Bytes]
Downloaded 726 times

This post by VietGo was liked by: Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #84 Posted: Wed Feb 10, 2021 10:27 am 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
BadukAI v1.5 40b(optimized version s604, maxPlayouts 1, numSearchThreads 1) - Zenith 7 9d, komi 0, 3:1
all games BadukAI plays white






Attachments:
BadukAI - Zenith.sgf [1.29 KiB]
Downloaded 736 times
BadukAI - Zenith 2.sgf [1.58 KiB]
Downloaded 742 times
BadukAI - Zenith 3.sgf [1.49 KiB]
Downloaded 749 times
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #85 Posted: Wed Feb 10, 2021 10:59 am 
Lives with ko

Posts: 143
Liked others: 190
Was liked: 29
Rank: 25 kyu
Wow, so am i understanding correctly that out of 8 games with no komi, the optimized net won 4? so with 1 playout it is stronger than zen 7 9d. That is really incredible for such a fast net. It really makes katago very mobile friendly to play against, especially with the kyu_rank option. I think the non-optimized net was too big and slow for some very older or underpowered phones. Thanks for the tests!


This post by go4thewin was liked by: Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #86 Posted: Wed Feb 10, 2021 12:30 pm 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
go4thewin wrote:
Wow, so am i understanding correctly that out of 8 games with no komi, the optimized net won 4?

Yes

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #87 Posted: Thu Feb 11, 2021 9:22 am 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
BadukAI v1.5 40b(optimized version s604, maxPlayouts 1, numSearchThreads 1) - CS Zero 9d, komi 0, 0:5
all games BadukAI plays white
BadukAI resigned on move 74, 112, 128, 74, 96

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #88 Posted: Thu Feb 11, 2021 9:35 am 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
BadukAI v1.5 40b(optimized version s604, maxPlayouts 1, numSearchThreads 1) - CS Pro 5d(android), H5


Attachments:
BadukAI - CS Pro.sgf [3.63 KiB]
Downloaded 689 times

This post by And was liked by: Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #89 Posted: Sun Feb 14, 2021 10:17 am 
Lives with ko

Posts: 143
Liked others: 190
Was liked: 29
Rank: 25 kyu
VietGo wrote:
Katago s616 70 visits, 1 playout, NumberOfSearchThread: 1 vs Golaxy Lion two star(3700 elo rating): 1-0.
Incredible power of the strongest network.

@Vietgo you may like this. Kata s634 1 playout beat galaxy cow 9d


newer s640 1 playout (sgf not uploaded) may be stronger than golaxy cow 9d, won both as black and white: 2-0


Attachments:
s634VSgolaxy9d.sgf [1.35 KiB]
Downloaded 626 times


Last edited by go4thewin on Wed Feb 17, 2021 8:34 am, edited 2 times in total.

This post by go4thewin was liked by: Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #90 Posted: Mon Feb 15, 2021 7:51 am 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
Is it possible to estimate the limit according to this schedule?
https://katagotraining.org By Data Rows (linear)
or calculate its value for example for 10G? (of course the relative values are interesting)

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #91 Posted: Mon Feb 15, 2021 3:02 pm 
Lives in gote

Posts: 660
Liked others: 108
Was liked: 800
Rank: maybe 2d
If you're going to calculate any values or do any estimation, the log scale is the proper way to do it, not the linear scale.

In general across a wide variety of games, to first order Elo grows logarithmically with the amount of learning, or the amount of computation time invested per move, or the size of a model or a pattern or feature database, etc. If it makes it more intuitive for you, you could also mentally conceptualize it as there being some underlying amount of "computation" (where learned knowledge is like a cached form of computation too) that scales linearly with all these things, and then your Elo is just the logarithm of that value.

So if you are going to extrapolate anything based on the existing data, the correct relationship to start with to try to extrapolate would be Elo versus log(data), not Elo versus data.


This post by lightvector was liked by 2 people: And, Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #92 Posted: Wed Feb 17, 2021 12:21 pm 
Lives with ko

Posts: 143
Location: Pasadena, USA
Liked others: 59
Was liked: 9
Rank: OGS 11 kyu
OGS: Maharani
spook wrote:
@Maharani,
Right now, it's just the latest.

I use an API to query the list of networks. And I wasn't really sure how to determine which one's the "strongest confidently rated" network.
If I would have to make an educated guess, it would be the latest network with "log_gamma_uncertainty" < 0.05.
Maybe somebody reading this, can confirm or deny. :)


From this thread: https://lifein19x19.com/viewtopic.php?p=263568#p263568

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #93 Posted: Sun Feb 21, 2021 7:20 am 
Lives in gote

Posts: 660
Liked others: 108
Was liked: 800
Rank: maybe 2d
Strongest confidently rated network filters down to networks whose standard deviation of Elo uncertainty is less than 100... which is almost all of them, this usually just excludes a new net that barely has any data yet... and picks the net with the highest (mean - 2 * standard deviation).


This post by lightvector was liked by: Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #94 Posted: Sat Feb 27, 2021 12:35 am 
Oza
User avatar

Posts: 2341
Location: Tokyo, Japan
Liked others: 2244
Was liked: 1317
Rank: Jp 6 dan
KGS: ez4u
It seems to me that the " Strongest confidently-rated network": kata1-b40c256-s6485784576-d1573360039 declined significantly as a result of recent rating games. As of this post it is showing "13226.9 ± 12.4 - (3,292 games)". However, just a couple of days ago it was above 13250 [+/-??? (I don't remember)] based on 2,000 games or so. At that time it appeared as a clear anomaly in the rating graph on the project page as the only net over 13,250.

Is there anywhere that we can see the historical development of each net's rating?
Do we need a new definition of "confidently-rated"? :)

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #95 Posted: Sat Feb 27, 2021 7:39 am 
Lives in gote

Posts: 660
Liked others: 108
Was liked: 800
Rank: maybe 2d
No, we don't need a new definition. :)

Let's clarify what the purpose of this selection metric is - to pick a reliably good network with high confidence:

Even if the gap is smaller now, "kata1-b40c256-s6485784576-d1573360039" is still rated as being among the strongest all nets before and nearby it, right?

And also the error bar on that net is small, so even it turns out not to be the strongest, with very high likelihood it's not one of the nets which performs unusually poorly, right?

So in both ways, the selection criterion is doing its job well. With decent reliability, it picks out a recent and strongest or nearly-strongest net compared to its neighbors, and with high reliability avoids really bad nets, despite major uncertainty in the ratings relative to the magnitude of differences it's attempting to discriminate between.

Additionally, keep in mind about Elo values in general:

In general, across almost all Elo systems, pay more attention to Elo differences, than to absolute Elo numbers.. This is true in general for Elo systems, except for perhaps ones that take extreme pains to maintain stability across time. You can see how Go server and association ratings are all over the place relative to one another, as well as sometimes having inflation or deflation over time. In Chess world things are more stable, but still there is sometimes a little bit of noise or drift, and mild inconsistency between systems. And every different published research paper also uses an Elo scale whose absolute offset is incomparable to that of any other paper. In all cases, the differences are more meaningful than the absolute numbers.

In KataGo's case, the anchor point of the graph right now is arbitrarily chosen as 0 = random, and new rating games are played all the time even between very old nets. If back when KataGo was moving through "DDK" level, new-games indicate that over a span of some nets only 2000 Elo was gained instead of 2050, the entire rating graph above it will shift by 50 Elo, even though nothing practical has changed about our belief of the strength of the current nets. So the absolute number, really, really doesn't matter here.

And, a note about Elo locality:
Even more than ignoring absolutes and paying attention to just differences, in any Elo system you ever find in practice, you usually should only consider the local differences reliable - the ratings difference between a player and other players near them. For larger differences, they are the transitive sum of smaller differences, rather than directly measured. So when P1 is 1150 Elo better than P2 in *any* practical system (not just KataGo), that should be understood to mean something like:

"P1 is measured to approximately win 3:1 against players who win 3:1 against players who win 3:1... against P2", in total iterated 6 times.

It does NOT mean:

"P1 is measured to approximately win 750:1 against P2".

Because in practice, no Elo system will have the games to measure that accurately. Plus, we know that Elo itself is only an approximation of reality. In truth "skill level" is more complex and multidimensional, and precisely one of the places that approximation starts becoming unreliable is in very large differences. So that means that the interpretation of the vertical confidence bands in KataGo's rating graph is a bit subtle. The confidence bands around the nets should be understood to be confidence bands with respect to the Elos of the population of nets around it, say, within the nearby +/ 300 Elo or so. If the local population as a whole moves up or down by more, it doesn't matter.


This post by lightvector was liked by 3 people: Bill Spight, Harleqin, Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #96 Posted: Sat Feb 27, 2021 7:52 am 
Lives in gote

Posts: 660
Liked others: 108
Was liked: 800
Rank: maybe 2d
Or, if you want the TLDR:

Only the Elo differences between nets are meaningful. If tomorrow I were to reanchor or recalibrate the graph and it causes everyone's Elo to shift by 1000 points, but all the differences between the latest nets stay about the same, it does not mean something like 13200+/-25 was wrong by 1000 points. Because the absolute Elos are not meaningful, only the Elo differences.


This post by lightvector was liked by 4 people: gennan, Harleqin, johnsmith, Maharani
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #97 Posted: Sat Feb 27, 2021 8:58 am 
Honinbo

Posts: 10722
Liked others: 3612
Was liked: 3351
lightvector wrote:
Because in practice, no Elo system will have the games to measure that accurately. Plus, we know that Elo itself is only an approximation of reality. In truth "skill level" is more complex and multidimensional, and precisely one of the places that approximation starts becoming unreliable is in very large differences. So that means that the interpretation of the vertical confidence bands in KataGo's rating graph is a bit subtle. The confidence bands around the nets should be understood to be confidence bands with respect to the Elos of the population of nets around it, say, within the nearby +/ 300 Elo or so. If the local population as a whole moves up or down by more, it doesn't matter.
(Emphasis mine.)

Because skill level is multidimensional, it is only partially ordered. Hence, cases where player A usually beats player B, who usually beats player C, who usually beats player A are not uncommon. And because of shared history, it is not unusual for groups of players to have similar strengths and weaknesses. To some extent a player's rating will thus depend upon who they play against.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #98 Posted: Mon Mar 01, 2021 8:32 am 
Dies in gote

Posts: 46
Liked others: 2
Was liked: 12
It comes to mind when Remi Coulom started Goratings.org, there was someone who commented about how they found it wrong that the ratings of the top go players were lower than those of top chess players.

After some discussion about why this didn't make sense, I don't know if it was on twitter or some post here, Remi then replied (sarcastically I suppose) that he adjusted all ratings upwards by 1000 points to 'show the superiority of go players over chess players'

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #99 Posted: Mon Mar 08, 2021 10:16 am 
Gosei

Posts: 1501
Liked others: 453
Was liked: 421
Rank: 5d
GD Posts: 1000
It seems to me that one way to compare ratings between chess and go would be to look at the histograms of the two games and match up the percentile levels. Of course that is assuming that the distributions of the ratings of the two games are similar, but on the face of it I would expect the distributions should be similar. There would be some technical issues, such as that the world Elo chess ratings include the pro grandmasters while go ratings such as AGA or European amateur associations do not include pros.

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #100 Posted: Sun Mar 14, 2021 11:45 am 
Lives in sente
User avatar

Posts: 755
Liked others: 153
Was liked: 134
New b60 net https://media.katagotraining.org/upload ... 314.bin.gz

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 118 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group