An SGF Collection (2)

[aeb]

Let me try a new thread, now that the previous one ended a bit sadly.
The past few days I added another 3000 games to the collection at
http://homepages.cwi.nl/~aeb/go/games/index.html

I wondered what fraction this is of the Japanese tournament games that
are available in SGF form.
Look at the Japanese tournaments where go4go has at least 100 games.

Code: Select all

tournament           go4go       aeb
Kisei                1294        1846
Meijin               2182        2841
Honinbo              2048        3094
Tengen               1211        2054
Oza                  1095        1616
Gosei                1310        1679
Judan                1745        2270
Shinjin O            564         678
NHK                  808         1705
Agon                 285         579
Ryusei               2330        2343
FHoninbo             303         440
FMeijin              320         883
FKisei               317         318
Kansai Kiin 1st      153         373

tot                  15965       22719

Gobase is asleep, I count 11268 games there (for these tournaments).
Remains the question about statistics for GoGoD.

Part of the required infrastructure was a list of pro names
with their rank development over time, see
http://homepages.cwi.nl/~aeb/go/misc/progression.html
Comments are welcome.

Maybe this more or less completes the Japanese tournament part.
It will take a nontrivial amount of time to do China/Korea in a similar way.

[macelee]

You might want to have a look at Gosei/Gos32/14.sgf. This game is not on Nihonkiin's Gosei page. On the other hand, there is a game in Kisei tournament having the same date and the same players. Please verify your source. If you rely on certain Asian servers' record, they mix up tournaments quite often.

[aeb]

You might want to have a look at Gosei/Gos32/14.sgf.

Thanks! Fixed.
Good that someone checks the details carefully.
(In fact this was on my todo list: I had 11 sources for this game, 6 called it Gosei and 5 called it Kisei.)

Now that I write anyway, I can update the sentence "Gobase is asleep". It woke up for a short while, and today it has 88642 games.

[macelee]

Shinjin-O/ShO36/P12.sgf and Shinjin-O/ShO36/P13.sgf are the same game. The former contains incorrect player name - should be removed.

In the same tournament Shinjin-O/ShO36/P6.sgf's kifu is broken. In fact it is the same game as P7.sgf. P6 should be removed.

[aeb]

P6.sgf and P12.sgf should be removed from ShO36.

Thanks again! Fixed.

[aeb]

I wondered what fraction this is of the Japanese tournament games that
are available in SGF form.

On http://homepages.cwi.nl/~aeb/go/games/index.html one can find
a freely available collection of Japanese professional go games
in SGF format.

Last time I published stats was when it had about 30000 games.
Today this collection has some 35000 games, mostly from Japanese
pro tournaments. There are a few older games as well.

I think this is the most extensive resource on Japanese games
but I may be unaware of recent or Asiatic sources.

An attempt to compare with gogod and go4go:

The Winter 2015 GoGoD collection has 86093 SGF files. There is no easy
way to sort them into Japanese / Chinese / Korean / other.
My collection has a bit over 10000 games from before 2016 not in GoGoD.
(JF, or anybody else, is of course free to take them.)

Go4Go reports 55831 games today. It is very uptodate, and always has
the latest games. It does not often add old games. Statistics of the
difference between aeb and go4go are similar to last time: go4go
has almost 18000 games in Japanese tournaments, and for these same
tournaments my collection has somewhat more than 24000 games.
The games missing in go4go are mostly older games.

[Bonobo]

Are the individual games which you posted there separately also included in the big archive?

[aeb]

Are the individual games which you posted there separately also included in the big archive?

Yes. The many smaller per-tournament archives are updated more frequently, but eventually all ends up in the single big archive.

[aeb]

..mix up tournaments..

A moment ago I sorted some Meijin games and noticed that some sources give
2016-06-15 Nishi Takenobu vs Yukawa Mitsuhisa
as 42nd Meijin, and other sources call it 43rd Tengen.
I think it should be Tengen, but go4go says Meijin. Am I mistaken?

[hyperpape]

Go4Go reports 55831 games today. It is very uptodate, and always has
the latest games. It does not often add old games. Statistics of the
difference between aeb and go4go are similar to last time: go4go
has almost 18000 games in Japanese tournaments, and for these same
tournaments my collection has somewhat more than 24000 games.
The games missing in go4go are mostly older games.

I understand everyone wishes to be very careful with their own database, but I look at this and see opportunities for collaboration.

[macelee]

..mix up tournaments..

A moment ago I sorted some Meijin games and noticed that some sources give
2016-06-15 Nishi Takenobu vs Yukawa Mitsuhisa
as 42nd Meijin, and other sources call it 43rd Tengen.
I think it should be Tengen, but go4go says Meijin. Am I mistaken?

You are right this is a Tengen game. My original source 新浪围棋 was wrong. This is a Kansai Kiin game - they do not keep tournament records as well as Nihon Kiin. They only get three static web pages covering tournament games from previous 3 weeks. These pages then get overwritten when new data is available. Luckily from about a year ago I started using a script to collect and save these pages. So I have an official record to check against.

[aeb]

I see opportunities for collaboration.

On the one hand, it is unclear how cooperation between an open source project and a commercial project should be organized.
On the other hand, to each the existence of the other is somewhat useful.
Let me take the opportunity to mention that more Ibero-Japan Cup games are available.

[hyperpape]

Your collection is public domain, so that by itself opens up some potential--I don't know whether you thought about the possibility of some other database incorporating yours, and whether you'd welcome it or regard it as a bit rude (the law isn't everything).

Anyway, I'm not trying to tell anyone what to do. That would be completely unreasonable, since you two are doing the legwork of creating those collections, not me.

[aeb]

Your collection is public domain--I don't know whether you thought about the possibility of some other database incorporating yours, and whether you'd welcome it or regard it as a bit rude

I would welcome it, of course. It happened several times, sometimes because I suggested it.

[aeb]

Today this collection has some 35000 games, mostly from Japanese pro tournaments.

That was half a year ago. The first published collection was on 2000-01-01 and had 1111 games, probably the only largish public domain collection. Yesterday the count passed 40000.