Mass downloader??? Help!

Bantari · **#21**

Pippen wrote:

I'd definitely support the idea to harvest 500.000 games or so from Tygem 9/8d's with a script that does not misuse or damage the server. Right now fuseki.info and http://ps.waltheri.net/ are the best available sites.

Is there any word from the owners of that database about all that? I haven't seen anybody mentioning that here.
Or do we really care what they think? Its there so lets take it, and if it breaks their website, boo-hoo their own fault?

Long live wild west?

longshanks · **#22**

Pippen wrote:

I'd definitely support the idea to harvest 500.000 games or so from Tygem 9/8d's with a script that does not misuse or damage the server. Right now fuseki.info and http://ps.waltheri.net/ are the best available sites.

I don't really know much about Tygem .. but..

The problem would be if you had to retreive 500k sgfs, and they don't provide a bulk download mechanism of any kind, that's 500k hits on their server. They're not going to be too happy about that! If not done with care this would be a DoS and could cause them a major outage. If you did one hit a minute (reasonable).. ~347 days to get the entire game set.

I guess the problem is that no one is selling a complete set? I have some fairly large sets that are available via apps I have on my iPad. None of them are complete but are supplementary.

sybob · **#23**

Not all technically available means are morally acceptable AND also legally accepted.

Publicly accessible does not mean unlimitedly downloadable.
Think Spotify: you can stream it, not download it.

Without evidence stating otherwise, one has to assume the owner of a database is also the owner of the information contained therein.
Think this forum: this forum is (more or less) publicly accessible, but it is probably legally prohibited to download it, either in whole or in part (remember, this forum is also just a database) and/or restricted by the terms of use.

If you want to have or use some other's ownership, property or means, you need to have subject's consent.
Also in this information day and age.

Kirby · **#24**

sybob wrote:

Not all technically available means are morally acceptable AND also legally accepted.

Publicly accessible does not mean unlimitedly downloadable.
Think Spotify: you can stream it, not download it.

Without evidence stating otherwise, one has to assume the owner of a database is also the owner of the information contained therein.
Think this forum: this forum is (more or less) publicly accessible, but it is probably legally prohibited to download it, either in whole or in part (remember, this forum is also just a database) and/or restricted by the terms of use.

If you want to have or use some other's ownership, property or means, you need to have subject's consent.
Also in this information day and age.

Tools like curl and wget are not illegal – just like using your web browser is not illegal. If someone makes a script to download from your site using one of these tools, you’ll be hard-pressed to file charges against them.

Like any other tool, wget and curl can become illegal when you use them in illegal ways. Legal issues might arise when you do things like:
• Violate terms of use agreements or ‘robots.txt’ instructions.
• Hit the site with repeated requests in a manner that might impact other users – this could be considered a DoS attack.

Writing a properly timed script to download files that you would otherwise be clicking to download in your browser should be equivalent if you’re not causing harm to the bandwidth of the site, especially if you’re not violating any sort of TOS agreement.

Please note that I’m not a lawyer, and this is my opinion. But I think it’s probably correct.

sybob · **#25**

Kirby wrote:

Tools like curl and wget are not illegal – just like using your web browser is not illegal. If someone makes a script to download from your site using one of these tools, you’ll be hard-pressed to file charges against them.

Like any other tool, wget and curl can become illegal when you use them in illegal ways. Legal issues might arise when you do things like:
• Violate terms of use agreements or ‘robots.txt’ instructions.
• Hit the site with repeated requests in a manner that might impact other users – this could be considered a DoS attack.

Writing a properly timed script to download files that you would otherwise be clicking to download in your browser should be equivalent if you’re not causing harm to the bandwidth of the site, especially IF (edit/emphasis changed: sybob) you’re not violating any sort of TOS agreement.

Please note that I’m not a lawyer, and this is my opinion. But I think it’s probably correct.

So, we almost agree?
(see edit in above quote)
I think one should (try to be) correct, not just "probably correct".
Same as in a go game ;-)

Tools, instruments etc. in itself are almost never illegal. But ownership and use are limited.
A hammer is a useful tool. But it illegal to smash someone's head in with it. (I'm not a lawyer, but I suppose it works this way).
The internet is not illegal. But it can be used for inappropriate or even illegal purposes.
Nuclear devices and WoMD are not illegal per se. But in the hands of certain people, they are. Making use of them will probably also be considered illegal, both under national and international law.
So, I think it is wise to give consideration to certain uses of tools, even more so if you do not consult or get consent from the owner(s).

Kirby · **#26**

sybob wrote:

I think one should (try to be) correct, not just "probably correct".
Same as in a go game ;-)

I am trying to be correct. I'm just providing a disclaimer that there is chance I might be wrong - just like in my go games.

sybob wrote:

So, I think it is wise to give consideration to certain uses of tools, even more so if you do not consult or get consent from the owner(s).

I agree that you should give consideration to use of tools. But to me, if the website owner makes information publicly available for download, does not specify restriction in a 'robots.txt' file, does not provide a ToS indicating that you shouldn't use a script to download, and makes no technical attempt to stop you from using such tools...

I don't feel bad about using a script to download without explicitly asking them. I'm not harming their site, and I'm not violating them in any way.

I don't ask them if I can use Chrome to visit their website, either. I don't think it's a problem.

Pippen · **#27**

In fact this could be a nice wiki-project of some kind. If some people program harvesters not too big and invasive and then collect the data one could easily build a huge database by a soft approach. Tygem has a lot of 8d or 9d players with more than 5.000 games. One would just need to write down 100 of these players and then use a harvester just for one player at a time and you'd have 500.000 games together soon. I'd have tried it a long time ago, but I am not a programmer and I cannot afford the time to learn how to program those software.

skydyr · **#28**

Pippen wrote:

In fact this could be a nice wiki-project of some kind. If some people program harvesters not too big and invasive and then collect the data one could easily build a huge database by a soft approach. Tygem has a lot of 8d or 9d players with more than 5.000 games. One would just need to write down 100 of these players and then use a harvester just for one player at a time and you'd have 500.000 games together soon. I'd have tried it a long time ago, but I am not a programmer and I cannot afford the time to learn how to program those software.

You'll undoubtedly have duplicates in your games, because players have to play each other.

Mass downloader??? Help!

Who is online