Life In 19x19

Go4Go.net has recently survived another big round of cyberattack from spambots. At the peak of the attack, almost 95% of my bandwidth was wasted by these robot programs trying to gain access to the server. And my host threatened to shut down the site. While the situation is under control now, I had to spend many many hours implementing better defence system. I'd rather use the time to create more contents for the Go community.
While the spam activities are mostly out of out controls, from time to time there are incidents where intelligent Go players attempting to misuse/abuse the website. In a recent case, I was forced to block an IP address because tens of thousands of requests per hour from it (to mirror the website?) overwhelmed the server. This guy was very smart that in less than an hour he modified his script to direct the attack via a free proxy server (which abused the proxy service as well)!

So here is a plea to the Go community: please take good care of our content providers, who dedicate time and money to promote Go, often with little or no financial gains.

In my case, if you are interested in the Go4Go database, a polite email explaining your purpose is often sufficient for me to send you a copy of my entire database.

Thanks for your understanding!

macelee wrote:This guy was very smart that in less than an hour he modified his script to direct the attack via a free proxy server (which abused the proxy service as well)!

Hi macelee, sorry to hear that. It's always these $@#%?&!* guys who spoil it for everyone!

Although I agree with your sentiment, and completely agree that automatic crawlers should never be misused (I have written quite a few, and almost in all cases I set hard limits to keep the traffic to an "human-clickable" level, so that the automated traffic could come from a human and thus not overwhelm a bad configured or underpowered server,) here there's an issue of general web security. If a website doesn't have enough security measures, *it will be attacked*. Actually, even if it has, it will be attacked.

The web servers of the company I work at, get periodic DDOS attacks, some more severe, some less. Occasionally our email servers have a spam-rise that blocks email queues. Hard drives fail. Every time something gets fscked off, another layer of security is added (automatic firewalls, CDNs, newer filters, more redundancy.) But in the end, it boils down to "it's out there, with open ports: be prepared."

Addendum: your hosting provider seems to offer unlimited bandwidth per month. This is probably oh-so-wrong from them... Neither money grows on trees nor unlimited bandwidth exists.

RBerenguel wrote:Addendum: your hosting provider seems to offer unlimited bandwidth per month. This is probably oh-so-wrong from them... Neither money grows on trees nor unlimited bandwidth exists.

'unlimited bandwidth' is just a marketing trick. In practice, those hosts offering unlimited bandwidth often impose other limits, such as memory limit or percentage of CPU (in particular on shared virtual servers). So I agree with you there's no such thing as unlimited.

macelee wrote:

RBerenguel wrote:Addendum: your hosting provider seems to offer unlimited bandwidth per month. This is probably oh-so-wrong from them... Neither money grows on trees nor unlimited bandwidth exists.

'unlimited bandwidth' is just a marketing trick. In practice, those hosts offering unlimited bandwidth often impose other limits, such as memory limit or percentage of CPU (in particular on shared virtual servers). So I agree with you there's no such thing as unlimited.

Indeed: selling unlimited bandwidth and hiding "only 256 MB of RAM available" (for example) would be a neat trick. Try to serve more than 10 concurrent users with MySQL, Apache 2 and just 256 MB (I don't think a Varnish caching could fit in there, anyway)

RBerenguel wrote:Although I agree with your sentiment, and completely agree that automatic crawlers should never be misused (I have written quite a few, and almost in all cases I set hard limits to keep the traffic to an "human-clickable" level, so that the automated traffic could come from a human and thus not overwhelm a bad configured or underpowered server,) here there's an issue of general web security. If a website doesn't have enough security measures, *it will be attacked*. Actually, even if it has, it will be attacked.

The web servers of the company I work at, get periodic DDOS attacks, some more severe, some less. Occasionally our email servers have a spam-rise that blocks email queues. Hard drives fail. Every time something gets fscked off, another layer of security is added (automatic firewalls, CDNs, newer filters, more redundancy.) But in the end, it boils down to "it's out there, with open ports: be prepared."

This, a thousand times this. Everyone I know working in this business says the same thing: you will be attacked, get used to it. It's a horrible reality for people doing things out of their own pocket or trying their best to keep a site free to use.

IMO, the content providers can spare themselves a little headache by providing the content in a bulk downloadable package. I've seen this with ncbi's data stores, grad students that only know Perl or Python aren't intentionally taking down the server, they just don't know or think about what resources are being used, so they write some naive queries against a poorly documented/designed web server database, because the data are so poorly documented in the bulk download. Though, you may not be interested in making the data easily bulk downloaded, but then that is your responsibility manage. Also, in my opinion web providers, providing free content, don't necessarily deserve a white hat, since they are competing with those who are trying to make money. IE if the server weren't giving their content away for the price of ads, there might be a market for the content. This is one of the things, I find really annoying about the Google/Open source/web server culture, they tend to burn the Content creators, it seems they want all information to be free, but paid for by ads, because they feel entitled by owning their hardware. I am not saying, spam attacks are right, they are definitely wrong, but it's part of the I have hardware therefore I can mentality.

I hate that this happened. Stink that people mess up good things by doing stupid stuff like this.

If OP is concerned about someone legitimately interested in his/her content ripping it all and overwhelming the server, perhaps offering a monthly or quarterly bulk package via torrent would help? That way you don't need to eat all the download bandwidth directly.

Drew wrote:If OP is concerned about someone legitimately interested in his/her content ripping it all and overwhelming the server, perhaps offering a monthly or quarterly bulk package via torrent would help? That way you don't need to eat all the download bandwidth directly.

Personally, I don't think macelee is asking for a lot. He even says:

In my case, if you are interested in the Go4Go database, a polite email explaining your purpose is often sufficient for me to send you a copy of my entire database.

The post is simply asking us not to abuse the service.

Yes, maybe he can take steps to make the site more secure, etc. But this is beside the request he's making of us.

I don't think it's an unreasonable request.

I don't think anyone thinks it's unreasonable. I think some of us believe that the people who need to hear that option aren't going to hear it, and that such a policy should be obvious on the site itself - via bulk download link, publishing torrents, or some other such distribution method.

As for the DoS attacks, that's life on the Net. Unfortunately it sounds like the hosting provider isn't very friendly.

Though, if you did happen to have an ad, on a site, that was being hit with thousands of http requests... nah, that would be fraud.

SmoothOper wrote:Though, if you did happen to have an ad, on a site, that was being hit with thousands of http requests... nah, that would be fraud.

Most ad serving platforms are more clever than just replying to an http response===increase ad count. AdSense is a particularly good example of this.

RBerenguel wrote: Most ad serving platforms are more clever than just replying to an http response===increase ad count. AdSense is a particularly good example of this.

That's true. I used to have AdSense. Unfortunately 'Go' just isn't a good keyword so the content of the Ads are often very irrelevant. To be fair to Google, it does a reasonable job. I often got board game type of ads on the homepage. Further inside the site, more often I got ads from dating sites trying to get you a girlfriend from those Go-playing countries