It is currently Tue Apr 30, 2024 1:30 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 4 posts ] 
Author Message
Offline
 Post subject: EGD machine friendly access
Post #1 Posted: Sun Oct 04, 2015 12:34 am 
Dies with sente

Posts: 94
Liked others: 6
Was liked: 4
Hi,
I am a programming newbie so sorry if I don't get the obvious etc. I would like to play with the European database data a bit but cannot find a good way to extract tournament wall lists. Is there some way I can querry the database in the fashion "List tournament codes of all tournaments which ..." and "Send just the clean tournament wall list for the tournament code xxxxxxxx"?
Thanks in advance

_________________
“Discussion is an exchange of knowledge; argument an exchange of ignorance.” ― Robert Quillen

Top
 Profile  
 
Offline
 Post subject: Re: EGD machine friendly access
Post #2 Posted: Sun Oct 04, 2015 2:32 am 
Gosei
User avatar

Posts: 2011
Location: Groningen, NL
Liked others: 202
Was liked: 1087
Rank: Dutch 4D
GD Posts: 645
Universal go server handle: herminator
Have you tried emailing the EGD developers? They have an email address for questions listed at the bottom of the website.

Top
 Profile  
 
Offline
 Post subject: Re: EGD machine friendly access
Post #3 Posted: Sun Oct 04, 2015 7:15 am 
Lives in sente

Posts: 1037
Liked others: 0
Was liked: 181
tiger314 wrote:
Hi,
I am a programming newbie so sorry if I don't get the obvious etc. I would like to play with the European database data a bit but cannot find a good way to extract tournament wall lists. Is there some way I can querry the database in the fashion "List tournament codes of all tournaments which ..." and "Send just the clean tournament wall list for the tournament code xxxxxxxx"?
Thanks in advance


Since you say you are a "newbie" let me point something out.

You have to think about the behavior of the application you are trying to design. If it will be making only a few queries against the database, go after it normally. But if you think it might be making a lot of them (more than a couple percent of the database) it could be MUCH faster to first get a (flat file) copy of the backup of the database and extract your data en masse.

CONTACT THOSE IN CHARGE OF THIS DATABASE.

For testing, at least, you will want to be using YOUR OWN COPY of this database, or perhaps they can point you to a test copy of theirs. You DON'T test against "production", at least not in the early stages of development.

BTW -- the sort of goof using "direct access" when the volume meant you should have used "batch access" is the sort of thing "but it worked fine in testing" <<because in testing, your volume was tiny>> Emergency rewriting of a new application because the developers made this goof and when they went into production brought the database to it's knees (because of a huge increase in access transactions) one of the things I used to do.

Top
 Profile  
 
Offline
 Post subject: Re: EGD machine friendly access
Post #4 Posted: Sun Oct 04, 2015 4:51 pm 
Tengen

Posts: 4380
Location: North Carolina
Liked others: 499
Was liked: 733
Rank: AGA 3k
GD Posts: 65
OGS: Hyperpape 4k
Mike Novack wrote:
For testing, at least, you will want to be using YOUR OWN COPY of this database, or perhaps they can point you to a test copy of theirs. You DON'T test against "production", at least not in the early stages of development.

BTW -- the sort of goof using "direct access" when the volume meant you should have used "batch access" is the sort of thing "but it worked fine in testing" <<because in testing, your volume was tiny>> Emergency rewriting of a new application because the developers made this goof and when they went into production brought the database to it's knees (because of a huge increase in access transactions) one of the things I used to do.
Also, make sure to discuss the changes with all stakeholders, get risk assessments, six sigma certification, and dedicated test personnel. Create rollback plans, in case deployment goes badly, and hire several consultants.

Jiminy cricket.

Seriously, though:
1. Ask for a dump of the database, ideally in a textual format (good suggestion, Mike).
2. Probably don't get it.
3. Write some python to scrape the html (I'm not sure what people use for scraping, but I know some people swear by beautiful soup for parsing HTML). Be diligent about rate limiting your access!

So to start:

1. Install python.
2. Install pip.
3. Install beautiful soup.
4. Starting figuring out how to use beautiful soup to get the data you need.

_________________
Occupy Babel!


This post by hyperpape was liked by: oren
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group