Hey everyone! As you've all certainly noticed by now, Life in 19x19 was down for about 11 days. The good news is that we're now back and (hopefully) better than ever!


What Happened

Without going into too much technical detail, there was some sort of quota that was hit on the database server hosted by GoDaddy. Jordus (the admin who owned the actual hosting account) was having trouble getting timely action from GoDaddy, so not much progress was being made. After about a week, Brian Kirby and I offered to move everything to my AWS account, where we would have complete access to both the hardware and software, instead of being at the mercy of GoDaddy's tech support. At first we were scraping together old backups from Kirby's hard drive and what was left of the FTP server (which were over a year old! ), but eventually Jordus saved the day and gave us a raw dump of the database just prior to the outage. After some scripting, I had the full site up on an EC2 instance (for phpBB/Apache) and RDS instance (for MySQL).

The Future
Obviously the length of this downtime re-emphasized the need to reduce our bus factor. There were a couple of us with admin access to the board, but only two that could directly connect to the database, only one that controlled the domain, and 0 of us that had root on the physical hardware everything ran on (thanks GoDaddy). Going forward, we're going to be:

• Posting the phpBB source, with all our theming and modifications, to a public GitHub organization, so anyone can clone it and help develop plugins and themes.
• Scheduling automated daily backups of the DB. (This is actually already done as of today)
• Make a sanitized (i.e, PMs and password hashes removed, etc) copy of this daily backup publicly available, as Linus Torvalds himself recommends
• Get a few more technically-savvy admins access to the AWS account this is running on. More on this soon.
• We're running on a downright ancient version of phpBB (3.0.8, from November 2010 !!). Once everything's calmed down, I've verified the backup procedures work, and the versioning to GitHub is complete, I'm going to run an upgrade to the latest phpBB 3.2. Security and performance are the main benefits. This should be complete by end of day Monday February 20th.

Are we missing anything? Are there any other points of failure the community wants to see plugged?

Known Issues
Pretty much every feature that existed before the downtime has been restored. The below are the issues that I'm aware of, but decided weren't urgent enough to delay launch any further:

• The database backup we were operating on had some character encoding issues; as such, you may notice some UTF-8 characters (e.g, accented names like Törmänen, or CJK ones like 古力) in usernames and topics be malformed. If that is the case, please contact an admin/mod and we will take care of it. I've fixed a couple of the glaring ones already, but I'm sure some have escaped me. NOTE: This should not affect actual post text, since that was binary-encoded.
• The search index is being rebuilt overnight. Until then, search terms won't work for any topics older than today's.
• For the next ~48 hours, your DNS settings may flip back and forth, leading you back to the old site (i.e, the error page). This will just solve itself within a day or two, as the new address reaches all the corners of the world.

If you notice any other problems not on this list, please reply to this thread with it!


Good luck and have fun, everyone
-Adrian

_________________
The road to wisdom? Well, it's plain, and simple to express: Err, and err, and err again; but less, and less, and less!

Glad the site is back. There are both content and members here that would be sorely missed if the go community were to lose them.

I'm moderately tech-savvy; let me know if there's anything I can do to help.

I'd like to extend thanks to Adrian, once again, who offered to host us on his AWS instance, and who also set things up with a pretty quick turnaround.

He received the database backup yesterday, and we are up and running today!

With multiple people having access to the AWS account, along with a daily backup of the database publicly available, we should be able to avoid the problem we had this time around - we won't be bottle-necked in fixing the site if an unexpected area fails.

_________________
be immersed

Thank you all for resolving this!

I've got one point to add to the list:
Hopefully there won't be a next time, but I d appreciate it in case of long downtime if there was, after some delay of course, a more informative error message saying "give us a week or two" or pointing to senseis or reddit where there was some status update.

_________________
If something sank it might be a treasure. And 2kyu advice is not necessarily Dan repertoire..

Many, many, many thanks to Adrian and Brian (Apetresc and Kirby)!!!

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Yup, that's the sort of encoding problem I was referring to in the "Known Issues" part. Thanks for pointing it out, I've fixed it now


Yeah, for sure. Sometimes it's not possible to put an error message on the lifein19x19.com URL itself, depending on the nature of what the problem is, but at the very least Reddit/SL.

_________________
The road to wisdom? Well, it's plain, and simple to express: Err, and err, and err again; but less, and less, and less!

Thanks a lot, keep up your outstanding work!

*dances*

_________________
Confucius in the Analects says "even playing go is better than eating chips in front of tv all day." -- kivi

_________________
The road to wisdom? Well, it's plain, and simple to express: Err, and err, and err again; but less, and less, and less!

Wow, SO happy to be here again.

HUGE THANKS to everybody involved ♥

I had reloaded my “view unread posts” tab almost every hour, only to get “The host does not exist.”, but just a minute ago Schachus wrote on the German DGoB forum that the URL without the WWW, i.e. http://lifein19x19.com/, DOES work, while the URL with WWW doesn't.

So:
http://www.lifein19x19.com/ BAD
http://lifein19x19.com/ GOOD
Would be nice to have this resolved, too.

• Also, could we get https?

• And about the “Donate” button … does it point to the correct PayPal acct already?


Thanks folks, you’re cool!

_________________
“The only difference between me and a madman is that I’m not mad.” — Salvador Dali ★ Play a slooooow correspondence game with me on OGS?

Good catch. I've just added the DNS entry and VirtualHost for www.lifein19x19.com too, it should start working in a few hours, again as DNS propagates. Thanks!


Yes! Now that Let's Encrypt is giving out free certificates, that's a possibility. I'll add that to the roadmap.


Nope, haven't sorted that part out at all yet. I guess I could remove the button for now.

_________________
The road to wisdom? Well, it's plain, and simple to express: Err, and err, and err again; but less, and less, and less!

Awesome



NOOOOOO, totally inacceptable, a new and valid button, please

_________________
“The only difference between me and a madman is that I’m not mad.” — Salvador Dali ★ Play a slooooow correspondence game with me on OGS?

Thanks for getting everything running again!

Thanks so much for all the efforts to get the site back online!

I just want to add my appreciation that Jordus, Brian and Adrian had the will and technical wherewithal to turn a database dump back in to L19 gold. Thanks for your time and work!

_________________
Patience, grasshopper.

Thank you all for your hard work and long life to the nineteen.
I blame the outage for my bad tournament results this weekend.

_________________
La victoire est un hasard, la défaite une nécessité.

Godaddy is probably the worst well-known host today, congrats on finally moving off them
Easier way to get https is go through cloudflare, only a few dns changes needed, unsure about cost vs traffic though

One thing I notice is that pages seem to load a lot faster.
(Or is it the “Safari is snappier now after the OS update” illusion? )

I’m even tempted to raise the number of displayed comments on page … you’ll all remember the many problems we had loading with loading pages … would you recommend trying it or should I better not?

<edit>
Oh, and could we have two shrines, one for the folks who managed the site before the resurrection, another for the new admins? I’d want to burn some incense there
</edit>

_________________
“The only difference between me and a madman is that I’m not mad.” — Salvador Dali ★ Play a slooooow correspondence game with me on OGS?

My apologies for the long downtime. Apetresc gave a good summary of the issue, the root problem was our original hosting service changed terms and put a 1gb size limit on the database. We outgrew that limit. I did not receive any warnings about the database and was not monitoring for this as I was unaware of the service change. The hosting service locked the database down which caused the database error everyone was seeing at the beginning of our downtime. I had made decisions throughout our downtime that I thought would help us get back up faster. Unfortunately it all backfired. I won't go into more details here but the short of it is after $500 and a lot of wasted time things went nowhere. I also was not able to be as attentive to the issue as I would have liked due to demands of the job(Network Engineer w/ on-call schedule falling in the middle of the issue.... ) and family (wife and 3 young children.... help me? ).

Again, I humbly apologize for the downtime. As a member of the tech community I take it as a personal failing on my part. Despite issues with our hosting I should have had contingencies in place. I've learned a lot since Lifein19x19 first came into existence. There are somethings I think I would have done differently and somethings I know could have been done better as I looked back at my old work during this migration process and thought "What was I doing when I did that?", "Oh wow I was definitely an amateur then.", "OMG what was I thinking?! ". As Apetresc pointed out in his posting we now know where we came short and will work to make this not happen again.

Special thanks to Kirby for facilitating the communications with Apetresc to get us back up and running. Without Kirby we still may not have been back up at this time.

Also special thanks to Apetresc for using his servers and technical expertise to get us back up and running.

I'm glad that while the site was unavailable for some time, we managed to keep the data intact and prevent the loss of this treasure of data this forum holds like that which we felt from the loss of our predecessor site.

My Humblest Apologies,
Jordus

Long live Lifein19x19!!!!!

_________________
I'm thinking...

Thank you, Jordus!

I appreciate how all awesomely you all co-operated in reanimating L19 again, and it’s very nice to see you’re still on board; knowing how voluntary work can sometimes “grow over one’s head”, as we say in Germany, I had feared you’d opted out. Everybody else also still there?

This here is one of the friendliest places I’ve ever been on the ’net. Thanks for that also to all former and current team members.

♥

(dammit, can we have font sizes >200 pls? That’s supposed to be a HUGE heart )

_________________
“The only difference between me and a madman is that I’m not mad.” — Salvador Dali ★ Play a slooooow correspondence game with me on OGS?