We're Back! Extended Downtime Postmortem and Future Plans
Posted: Sat Feb 18, 2017 4:38 pm
Hey everyone! As you've all certainly noticed by now, Life in 19x19 was down for about 11 days. The good news is that we're now back and (hopefully) better than ever!
What Happened
Without going into too much technical detail, there was some sort of quota that was hit on the database server hosted by GoDaddy. Jordus (the admin who owned the actual hosting account) was having trouble getting timely action from GoDaddy, so not much progress was being made. After about a week, Brian Kirby and I offered to move everything to my AWS account, where we would have complete access to both the hardware and software, instead of being at the mercy of GoDaddy's tech support. At first we were scraping together old backups from Kirby's hard drive and what was left of the FTP server (which were over a year old!
), but eventually Jordus saved the day and gave us a raw dump of the database just prior to the outage. After some scripting, I had the full site up on an EC2 instance (for phpBB/Apache) and RDS instance (for MySQL).
The Future
Obviously the length of this downtime re-emphasized the need to reduce our bus factor. There were a couple of us with admin access to the board, but only two that could directly connect to the database, only one that controlled the domain, and 0 of us that had root on the physical hardware everything ran on (thanks GoDaddy). Going forward, we're going to be:
Are we missing anything? Are there any other points of failure the community wants to see plugged?
Known Issues
Pretty much every feature that existed before the downtime has been restored. The below are the issues that I'm aware of, but decided weren't urgent enough to delay launch any further:
Good luck and have fun, everyone
-Adrian
What Happened
Without going into too much technical detail, there was some sort of quota that was hit on the database server hosted by GoDaddy. Jordus (the admin who owned the actual hosting account) was having trouble getting timely action from GoDaddy, so not much progress was being made. After about a week, Brian Kirby and I offered to move everything to my AWS account, where we would have complete access to both the hardware and software, instead of being at the mercy of GoDaddy's tech support. At first we were scraping together old backups from Kirby's hard drive and what was left of the FTP server (which were over a year old!
), but eventually Jordus saved the day and gave us a raw dump of the database just prior to the outage. After some scripting, I had the full site up on an EC2 instance (for phpBB/Apache) and RDS instance (for MySQL).The Future
Obviously the length of this downtime re-emphasized the need to reduce our bus factor. There were a couple of us with admin access to the board, but only two that could directly connect to the database, only one that controlled the domain, and 0 of us that had root on the physical hardware everything ran on (thanks GoDaddy). Going forward, we're going to be:
- Posting the phpBB source, with all our theming and modifications, to a public GitHub organization, so anyone can clone it and help develop plugins and themes.
- Scheduling automated daily backups of the DB. (This is actually already done as of today)
- Make a sanitized (i.e, PMs and password hashes removed, etc) copy of this daily backup publicly available, as Linus Torvalds himself recommends
- Get a few more technically-savvy admins access to the AWS account this is running on. More on this soon.
- We're running on a downright ancient version of phpBB (3.0.8, from November 2010 !!). Once everything's calmed down, I've verified the backup procedures work, and the versioning to GitHub is complete, I'm going to run an upgrade to the latest phpBB 3.2. Security and performance are the main benefits. This should be complete by end of day Monday February 20th.
Are we missing anything? Are there any other points of failure the community wants to see plugged?
Known Issues
Pretty much every feature that existed before the downtime has been restored. The below are the issues that I'm aware of, but decided weren't urgent enough to delay launch any further:
- The database backup we were operating on had some character encoding issues; as such, you may notice some UTF-8 characters (e.g, accented names like Törmänen, or CJK ones like 古力) in usernames and topics be malformed. If that is the case, please contact an admin/mod and we will take care of it. I've fixed a couple of the glaring ones already, but I'm sure some have escaped me. NOTE: This should not affect actual post text, since that was binary-encoded.
- The search index is being rebuilt overnight. Until then, search terms won't work for any topics older than today's.
- For the next ~48 hours, your DNS settings may flip back and forth, leading you back to the old site (i.e, the error page). This will just solve itself within a day or two, as the new address reaches all the corners of the world.
Good luck and have fun, everyone
-Adrian
