Sorry for the extended outage last night. It was one of those perfect storm situations: Ordinarily, it would have rebooted on its own and been back up within minutes, but it locked up instead. Ordinarily, I would have been reading my email and seen the alert quickly, but I didn’t turn on my computer until almost midnight, and wasn’t reading it on my phone. Ordinarily, rebooting the webserver manually would have solved the problem, but it couldn’t connect to the DB. Ordinarly, tech support would have responded to my trouble report sooner, but they were doing planned maintenance on another system.
For the techies & tech-curious: Something caused memory usage on the web server to spike and lock up. Rebooting (hours later) solved that issue, but the web server couldn’t connect to the database. As far as I could tell the DB server wasn’t responding, but when tech support got involved, it turned out to be a network issue on the webserver.
It’s really frustrating. There were problems earlier last year, but they’d mostly settled out by late summer. From then on it was a few minutes here, a few minutes there. In fact, there was no downtime at all between early December and last Thursday. And now to be out for ~10 hours? It’s especially frustrating because I just consolidated all my domain names at Dreamhost last week.
In the future, if the site’s down and you want to know what’s going on or whether I already know about it, please check @SpeedForceOrg on Twitter. I’ll post updates on the site status there, and you can read the page even if you don’t have a Twitter account. If you don’t see anything there, please e-mail me at speedforce at pobox dot com. And if you just want to read recent posts, you can always check the Tumblr blog.
