There are lots of different subtitles this post could have:

• Just because a Synology update SAYS it doesn’t need to reboot the array doesn’t mean it won’t…
• Yanking storage out from under a running MySQL database while it’s live is a bad idea…
• Backups are a good thing…
• Setting up health check rules based on content matching rather than just port response is a good idea…

Thanks to a timely tweet from Andrew Miller this afternoon, I noticed this blog was down, and unable to connect to its database.  I did what any good Windows user would do, and logged into the server and rebooted it.  Oops.  No bueno, boots straight to a BusyBox prompt.

While I’m all for learning new things, I also didn’t want my blog down while I did, so I reached out on Facebook, and luckily Matt Cowger had some time to jump in.  I built a new VM, attached the old .vmdk to it, and Matt was able to work his magic and get the old drive mounted.  I watched him do it, but I’m still not sure exactly what he did.  Ninja.

Once I had the drive mounted, I was able to take it from there.  Copy out all the old WordPress and MySQL files, move them over to a replacement WordPress appliance, get everything back in place and booting.  There were some small issues, most notably that a handful of the WordPress MySQL tables were coming up as read-only, but a quick dump and reimport took care of that.

All told, the site was down for about 6 hours.  Now, there are definitely some things I could do in order to prevent that from being the case next time.  If I’d had a nightly backup, I could have just deployed the new appliance, logged in and restored.

There are also some services/plugins (like http://updraftplus.com/) that can make this process more seamless, which I’ll definitely be looking into.

One option I won’t take is letting someone else host the site (sorry, Tips).  Outages and headache aside, this site is one of the things I use to stay hands-on, and it’s one of a few “production” workloads that I run on my own hardware in my own co-lo space.  Maybe, someday, I’ll move it up into the cloud, but for now it’ll stay right where it is.  I’ll just work harder to back it up regularly. 🙂

Oh, and for the sake of posterity, here’s the song of thanks that I wrote and sang to Matt on Facebook.  Thanks again, brother.

Come and listen to a story about a man named Matt,
A cool vSpecialist, Linux he had down pat,
Then one day he was workin’ at his desk,
And up through Facebook came a dumb request.

VM crashed that is, blog down, crisis!

Well the first thing you know ol Matt jumps right in,
Facebook people said “Self-hosting is a sin!”
Said “The Public Cloud is the place you ought to be”
But we had to get the VM recovered first, start the recovery.

Manual, that is. SSH, LVM.

Well now its time to give some thanks to Matt for all his help.
He fixed my stuff and did it quick, responding to my yelp.
Some booze he’ll find to grace his home, it’s really much deserved
Friends like that are rare to find, the best of all the nerds.

A compliment that was. It could have been worse, I could have been stuck with Damian.