Sometime on December 26th, 2007, the day after a very pleasant and mellow Christmas, I started getting emails and SMS pings from friends and readers that my site was looking very weird. I quickly logged in via my iPhone and was greeted with a non-so-happy surprise. Somehow, and without explanation, every entry of my blog had been twisted into the same weird blob of text. It was like being kicked in the gut to see the work that I'd done seemingly vanish into thin air.
It took me a while to even start figuring things out, and I'm still not sure exactly what happened. I don't know if something went wrong with the supposedly advanced server cluster that my site was sitting on, of if somebody somehow broke in and mixed things up. At this point, I don't think it was an intrusion. But, given the insulated nature of the system I was using, I'll never know for sure. In any case, the old database is useless for recovery.
Worse, I've been waiting for a while for the service provider I was using to setup their backup solution. It's been a "Coming Soon" state for a very long time, but has never arrived. I should know better, but I didn't have local backups of the database. It's sadistically humorous that I've fallen prey to exactly one of the scenarios that I've helped several clients prepare for. The story of the cobbler's children having no shoes seems to apply here.
On the bright side, I've just spent the last few days of the holiday season taking time off from most of the Internets and thinking about what I want to do with my site and the tools I use to manage my online presence. I've launched into a sizable plan of action that includes all the bullet point items that I'd advise any client to implement, including a new server, more capable software stack, and more. Part of that plan does include recovering some older content, and thanks to the efforts of many friends and readers, I've now recovered all of the old content in various forms. But, manual tedious processing will be required to restore that data and I have no timeline for restoration and will make no promises as to how much will return.
I'm glad this happened during a quiet time of year. All things considered, this might be a good thing in the long term. Forests must burn so that they can grow again. So to will this site rise from the ashes.
You don’t have a personal back-up of your site? (Maybe that’s a limitation of blogging software I haven’t encountered.)
Well, I haven’t dumped out the database in a while and WordPress doesn’t write out static files, so I don’t have the same kind of on disk artifacts that an installation like MovableType has. I do have bits and pieces of the content in various places, so will be able to reconstruct some of the more important bits.
The crux of the problem is that I put off backups while waiting on a provider to set that kind of thing up. I should have been more proactive about it. /sigh/
I beg you: don’t make my mistakes. I load my blog from work on my C drive, but the photos were always on an external hard drive which died. I lost years of shots. Back up, back up, back up. Your blog thoughts are probably as valuable as your shots, since they both involve your time.
I lost shots I never got to saw.
Indeed, the photos are the number one backup priority and documents are number two. Unfortunately, the blog data was number three and fell off the list. That won’t happen again. Also, I’ll be reviewing my backups for the photos and documents in the next few days–you can be sure of that!
Yikes! A corrupt WordPress database and no backup is a bit of a disaster :( I’ve dealt with a couple of WordPress hacks before but I’ve not seen anything like that output, looks more like a software error on the server?
Google’s cache will help with most of the lost content. It’ll be a pain going through it it all – I’ve had to do this myself once – but it’s better than nothing and a reminder as to why you should backup your data ;) Best not leave it too late, though, I’m not sure how long it’ll persist for:
http://www.google.com/search?q=+site:duncandavidson.com
I’ve got your RSS feed items back to Nov 28 if that’s any help? Can pass them on as HTML is you’d like, should be easy enough to import.
Might also be worth giving the Wordpress built-in database backup plugin a look for the future? It’s a one-click SQL dump so is really simple to use. (Don’t mean to rub it in!)
Yah, I’m not sure it was a hack. I think it may have just been something horribly wrong in the database the hosting company provided. Though I could be wrong. The easiest explanation is that something executed an UPDATE on the post.content column without a WHERE constraint.
I dunno. The good news is that I’ve been given a few backups of the RSS feeds and pointed to a few different places to pick up content. Those, along with the link you provided, will be helpful in the rebuild process. It’ll be a long couple of days.
The good news is that I’ve got a new dedicated system with a much better backup system for daily and weekly system images. One that’s not “coming soon”. I’m also putting into place a plan for offsite backups to home that fits in with my wider data backup plan. That should keep me busy for a little bit.
One more thing you might want to consider: address redirect (301 I think). I’ve got blog.duncandavidson.com as the bookmark to your site, and I suspect a few others have as well. Hence at the moment you’re losing traffic and might also have issues with search engine rankings - read diminished Google juice - as best I can guess.
Steve: Good catch. Thanks for that. I’ve got that set up in DNS now and hopefully it’ll propagate out fairly quickly.
And also the RSS feed is notifying when you update the Elsewhere part of the website as well as the central column… You may have meant to, in which case just ignore me. I just thought it was a mistake.
Jeff: That’s on purpose. I want a single feed for the site, but I don’t want the front page cluttered with the small stuff.
James,
So sorry to hear about your troubles. It feels like losing the manuscript to a novel, or losing your old negatives in a fire or something when a blog is lost.
Perhaps it’s a good opportunity… to switch? =) There’s a nice back-up plugin, among other goodness.