Thursday, June 11, 2009

'Twas A Dark And Stormy Night

One of the online "cloud" services I use - it doesn't matter which one - had a bit of trouble last night

So, here's the story. A big storm went through the city where our datacenter is located. The datacenter decided to proactively switch to generators. During the switch, something got screwed up, and the power went off for a few minutes. As (bad) luck would have it, this caused our database to get corrupted. We are currently working to bring it back online and restored from the live backup. The crack team at Rackspace is on the job. Thanks Rackspace! Unfortunately, the database is so large, that it will take some time to transfer and verify all the data. Hopefuly not more than a few hours. We know that this is very bad, and we apologize for any inconvience that this will cause. Please check the forums when we are back online for a full report.

The challenges of running services for others are extreme ... this is the kind of thing that awaits shared service providers; and even if they don't think it will happen to them, they still have to convince everyone that it won't. And customers, they're sceptics about this kind of thing.

1 comment:

  1. Cyril7:50 pm

    I think customers are sceptical about paying for mitigation of these risks.  The question always is..why don't you prevent it in the first place.  Which is fair I guess.  

    ReplyDelete