OK, I’m going to shamelessly link to one of my own companies blogs because the post and embedded YouTube video about a data center disaster needs to be further spread. I don’t know if this is an old or new incident (didn’t look up dates) but it is quite telling and certainly not funny from a CIO perspective. However, the video might be of use to someone trying to make a point about the need for real disaster recovery planning and business continuity planning.
A year ago, I wrote on my internal company blog about why we do disaster recovery planning and why we make investments to protect our assets in our data centers. As can be easily expected, that night we had a major power outage and one of our data centers was off-line for several hours. Great timing. I wrote post about it the next day and fell on my sword.
We’ve experienced a situation where the line power from the local utility becomes unstable and goes off and on in a random, rapid fashion and the resulting up and down power fluctuation damages one of our backup power systems which then brings down the whole grid in the site. The backup system fails. Thank you.
We’ve also planned around the SARs outbreak and we had to split some of our teams into two groups that did not cross paths in order to keep each team separate and hopefully healthy. In the case of SARs, we even saw were some local governments were considering closing facilities where outbreaks were happening in order to contain the spread. How would your data center work if it was closed and local staff could not get in for a period of time? Have you planned for that one?
My favorite is when an contract electrician doing work in the site elects to shutdown power to the whole data center without talking to anyone in advance.
In short, as I posted earlier, you can’t really predict all the things that might go wrong.