Bucket Brigades and Flooded Basements: Disaster Recovery Lessons from Hurricane Sandy

From the archive: this article was written in 2012 and reflects the technology and threats of that time. For current advice, see our latest articles or ask us directly.

Two weeks ago, Hurricane Sandy pushed a record storm surge into Lower Manhattan. Streets, subway tunnels, and building basements filled with seawater, and a large part of the city lost power for days. Among the buildings hit were several data centers, the heavily protected facilities that host websites and servers for businesses around the world.

Some of those data centers went dark. Several well-known news and media websites were knocked offline for a time. At one facility on Broad Street, the flooded basement took out the pumps that fed diesel fuel to the rooftop generator, so staff and customers formed a bucket brigade, carrying fuel up the stairwell by hand to keep the servers running. It was an impressive effort. It was also a reminder that even professionals with generators and backup plans can be caught out when the water comes in.

Florida had its own close call

We do not have to look to New York for a reminder. In late August, Tropical Storm Isaac tracked up the Gulf west of Tampa Bay just as the Republican National Convention was getting under way downtown. Organizers cancelled most of the convention's first day as a precaution. Isaac stayed offshore, and the Tampa Bay area got off with wind, rain, and some flooding before the storm went on to strengthen into a hurricane and hit Louisiana.

For a lot of local businesses, Isaac was a dress rehearsal. Some people pulled out their hurricane plans and found them in good shape. Others realized, the Friday before, that they had no idea whether their server was backed up or what they would do if the office was closed for a week.

Lesson one: water finds the basement, and the ground floor

Many of the problems in Manhattan came down to where things were located. Fuel pumps, electrical equipment, and sometimes the servers themselves sat in basements and low floors that flooded.

Most Pinellas buildings do not have basements, but the principle carries over. If your server closet is on the ground floor of a building in a flood or evacuation zone, which describes a lot of Pinellas County, storm surge or heavy rain can reach it. At a minimum, keep equipment off the floor. Better yet, make sure there is a copy of your data somewhere the water cannot reach.

Lesson two: generators are not a plan by themselves

Generators need fuel, fuel needs delivery, and deliveries stop when roads are closed or flooded. Sandy's extended outages showed how quickly that chain breaks. For a small office, a generator may keep the lights on, but it will not help if your staff cannot get to the building or your internet provider is down across the whole area.

The more resilient approach is to make sure your business can keep working somewhere else: from home, from a branch office, or from an office outside the storm's path. That means your data and critical applications need to be reachable from outside the building.

A backup in the same building as your server protects you from a failed hard drive. It does not protect you from a hurricane.

Lesson three: off-site backups have to be truly off-site

Taking a backup drive home at night is better than nothing, but if home is five miles from the office, the same storm is likely to affect both. A copy stored with an online backup provider, ideally in a data center in a different region of the country, keeps your data safe even if everything local is underwater.

And as we noted this summer after a Wired reporter lost irreplaceable family photos in a single afternoon, a backup only counts if it exists before you need it. Read what the Wired reporter hack teaches about backups.

Lesson four: know how long recovery takes

Having your data safe is step one. Getting back to work is step two, and it is the one that often surprises people. Restoring a full server from an online backup over an ordinary internet connection can take days. Ask yourself two questions:

  • How much data can we afford to lose? If the answer is "not more than a few hours," a once-a-week backup is not enough.
  • How long can we afford to be down? If the answer is "a day," you need a plan that can bring your key systems back that fast, not just a copy of the files.

Newer backup systems can keep a ready-to-run copy of your server that can be started up quickly, either on local hardware or in the cloud, while the original is being repaired. For many small businesses, that is the difference between a bad day and a lost week.

Lesson five: write it down, and practice

Every business that comes through a storm well has one thing in common: someone thought it through in advance: who does what, who calls customers, where staff work from, and how everyone gets in touch. In a real emergency, nobody wants to be making those decisions for the first time.

What to do before next hurricane season

  1. Confirm what is backed up, how often, and where the copies live.
  2. Add an off-site or online backup in a different region if you do not have one.
  3. Test a restore and time how long it takes.
  4. Raise servers and network gear off the floor and put them on battery backups.
  5. Make sure staff can work remotely if the office is closed.
  6. Write a one-page plan with contacts, responsibilities, and the steps to shut down and restart equipment.
  7. Review it every spring, well before June 1.

The bottom line

Sandy showed that even well-funded, professionally run data centers can be pushed to the edge by a big storm. Florida businesses live with that risk every year. The good news is that solid off-site backups and a simple, tested plan are well within reach of a small business budget. Our backup and disaster recovery services are built for exactly this. If you would like help getting ready before next season, email us at info@pinellasit.com.

Need help with this?

Talk to a local IT team that has supported businesses across Pinellas & Pasco County since 2010.

Get a Free Consultation Free Security Assessment