New Yorkers know all too well that disasters can come in all shapes and sizes, and this fall’s Superstorm Sandy gave data center recovery plan architects plenty to think about when it comes to infrastructure, applications and staff.
Flooding, transportation snarls, and power and communications outages brought businesses in the Tri-State area to a halt for days, and in some cases weeks. And now, several months later, data center recovery precautions that used to seem too onerous or expensive are suddenly anything but.
One key disaster recovery (DR) lesson that many companies learned was the importance of having multiple communication carriers, said Francis Poeta, president of P&M Computers, Inc., an IT consultancy based in Cliffside Park, N.J. Since the storm, many of his customers are drawing up plans to have a second and sometimes third provider that is wireless and above-ground.
“You can’t just have a standard OC3 or Fibre connection that runs into the basement,” Poeta said.
Speaking of basements, organizations are also thinking twice about the wisdom of keeping expensive equipment and diesel reserves below ground where they can get flooded—or even on the ground floor.
“A lot of data centers are built on the ground floor for weight reasons—and that’s a problem,” Poeta said. It will be hugely expensive to move existing data center equipment upstairs, but “people are going to take a look at changing that,” he said, “especially if a second storm hits.”
Don’t put all your eggs in one basket
Another post-Sandy trend is building out more data center space further from cities and coasts.
Xand, a data center provider based in Hawthorne N.Y., recently announced that it is adding 35,000 square feet of data center space to its Hawthorne and Marlborough Mass. facilities to keep up with post-Sandy demand. Hawthorne is about 30 miles north of Manhattan, and more importantly, at a higher elevation than low-lying New Jersey, where many of New York’s data center recovery sites have traditionally been located.
Other organizations are looking for data center space even farther afield. Festo, a manufacturing firm with U.S. headquarters in Hauppauge, N.Y. on Long Island, suffered from multi-day power and connectivity outages, but didn’t sustain any damage to its facilities. Nevertheless, the storm was a wake-up call to the company about the dangers of overconsolidating, and it is now fast-tracking a new data center in the Midwest, said Steve Damadeo, IT operations manager.
“We always knew that we’d have some capabilities there, but we’re expanding them based on what we learned during the storm,” Damadeo said.
That’s particularly true if a data center services more than a single location. Festo’s Hauppauge data center also services the company’s South American locations, and while employees there understood the situation, their patience quickly wore thin.
“They were like, ‘Great, I feel bad for you, but I want my email!’” Damadeo said.
Lines of Communication
Email outages during the storm were particularly painful, and showed many organizations what they need to do to prepare for the next disaster.
“The first step is to jettison on-premises email,” said P&M’s Poeta.
What kind of cloud-based email solution a company chooses—Microsoft Office 365, Google, or hosting at an Infrastructure as a Service provider like Amazon or Rackspace—depends on the needs of the organization, but Sandy made it clear that hosting email in a building that is vulnerable to power and connectivity failures is no longer an option.
To hedge against an email outage, Festo’s Damadeo also recommended gathering employees’ personal cell phone numbers as part of its data center recovery plan in order to communicate with employees via SMS.
With email and landline phones down, “we had no ability to communicate with our employees,” he said. Cellular carriers, however, were for the most part up, and so the company was able to blast out text messages to employees whose numbers the company happened to have stored off-site.
Finally, organizations need to reassess whether they need to plan for alternate workspaces, said Bruno Berti, senior director for product management at SunGard Availability Services, a DR services provider.
Before Sandy, many businesses had started to take the attitude that when it came to employees, “working from home was an acceptable DR policy,” Berti said. But when power outages are widespread across an entire region, working from home simply isn’t an option.
Not surprisingly, the firm has since had an uptick in interest in its recovery centers that provide displaced workers a desk, chair, workstation and phone. SunGard also has mobile units in trailers that come with satellite connectivity and a generator.
“You need to give people a place to work and be productive,” he said.
Let us know what you think. Write to us at firstname.lastname@example.org