When Hurricane Katrina hit, the storm was so powerful, so widespread and ultimately so destructive that even the...
most disaster-proof data centers in its path were left with a worst-case scenario so unbelievable that no one could have possibly planned for all of it.
As the waters recede from a flooded New Orleans and the arduous work of clearing away debris from the Gulf Coast continues, IT pros in areas affected by Katrina are ready to evaluate just what lessons can be taken away from one of the worst natural disasters in American history.
At least one thing became clear to Gulf Coast companies in the days following Katrina. As crazy as it once might have sounded, companies must have a backup plan in place for the possibility that the city hosting your data center might someday vanish.
"You can have so much go wrong, that even your fall-back plans fall through," said Sigmund Solares, CEO New Orleans-based hosting company Intercosmos Media Group Inc. "You have to be resourceful to find a solution."
When Hurricane Katrina hit the Gulf Coast at the end of August, it left 80% of New Orleans under water. Just weeks later, Hurricane Rita struck the Louisiana-Texas border, causing additional catastrophic damage. Hurricane Katrina alone is expected to cost U.S. property/casualty insurers an estimated $34.4 billion in insured property losses, making it the costliest U.S. catastrophe ever.
Not every data center in the Gulf Coast went down during Katrina. One of the most publicized tales of survival came from Intercosmos, which supports directNIC.com, the company's domain registry and hosting arm. Intercosmos managed to keep its 6,000-square-foot data center up and running throughout the storm. Located two blocks from Canal Street and just three blocks from the French Quarter, Intercosmos' building was spared from the flood waters, but not the blinding rain and wind gusts that blew out a majority of the windows surrounding the data center, which was located on the 11th floor.
With rain flying through the cracks in the room, metal chunks from other buildings battering the outside walls and looters gathering in packs on the street below, Solares and the four other members of his staff, were forced to create a makeshift levee, built with 2,000 T-shirts that Solares said came from "a failed business venture," to keep the water from reaching mission-critical hardware. Powered by generators, Intercosmos' data center never went down.
The crew even kept a running blog of their ordeal, which earned the company mainstream notoriety and hero status in the IT community. But when Katrina mercifully blew away, the company found itself trying to solve problems it never imagined.
To stave off looters, Solares and his team had to figure out how to lock the elevators so they wouldn't reach their floor. The UPS devices needed to be replaced, but their vendor was delayed 24 hours because he had to be inoculated against diseases health officials feared were floating in the decrepit Crescent City waters. That vendor was then turned away twice at roadblocks, and Solares had to scramble to find out how to get people through the barricades, then track down the vendor -- not an easy task considering spotty cell phone service in the weeks following the storm -- and tell him what to say to the guards to gain entry into the city.
Solares suffered similar roadblock headaches with the vendor who supplied diesel fuel for the generators. To make matters worse, Intercosmos had never issued employee photo ID or access cards, which meant that employees who wanted to re-enter New Orleans to help out were also turned away at roadblocks. And the nature of the evacuation was such that most evacuees left their laptops behind, leaving no way for Solares to locate many of his employees.
In an attempt to reestablish communications to simply locate his staffers, Solares and his team set up an Internet relay chat, and were eventually able to find about 30 of the company's 55 employees. Through it all, he found that when faced with challenges no one ever though would arise, good people are a company's best asset.
Limited resources, unlimited need
New Orleans wasn't the only area in crisis. Though Baton Rouge, La.-based CMA Technology Solutions was spared the brunt of the storm, the IT services firm had to support a variety of customers. Upward of 20 of CMA's clients, many of whom rode out the storm in Baton Rouge, found many of their mission-critical IT facilities offline for one reason or another, and they turned to CMA for help.
The company needed to handle on-site and remote assistance so it brought in three additional servers, on loan from IBM, and virtualized many of its physical layers to free up box space.
When the power came back on and it was time to start getting people back online, CMA faced a challenge it never saw coming -- bandwidth. Since many of its customers needed cables to get their networks rebuilt and up and running, CMA simply didn't have enough to go around. The company had to tell its employees to stay off the Internet unless it was absolutely necessary, and it took three weeks for it to secure the amount of cable it needed.
Like Intercosmos, CMA's CEO Chad LeMaire had trouble locating many employees and customers, mostly because people in affected areas had neither power nor cell phone service in the days following the storm. LeMaire said many of his customers expected to be fully operational within three days, but Katrina taught them a painful lesson.
"When you lose power, if you don't have a plan to hot-wire your heart somewhere else, you're done," said CMA co-founder Mike Mayer. "So those companies that realized that … they fared very well and were able to get back in business very quickly. Those who thought 'Well, that will happen to the other guy.' It happened to them."
To rebuild or relocate
With so much damage across the region, companies are determining whether to try to rebuild or relocate. Whether companies want to keep their data centers in the Gulf Coast isn't so much a question of civic pride. It's a question of whether they can afford to.
For many, the cost of relocating has suddenly become cheaper than the financial hit they'll take by operating at less than full capacity while waiting to see if and when the Gulf Coast will be able to fully support them.
"It became more affordable to build a second data center elsewhere. They used disaster recovery for a week, but after that they had to get back to business as usual," LeMaire said. At least one of his clients has built a new data center outside of the region.
Solares considers himself a proud, resilient citizen of New Orleans who harbors the dream that one day, the Crescent City will regain its place among the nation's great cities.
But Solares is also a realist, a man forced to put his family, and the 55 employees under his wing, before all else. So when asked to say whether he intends to do all he can to keep his company in New Orleans, Solares paused, and gave the only answer many in the Gulf Coast have.
He doesn't know.
"One of my partners put it this way. You can't run a business in a refugee camp," Solares said. "Personally, I don't want to bail on the city, and most of our employees feel the same way … but you have to be able to run a business."
Let us know what you think about the story; e-mail: Luke Meredith, News Writer