It was, hands down, the most feel-good IT story to come out of last year's Hurricane Sandy: an honest-to-goodness...
bucket brigade of people hauling more than 1,500 gallons of diesel fuel up 17 flights of stairs to keep a lower Manhattan data center up and running.
Power was cut to Peer 1 Hosting's 75 Broad St. data center on the evening of Monday, October 29, 2012, but Mike Mazzei, New York data center manager, thought he had everything under control. With 20,000 gallons of diesel fuel in the basement, they could simply pump fuel up to its data center and run off the generator until power was restored.
"What caught everyone by surprise was the extensiveness of the flooding," Mazzei said. "We lost two weeks of fuel and all of our pumping infrastructure."
Without access to the basement, Peer 1's generator only had enough fuel to last until about Tuesday afternoon. The company considered an orderly shutdown, but Mazzei lobbied for data center uptime, gathering diesel wherever they could get their hands on it.
"There's a lot of pride and work ethic that goes into fixing, maintaining and keeping networks up and running," Mazzei said.
The decision to keep going was made easier by the fact that Peer 1's network was still operational.
"If the network is down, you can argue that keeping a data center up doesn't have a lot of value, but we still had lots of data going in and out," he said.
Thus, the challenge of getting diesel to the 18th floor began.
"It became clear to us that as long as we could get fuel up to that generator, we could keep going indefinitely," he said.
Early efforts centered on teams of four or six men lugging half-full 55-gallon drums, but it quickly became evident that wasn't going to work. Over time, Mazzei and his team settled into a routine of one person carrying two five-gallon buckets of diesel up two flights of stairs. Approximately 40 people participated in the bucket brigade until early Thursday morning, when external pumping equipment finally went operational.
Data center uptime during natural disasters
The water is gone, but not forgotten, and Mazzei believes Peer 1 is in a much better position to weather a similar storm down the road.
"If we did have an emergency like this again, we are light years ahead of where we were," Mazzei said. "I collected a lot of goodies along the way," including external pumping equipment that could take fuel up to the 18th floor.
The 75 Broad St. proprietors have also done their bit to prepare the building, including work to upgrade to submersible fuel tanks and to move the pumping package up higher, away from possible flooding.
At the same time, both the building operators and Peer 1 are hamstrung by logistical and political realities. While some of the building's emergency power systems are being moved, relocating switch gear "is too big of an undertaking," Mazzei said. Likewise, diesel has to stay in the basement.
"You're never going to be able to put a 20,000-gallon fuel tank on the upper floors of a building. The New York Fire Department is never going to allow that after September 11."
Small data centers in the crosshairs
All this begs a larger question: What is the future of small data centers in geographically sensitive urban areas?
Going forward, Mazzei admits that Peer 1's strategic interests probably lay in purpose-built data centers such as its Toronto "Pullman" and Portsmouth U.K. data centers, designed as Uptime Institute Tier 3 facilities in geographically stable areas, with much lower Capex and Opex costs than urban facilities.
"Large Capex spends are more advantageous in a larger data center," he said.
But Peer 1's heritage is its small colocation facilities such as the one at 75 Broad St., and the company has a commitment to those customers, Mazzei said.
Unlike managed hosting of cloud users, colocation customers "don't migrate very well," he said. "We have a commitment to them that we're not going to create a migration challenge for them."
So while the long-term plan is to "sunset" smaller facilities into "more purpose-built environments," Mazzei still sees quite a bit of life ahead for Manhattan data centers.
"This facility still has strategic value. I have some companies that have gone on to become pretty big businesses, and they still operate out of this data center," he said. "I take a lot of pride in that."