Updated Nov. 1 – As Hurricane Sandy bore down on the East Coast on Sunday, a data center building in Manhattan lay directly in the path of the storm surge, leaving its staff to go above and beyond to keep operations running where possible, and to recover them when it wasn’t.
This meant that hundreds of customers of Peer 1 Hosting and Internap Network Services Corporation, located in Manhattan’s Zone A where floods took place, had the barest minimum of downtime during and after the unprecedented storm.
“The [IT] infrastructure has held up pretty darn well so far,” said Carl Brooks, analyst at Tier1 Research on Wednesday afternoon. “Overall, it’s been remarkable.”
Peer 1 Hosting, which has a colocation facility at the site, managed to avoid an outage when a basement fuel tank for the building’s diesel generators flooded. The company had its own auxiliary generator on the 17th floor, fed by a tank located on the 18th floor, but no way to pump fuel from a truck on the ground up to that tank.
“The fuel line coming out of the truck … is four inches [wide], whereas the fuel pipe from the basement coming from the large tanks was a three-quarter inch pipe,” said Robert Miggins, senior vice president of business development for Peer 1. “We got the fuel really close to where it needed to be, and all of a sudden we’re looking at these different sized delivery pipes – neither we nor the fuel company had the proper coupling to tie into that existing infrastructure.”
So, Peer 1’s five on-site staff began to carry fuel in small containers up the stairs to the auxiliary generator’s tank.
They were soon joined by independent contractors, friends working as volunteers, and ultimately, Peer 1 customers who’d arrived on the scene to try to take care of their servers – in the end, all told, 30 people participated in the bucket brigade to keep Peer 1’s data center powered on and its network connected.
“Customers turned up expecting to take care of themselves or take measures that they had in mind, but instead once they found out what [our people] were working on, they rolled up their sleeves and joined in,” said Miggins.
“They’ve been there ever since.”
This wasn’t the only crisis Peer 1 faced – with the building’s generators down, air conditioning for one of its data center sites soon became problematic. Though there were posts on Wednesday morning that Site 1 on the 17th floor must be shut down due to cooling issues, Miggins said no part of the data center was powered down throughout the hurricane and its aftermath.
As of Wednesday afternoon, the bucket brigade had amassed some eight hours’ worth of fuel in the auxiliary tank and on the ground beside it, not only catching up to the demand for more fuel but actually buying some breathing room.
Internap makes an amazing recovery
Meanwhile, Internap, which has operations at 75 Broad Street as well as 111 8th Street downtown, had staff at the location working around the clock at the site as well, in some cases for two days, according to Steven A. Orchard, senior vice president of development and operations.
On Sunday night, Internap switched over to generator power as it would do in any case of emergency or power outage. However, a diesel fuel tank and pump were located in the basement of the building, which soon began to flood. Salt water also eventually breached the fuel tank through a ventilation pipe which ran outside the building into the street.
Meanwhile, by Monday night, Internap’s staff found themselves trapped in a harrowing scene – on the 14th floor of a building where flood waters were rising into the lobby, reeking of diesel from the basement tank and gasoline from flooded-out cars.
At this point, customers were advised that plans were being made about how to evacuate the building in case of a fire, although nobody did evacuate, Orchard said.
Internap’s generators are up on the mezzanine level of the building, above the flood waters. And a header tank with about 1,500 gallons of fuel was up there, enough to last until about 11 a.m. Tuesday. Then, and only then, did Internap lose power to the site.
Internap’s primary diesel fuel vendor’s supplies were being exhausted by a number of hospitals also on its route, so Internap looked to a secondary vendor, which trucked fuel into the city out of Baltimore, arriving late Tuesday afternoon.
But the recovery was far from assured at that point. Two other things were needed: couplings and piping to be able to get the fuel out of the truck, and a pump to get it up to the 14th floor.
Here Internap turned to another of its vendors, an HVAC company, whose workers fabricated custom couplings and pipe fittings, attaching the truck to a pipe that ran up the side of the building, in through an opening in the wall, and into the header tank.
Tuesday evening, the infrastructure was turned back on. After an unprecedented storm and flooding, Internap’s customers had seen just a nine-hour outage.
“I don’t think people outside the city have a clear understanding of exactly how devastating it was and how frightening for the team that was there,” Orchard said. “I can’t say enough about them.”
Dig Deeper on Data Center Disaster Recovery
Beth Pariseau asks:
Did your data center experience an outage during Hurricane Sandy?
0 ResponsesJoin the Discussion