But in the midst of an outage, the best laid contingencies can fall through. Because, according to experts, while most data centers back up their power, they don't have back up power for data center cooling.
You can just hear those servers starting to sizzle.
It happened overnight to Ed Guild, electronic production specialist, research and development at Houghton-Mifflin's Boston office. Guild's department was running tests to evaluate a new publishing system. The servers weren't running critical applications, so IT staff was not watching them 24/7.
The servers were backed up on UPS, but the air conditioning was not. When the power went out, no one knew that it was down until the next morning when staff found some services not functioning.
"Turns out that it was a power outage. The servers were on a separate line and not affected, but the AC shut off due to lack of electricity," Guild said.
The data and the testing program are now being moved to data center within Houghton-Mifflin that operates at higher support.
According to Robert McFarlane, an expert in data center design and principal with New York-based Shen Milsom & Wilke Inc., a long-lasting uninterruptible power supply (UPS) is useless in keeping your servers running efficiently without air conditioning.
"People call for four hours of UPS battery but don't (or can't) put in a generator," McFarlane said. "Not only does the equipment eventually overheat, but so does the UPS."
Doug Sullivan, MCSE, a network manager at the Albany, N.Y.-based engineering firm, Clough, Harbour and Associates LLP, said that the problem with backing up air conditioning is the extreme power requirement on the UPS.
"Our AC unit draws more power than all of the equipment in the room combined," Sullivan said. "Ultimately, we're going to put a generator outside the building to power the environmental controls, lights and equipment."
The fact is, running air conditioning on the UPS isn't practical—and that's why data centers often avoid doing it. But, David Kelley, manager of environmental application engineering at Columbus, Ohio-based Liebert said data center managers facing outages without a generator need to do whatever they can to buy themselves a few precious minutes before thermal shutdown.
"If you don't have a generator, one thing you can do is put just the fan motors on the UPS to at least circulate the air you have," Kelley said.
But even a generator isn't a failsafe. According to McFarlane, there are a number of scenarios where generator power doesn't guarantee your AC unit.
- When all the technology, the air conditioners and the partially discharged UPS try to come online and start up at the same time, the generator isn't sufficiently over-sized to handle all that instantaneous starting current and the generator bogs down and fails.
- IT staff fails to maintain power draw records and forgets that they've added equipment, but haven't increased generator capacity to handle the additional draw.
- And, of course, there's always the issue of a generator not starting and having no backup.
Kelley tells horror stories of shops with diesel generators that come on, but the fuel pumps are on the UPS circuit. The battery on the UPS wears out and the fuel pump shuts down, shutting down the generator.
High-density equipment--blades in particular--can't function without cooling for more than a few seconds before going into self-protective thermal shutdown. And as vendors bring to market new servers with even higher heat densities, the problem will be even worse.
"This is, and will become even more so, a major problem and a very difficult and expensive design issue and consideration," McFarlane said.
Let us know what you think about the story; e-mail: Matt Stansberry, News Editor