Members of American Society of Heating, Refrigerating and Air-conditioning Engineers (ASHRAE) Technical Committee 9.9 have talked about data center liquid cooling for some time, and this week that discussion continued at the 2009 ASHRAE Winter Conference. In a recent paper discussed at the conference, Vali Sorell, an engineer at Syska Hennessy Group, explored two hypothetical situations. One involved retrofitting an air-cooled data center for liquid, the other focused on building a new, liquid-cooled data center from scratch.
In a 2008 SearchDataCenter.com survey of more than 600 data center managers and IT professionals, about 65% of respondents said they would never use data center liquid cooling, and yet the idea gained traction in the industry as compute density has skyrocketed. Liquid is thousands of times more effective at dissipating heat than air, many experts say. But convincing data center managers to bring liquid closer to the heat source isn't easy. Only about 7% of users surveyed said they use liquid cooling in every data center installation, according to the survey.
Yet it's unclear what kind of liquid cooling they use. Most data centers are already liquid cooled. Chilled water comes into the computer room air-conditioning (CRAC) units, which brings cool air to servers. But there are options for getting the liquid closer to the heat source, either through in-row air conditioners or back-of-the-rack door attachments that include liquid piping.
At the ASHRAE conference, Sorell discussed bringing the liquid into the rack, or even closer, into the server.Data center liquid cooling as a retrofit
Pacific Northwest National Laboratory (PNNL) chose the retrofit option with an 800-square-foot portion of a 10,000-square-foot data center. The liquid-cooled portion of the data center includes eight racks, five of which are liquid cooled and the rest air-cooled for comparison purposes. The racks have 192 IBM x3550 servers computing as a cluster, according to Daniel Sisk, a technical group leader at the laboratory.
The liquid-cooled racks pump an inert liquid called Fluorinert through pipes into the server and onto the heat sinks inside, using SprayCool. PNNL now measures and evaluates data on the effectiveness of liquid cooling is.
The rest of PNNL's data center is not liquid cooled, and so the company had to retrofit a portion of it for this experiment. To do so, they tapped into the existing chilled-water loop. After being warmed, that chilled water flowed into a thermal management unit – also called a cooling distribution unit – to cool the Fluorinert, which in turn cools the servers. But doing that retrofit raises questions, according to Sorell:
- Will there be downtime to tap into the chilled-water loop?
- Do you have valves that can section and isolate CRAC units so that you can still provide air conditioning to the rest of the data center while making changes to the rest of the chilled-water loop?
- Are there any obstructions – pillars, for example – that could obstruct the routing of the new loop?
- Are the pipes the right size to handle the retrofit and keep water pressure as high as it needs to be?
- Are the chilled-water pumps powerful enough to handle the extra load?
- Can the building maintenance system handle the new system?
- Can you prevent condensation from developoing on the floor, in the racks and in the servers?
That last issue is particularly important. Chilled water that goes to a CRAC unit is usually about 45 degrees Fahrenheit. If it transfers temperature that cold to the Fluorinert, you could have 50-degree liquid pumping inside a server and onto the processors, which are about 160 degrees Fahrenheit. That can cause condensation.
"I don't need 45-degree water," Sorell said. "85-degree water is actually very good into a system, so the Fluorinert would be around 90 degrees."
Sorell recommended – and PNNL uses this method – finding a way to warm that 45-degree water up before it interacts with the Fluorinert.
"There was a concern that condensation could occur over time and cause corrosion and whatnot," Sisk said. "So we installed heat exchangers to warm the water up a little bit to avoid condensation."Data center liquid cooling in new installations
Installing liquid cooling into new data centers, on the other hand, brings some benefits. You can install a separate chilled-water loop for the liquid-cooled portion of the data center, separate from the air-cooled portion, Sorell said. And yes, with the liquid-to-the-server solution, you still need supplemental air cooling.
"The spray device doesn't necessarily handle all of the heating components," he said. "You'll need to size the CRAC units to handle the outside component of it."
Similarly to retrofits, new installations still need to be concerned about piping obstructions, the temperature of the liquid cooling the servers, and integration with a new or existing building maintenance system.
Sorell also said you should install valves to isolate each server rack, so that if you have to roll in new equipment or perform maintenance on existing servers, the rest of the racks can continue operating.