The Massachusetts Green High Performance Computing Center does things a little differently, but their stance as responsibly cutting-edge means the ideas here can work in an enterprise data center.
The Massachusetts Green High-Performance Computing Center (MGHPCC) supports the research computing needs of Harvard University, Northeastern University, Boston University, the University of Massachusetts and Massachusetts Institute of Technology. Although the facility is managed 100% on-site in Holyoke, Mass., only 10 people are needed to run it, with all the IT management handled remotely. Running a lot of servers with a small number of people looking after the facility is a good model for any data center.
Two entrance rooms connect to the communications corridor. The traditional room's carrier and local connections are separated to prevent people from accidentally disconnecting things.
A dense wavelength division multiplexing system runs the Fiberlink, so there is a dedicated 10 Gb per second fiber connection to the schools' campuses that comes into the MGHPCC. A 100 Gbps upgrade will be implemented as soon as the cost comes down enough to make it worthwhile. The facility connects to Northern Crossroads, a high-bandwidth research network.
The power room, which is particularly warm, distributes current to five substations, which then convert down to the voltages for mechanical and computing equipment. Each distribution box weighs 200 to 300 pounds and has to be lifted with a crane, said John Goodhue, executive director of the MGHPCC.
Servers run on 230 watts to reduce electrical loss. The MGHPCC runs in some ways like a multi-tenant cloud, and its power density is higher than a colocation facility or typical enterprise data center.
Rather than participate in a demand-response power buying scheme or run on generator power to support high-power-use days, the MGHPCC instead powers down workloads as needed, because their user community is flexible. This could never work for mission-critical applications, but has potential for big data analytics and other batch processing applications.
The facility uses cable trays because they are cheaper and generate less concentrated heat than a cable bundle. The cables are 10 feet above the ground -- you need a ladder to access them, which keeps workers safe. An enterprise might adopt this design technique because it provides the most flexibility in the data center without an exorbitant cost.
The uninterruptable power supply (UPS) room at the MGHPCC is dark with rows of flywheel UPS cabinets. Although flywheels have higher upfront costs than batteries, they end up saving the data center money because they have a longer service life, Goodhue said. Data centers with flywheels don't have to worry about replacing failed or failing batteries.
Before a data center determines flywheels are best for their facility, determine if the amount of time that the UPS supplies reserve energy is acceptable. If not, stick with batteries.
The MGHPCC only runs UPS for 20% of the infrastructure. If the MGHPCC notices a considerable benefit in terms of maintenance by adopting flywheel UPS instead of batteries, the savings at a facility with five times as many UPS batteries could be astounding.
The MGHPCC keeps its server room at 80.5 degrees Fahrenheit and uses hot aisle containment. Air only moves a couple of feet from the backs of servers before it is cooled down again without using a raised floor cooling system.
By using a level floor rather than raised, the MGHPCC saved about $1 million U.S. in construction costs and eliminated the risk of critters living under the racks. The level floor makes it easier to clean and to move equipment, and the MGHPCC just puts wiring overhead rather than under racks.
MGHPCC's sprinkler system is filled with nitrogen. When they detect smoke, water is released to fill the sprinklers. Water only goes through fully X-ray-inspected pipes. These rigorous measures are taken to prevent leaks.
But a nitrogen-filled sprinkler system isn't for every data center. When selecting a sprinkler system, start with building code regulations. While MGHPCC's might be more appealing because it eliminates the fear of leaks, a wet-based sprinkler that is filled with water at all times might be mandatory.
The basement chiller room
MGHPCC has base load chillers and small compressors that fine-tune the cooling capacity to run as efficiently as possible.
Because the MGHPCC is in the cold New England climate, it only chills water the traditional way with a compressor about 30% of the time. Because the compressor uses more energy than free air, the rest of the year the MGHPCC lets the cold New England air filter through the facility, cooling the data center from outside.
"We just use Mother Nature to keep us cool 70% of the time," Goodhue said.