This is the fourth chapter of our data center construction runbook. Each chapter outlines a specific aspect of data center design and construction and walks you through the step-by-step process of a data center buildout. Chapter 1 focuses on data center site selection, Chapter 2 discusses selecting a data center design team and Chapter 3 addresses working with data center contractors.
As you undergo a data center construction project, your choice of data center commissioning agent may be as important as your choice of design firm. Data center commissioning is a rigorous, systematic process to ensure that you get the data center design, equipment interoperability and system redundancy that you paid for. Thus the central role of the commissioning agent firm is to advocate for your data center's needs and agreed-upon requirements.
Independent versus internal commissioning teams?
This job is preferably handled by an independent engineering firm, and not one affiliated with the primary data center designers -- but it doesn't always work that way.
"Somebody has to do an objective evaluation," said Robert McFarlane, an expert in data center design and a principal at New York-based Shen Milsom Wilke, Inc. "There has to be more than a theoretical arm's-length separation between the commissioning and design firm. If it's going to be the same company, they need to demonstrate to me that the commissioning is not just a rubber stamp of what the engineering team does."
But working with multiple design teams has its own challenges. "Every time I try to bring separate companies together, there are mistakes and reluctance to share information," said Pete Sacco, the president of Franklin Lakes, N.J.-based PTS Data Center Solutions.
If a single company does the design and the commissioning, the biggest advantage for the data center operator is that there is only one throat to choke, said Randall Hayter, the director of technical operations at Lee Technologies Inc. "They designed it, built it, and they told you it was good."
The most important role of a commissioning agent is to advocate for the owner's project requirements. Bringing in an outside engineering firm to review the design and installation processes is the surest way to mitigate the risk of project failure. Data center pros need to budget for the additional cost of retaining an outside firm and weigh it against the costs incurred from potential risks.
The data center commissioning process
There is no one way to commission a data center, though many engineering firms loosely adhere to the general ASHRAE guidelines for commissioning buildings. The project is typically done in five steps.
Step one. During this phase, data center stakeholders review of the design intent in the construction documents. In certain situations, commissioning agents observe the manufacture of the large mechanical components that will be installed in a data center.
Do you really need to pay a commissioning agent to go to every factory before the equipment is shipped? If you install commercial computer room air conditioners right out of the catalog, 100,000 built and installed, the answer is no. But if you purchase prototype or custom models, it may be worth ensuring that a manufacturer provides what you've paid for and that what's provided operates according to the specifications.
Step two. During this phase, your data center undergoes a site inspection to ensure that you received the equipment specified in the submittal documents. A commissioning agent makes sure that mechanical components are installed with the proper wiring, piping, weight requirements, etc. This includes air conditioners, pumps, chillers -- any piece of equipment that has a submittal form.
Step three. In this phase, a data center must ensure that the equipment was installed properly. Installation errors are some of the most common problems. "Very few contractors have experience building mission-critical data centers," McFarlane emphasized. "You're going to have contractors mounting switches and valves where it's easiest. It doesn't mean it's in the right place."
The commissioning agent should be on-site for spot checks on a regular basis, throughout the construction process. "You don't want to come up with a 200-page list of things that weren't done correctly at the end when everything is already put together," McFarlane said.
Step four. During step four, the commissioning agent begins component testing to design loads. Each individual component has its own checklist. At a minimum, the commissioning agent will start uninterruptible power supplies (UPSes) and power distribution units to verify that they work, run step load changes on UPSes, conduct battery run-time tests to make sure they hold charge. According to Sacco, commissioning agents can't start the air conditioners till the room is clean -- the air has to be particulate-free, or it will damage the equipment. That means the majority of construction must be finished at this point.
Step five. This phase involves system integration testing at full design load, testing the interoperability of the system. The commissioning agent works off a script and tests all the what-if scenarios for redundancy.
Often this involves bringing in load banks, which are basically giant modular toasters used to simulate racks of servers. The resistive coils suck the electricity out of a UPS, turn it into heat, and test the computer room air conditioning, or CRAC. Engineering firms have various ways to simulate your planned server environment.
Avoiding commissioning pitfalls and mistakes
During the data center commissioning process, there are some common errors to avoid, such as the following:
Bringing the commissioning agent on board late in the project. The commissioning agent should be involved from the early stages of a construction project. At key points in the design stage, drawings should be submitted to the commissioning firm. "The end of a job is the hardest time to get anything done," McFarlane said. "You can't go in after the fact and commission something," Hayter said. "If you think you can have some third party perform all of our commissioning after everything is constructed, really at that point, you've lost a lot of the value of the commissioning process."
Rushing the commissioning process. "A lot of people try to squeeze through the commissioning process as though it needs to happen flawlessly," Sacco said. "But if you're going to have a failure, have it now." Even if that means spending more time or budget on commissioning.
Backing down on the budget without balancing design. Often a commissioner finds mistakes or items for which budget was not allotted. The cost of the project increases and the client can't afford the fixes, so it is forced to reduce a facility's availability requirements. "The minute you back down the availability, you have to balance design," McFarlane said. "If you spent a bunch of money on redundant power but left the air conditioning vulnerable to cut costs, you haven't improved your tier rating. Contractors cut costs in the easiest way and often don't explain what's going on to the client. In this case, the commissioning agent plays a very valuable role."
Overlooking the EPO switch. According to Hayter, one of the most common things that can go wrong in a data center construction project is the emergency power-off (EPO) switch. "We've commissioned millions of square feet, and EPO switches almost never work the first time they are tested," Hayter said. "We've started looking at drawings, we want to see that someone has put some thought into the EPO system, has it specified right, installed right." Contractors overlook EPO switches, discounting them as glorified light switches, but if you can't make an EPO switch work, it's going to be hard to fix. "To re-certify an EPO in a live data center can cost millions," Hayter said.
Tips for selecting and working with a data center commissioning firm
When selecting a data center commissioning firm, review the documentation and commissioning scripts it has developed for previous clients. The commissioning agent scripts should be well written and thoroughly reviewed. "Demand to see what projects your potential commissioning agent has done," Sacco said. "If they don't have anything, it means they haven't done it."
As you do due diligence on your commissioning agent, check in with data center pros who have used the agent previously and find out how long it took to get the results delivered, McFarlane advised. "If I get the results three months after the work has been done, I'm already moved into the data center. Now what do I do? I can't take systems down to fix them. Now I've got a list of things that aren't right, and I can't do anything about it."
An independent commissioning agent can be an essential advocate for a data center owner throughout the entire data center design-build process. Bring the commissioning firm onboard early to make sure the project achieves its design objectives. And perform due diligence to make sure you select a firm with excellent documentation and a quick turnaround time on all reporting items.
ABOUT THE AUTHOR: Matt Stansberry is SearchDataCenter.com's senior site editor. Write to him about your data center concerns at [email protected].