An electric arc in the data center can damage equipment and shut down power. Worse, it can also have lethal consequences...
for electricians working on live electrical panels. Proper design can reduce arc-flash hazards as well as avoid the need for critical power interruptions.
Electric arcs result from ionized air breaking down into plasma that carries a stream of current between conductors or between a conductor and ground. If the arc is big enough and occurs quickly enough, it is called an arc flash. The heat, light and shower of sparks is very dangerous, sometimes fatal, which is why working on “hot” panels is becoming more rare.
With great power comes increased arc flash danger
Electric arcs have been a concern since we first harnessed electricity, but until recently they were not considered an actual hazard, either in our homes or in our data centers. Arcs lengthen and intensify as voltage and current increase. The astronomical growth in data center power use — and the huge electrical supply sources they require — creates the potential for bigger and more dangerous electric arcs, and more places where they can cause damage.
Any electric arc is potentially harmful and will result in a circuit breaker trip and power interruption. If an electrician working on your home breaker panel lets a grounded wire touch the main (energized) bus, the contact will send out sparks that could burn and injure, but it’s unlikely to kill.
An electrical fault in major switchgear of a data center can create an arc flash, when the current rises very rapidly and creates an explosion. A small flash can make a startling noise and create a shower of hot sparks. But a large arc flash creates a deadly fireball of 35,000 degrees Fahrenheit or more, sending out an eruption of molten metal. The pressure wave from an arc flash can even blow out eardrums or damage lungs, and the “flash” emits large doses of infrared and ultraviolet light that can blind temporarily or even permanently. This is not something that can be taken lightly, and it no longer is.
Arc flash and data center implications
Until recently, electricians and manufacturers’ field technicians were willing to work on live panels. But the dangers of arc flash have become so significant — from both safety and legal liability standpoints — that many technicians are prohibited from working on live power equipment without special precautions. Many vendors of UPSes, large power distribution units (PDUs), automatic transfer switches (ATS) and in-row power centers, as well as unions, building owners and maintenance firms, now have absolute policies regarding arc-flash hazard exposure and procedures.
Arc flash has become such a major concern that legally much of the electrical distribution equipment in a data center should now carry “Arc Flash Hazard” labels. For data center operators, this means you may be told to shut down power when maintenance is required or when a new circuit needs to be installed. Simply removing a panel cover may even require a shutdown.
In new data centers the problem can be minimized through proper, knowledgeable design and installation. But in legacy data centers that weren’t designed with the problems of arc-flash exposure in mind, you could be required to shut down both circuits to a row of cabinets just to work on one of them if the associated panels can’t be opened individually. This can be a major surprise and a serious problem. It all depends on how your data center was wired, from the incoming utility service to the distribution cabinets.
Arc-flash hazard protection often not wanted
The alternative to power shutdowns is for the electrician to wear special clothing, called an arc-flash hazard suit, which protects against fire and heat. Arc-flash hazard gear looks a lot like a space suit. While it protects the wearer, it is hot and uncomfortable, and an electrician may not agree to “suit up” unless it’s really necessary.
A Pueblo Chemical Agent-Destruction Pilot Plant crew electrician helps a co-worker put on an arc-flash protection suit in preparation for an electrical inspection. Image courtesy of Assembled Chemical Weapons Alternatives.
The bulky suit and gloves can increase the chances of creating an accidental flash, tripping circuit breakers and causing unplanned shutdowns. That means significant time to repair damaged switchgear as well as to restore computing systems. The problem increases dramatically on smaller panels and equipment that require adjustments with small tools not handled easily with large gloves.
New codes, equipment aim to reduce arc-flash hazards
Concerns over arc-flash hazards have increased dramatically with the amount of power we need to supply. The electrical service to our homes has grown from about 60 amps not so many years ago to 200 amps or more in order to support all our electrical appliances. For some years, electrical code has required Ground Fault Circuit Interrupter (GFCI) outlets and breakers in kitchens, bathrooms, outdoors or anywhere water may be present.
But newer electrical codes now also require Arc Fault Circuit Interrupter (AFCI) outlets in many places because the current draws of our major appliances can create arcing conditions severe enough to start fires. If the concern has become this great in homes, consider how much greater it is in the data center. One hundred kilovolt amperes (kVA) was once considered a lot of power for a data center, but a megawatt uninterruptable power supply (UPS) with 300 kVA PDUs supplied by 480 to 600 V feeders is almost commonplace.
A short in your home power panel may be dangerous, but in a big PDU or UPS it could be costly and even deadly. Even the circuit breakers themselves have become a concern. The contacts inside a breaker are designed to operate very fast to minimize the arc. The plastic case is designed to contain any “flash” that might occur if the breaker malfunctions. But things can still go wrong, and circuit breakers have been known to explode. Many electricians will shield their faces just to operate a small circuit breaker. Larger breakers and switchgear only increase this concern, and hazard suits are now required in many places to operate a big main breaker.
Steps to minimize arc-flash hazards
It is possible to design for minimal arc-flash exposure, but this could mean a major electrical project in existing data centers. If you’re designing a new data center or upgrading an existing one, there are two important questions you should be asking.
1. Since most everything today is dual-powered, will you be able to shut down one power source to work on it without affecting the other? Or will the design and the power centers require shutting down both panels to work on just one?
2. Has a professional arc-flash hazard analysis been done by a qualified electrical engineer experienced in the subtleties of arc-flash calculations and familiar with data centers?
There are four levels of arc-flash hazard specified in National Fire Protection Association 70E Standard for Electrical Safety in the Workplace, which are also cited by the Occupational Safety and Health Administration. Unfortunately, common electrical installations tend to result in worst-case Level 4 hazard conditions, when many situations could be eliminated, or at least reduced to a less onerous Level 1 or Level 2, using more carefully designed electrical systems. There are potential arc-flash hazards all through the power chain and not all can be mitigated, so power shutdowns may still be necessary. But let’s look at one situation that is frequently overlooked, but can be easily avoided.
In both new data centers and retrofit upgrades, breaker panels for the A and B circuits to the cabinets should be separated. If individual panels are mounted on a wall, it’s easy to keep them apart, but this can become more problematic with factory-built power centers such as the newer “end-of-row” distribution cabinets that have become popular. These cabinets can contain as many as four breaker panels – two in the front and two in the rear. One might assume that they are all fully isolated. Never assume!
Let’s say these four panels are serving a high-density cabinet row that requires all four panels to handle the load. Logic might tell you to wire the A and B circuits for one row to the front panels, and the A and B circuits for another row to the back panels (Figure 1). But what if you can’t open one panel without also exposing live parts in the adjacent front or rear one? In order to work on one panel, you would need to shut down both which, in this design, would result in cutting all power to the row. This is not the intent of dual-powering and negates your investment in dual-circuiting.
Figure 1: Dual circuits wired to adjacent panels in power management module
One solution would be to put A circuits to each row on the front panels and B circuits on the rear panels, assuming those are isolated (Figure 2). It would be even better to ensure each panel can be opened individually, but this may not be possible with some power center designs. A thorough bid specification should at least include a requirement for the necessary electrical barriers, which are often optional.
Figure 2: Dual circuits wired to panels on opposite sides of barrier
You must carefully review the equipment submitted by the contractor for conformance since it may be different from what the specification was based on. There should always be good supervision of the electrical installation as well since electricians won’t necessarily know the importance of these details and may wire the way they find easiest. In short, avoiding shutdowns requires thorough detailing, well beyond fundamental electrical design.
An analysis can reduce arc-flash hazards
One simple way to get started with arc-flash prevention is to study your data center designs with facilities electricians. Review what panels, switchgear, UPSes or ATSes may need to be shut down in order to make repairs or perform preventive maintenance and understand what computing systems might be affected.
But there is no substitute for a formal arc-flash analysis. It’s not a simple job, but if it’s done right — especially for a new electrical design — it might identify ways to mitigate arc-flash hazards and identify work that could be done without either a shutdown or a hazard suit. There are even power centers available that are specifically designed for arc-flash suppression so that circuit changes can be made on live equipment without concern with equipment from companies such as APC and Layer Zero.
A thorough arc-flash analysis requires a comprehensive short-circuit study first. While this is a standard requirement of the electrical design process, an arc-flash analysis must evaluate the electrical design from a different perspective to mitigate arc-flash hazards. A breaker coordination study must also be done with the same goal in mind, even though it may have been done before. Breaker coordination ensures that the breaker nearest the fault condition trips first. If done with arc flash in mind, a breaker coordination study could help you select different circuit breakers or fuses than might normally have been chosen.
If someone claims to be able to do an arc-flash analysis without doing the short circuit and breaker studies, it would be wise to seek a second opinion. Whoever is doing the studies should also have an understanding of data center power requirements and reliability concerns.
In short, unless both the designer and the installing electrician understand the issues of arc-flash hazards, you can make a large investment in dual-distribution and still find yourself exposed to accidental or intentional shutdowns.
ABOUT THE AUTHOR: Robert McFarlane is a principal in charge of data center design for the international consulting firm Shen Milsom and Wilke LLC. McFarlane has spent more than 35 years in communications consulting, has experience in every segment of the data center industry and was a pioneer in developing the field of building cable design. McFarlane also teaches the data center facilities course in the Marist College Institute for Data Center Professional program, is a data center power and cooling expert, is widely published, speaks at many industry seminars and is a corresponding member of ASHRAE TC9.9 which publishes a wide range of industry guidelines.