Is your uninterruptable power supply really near capacity, or is it just alerting you to a phase balance issue in your three-phase power system?
Many a UPS has been replaced because an uneven power draw gave the appearance the UPS was near overload. An unnecessary overnight replacement of a big UPS wastes money, disrupts an entire data center and exposes it to potential catastrophe. Follow these tips to avoid phase balance trouble in the data center.
The basics of three-phase power
Unlike your home, which uses single-phase power, commercial buildings are served by three-phase power. Large UPS systems — say 20 kVA and above — are generally three-phase because it’s a more efficient way to deliver large amounts of power.
When you read the display on a three-phase power UPS, the default, or home, display will usually show the percentage load on the highest loaded phase. For example, if a 100 kW UPS is reading 98%, that might be the load on Phase B. But if Phase A is only seeing a 50% load, and Phase C only 20%, that’s only 54% of the total UPS capacity. There is 46 kW available to power equipment. Time to replace the UPS? I think not!
There are different ways of electrically configuring three-phase service, but in the U.S. the output of a data center UPS — its downstream power distribution units (PDUs) — will almost always be 120/208 volt Wye-connected four-wire + ground. This is shown in Figure 1 without the ground wire.
Figure 1 – Circuit types available from three-phase, 120/208 V Wye output
Where high power capacity is required, there are many advantages in the use of three-phase delivery. In a modern data center, where cabinets often draw 5, 10, 15 or more kilowatts, fewer conductors are needed to deliver the same power level as a single-phase system, which is why more data centers are running three-phase circuits all the way to the cabinets. See Figure 2 for a table of conductor counts.
|CABINET CAPACITY||1,920 W||5,757 W||8,636 W||17,272 W||25,908 W|
|Conductor size||#12 AWG||#12 AWG||#10 AWG||#10 AWG||#10 AWG|
|No. of 120 V, single-phase circuits||2||6||6||12||18|
|Single-phase, 120 V, conductor count||4||12||12||24||36|
|No. of 208 V, three-phase circuits||0||2||2||4||6|
|Three-phase, 120/208 V conductor count||N/A||8||8||16||24|
Figure 2 — Conductor counts for different cabinet power capacities and circuit types: dual circuits per cabinet, two-wire single-phase and four-wire three-phase. Grounding conductors are additional.
At today’s copper prices, this is not trivial but regardless of how the branch circuits are wired, if the power going into and out of the UPS is three-phase, it is necessary to draw power as equally as possible from each phase to utilize the full capacity of the system.
Understanding phase balance
Imagine your three-phase UPS as three water canteens carried by soldiers in the desert. They need all three people and their equipment to carry out their mission. Each starts with the same amount of water, which is more than they should need for the journey, but one person doesn’t ration very well and runs out before they have reached their objective. Do they just keep going until that soldier dies and then abort the mission? Do they call for a replacement team and tell the next team to carry more water? Or do they reapportion what they have and then better manage the rest of the mission to get maximum use out of their critical resource? Hopefully, the answer is the latter.
A three-phase UPS works the same as the soldier’s canteens. Each phase has the same capacity, one third of the total. If only one phase is loaded to capacity, there’s no reason to buy a new UPS since there’s still enough total capacity in the other phases to handle the load. We just need to re-allocate our power demand on the three phases so we can utilize the full capacity of each. Then we need to manage our loads as we add equipment so we don’t repeat the same problem and cause the UPS to die before its time. So far, we have concentrated on the UPS, but the same principle applies to every element in any three-phase power chain.
How to balance loads in a three-phase power system
Phase balance simply involves moving loads from one circuit to another, in much the same way you would move appliances around at home if you overloaded a circuit.
Let’s say you have several small appliances plugged into a single outlet in your bedroom — lamps, hair dryer, curling iron, etc. — and you buy a large TV. Summer comes and you need to put an air conditioner in the window above the outlet to which you already have all these things connected. It all works fine until one hot summer morning when the air conditioner is running full blast, you’re watching the news, someone heats up the curling iron and starts the hair dryer. Suddenly, they all quit. The circuit is overloaded, and a breaker trips in the basement electrical panel. Does this mean you’re out of power for the whole house?
Of course not! You’ve simply overloaded one circuit. There’s still plenty of power available; you just need to move some of the load around to use it. Find an outlet on another circuit for the curling iron and hair dryer. Technically speaking, you need to reallocate the load to better balance it across your available capacity. Load-balancing your data center is the same process, except the computer loads are relatively constant and you need better information to get the job done.
In the home bedroom example we were trying to keep a circuit breaker from tripping, but in our data centers we’re trying to get the loads as close to the same wattage as we can on all three phases of our UPS, as well as on any downstream three-phase devices. That means we need information. If we simply drill down using the UPS control panel, we can read the actual load on each phase — not just the highest one on the default display — and see how far out of balance we are. If your UPS reads both kW and kVA loads be aware that we’re most interested in the kW loads, even if the kVA readings are higher. But the numbers should still be pretty close to the same, whether you’re reading kW, kVA or amps.
Once you know which phases are high and which are low, you need to find out which cabinets or power strips are on which phases. Unfortunately, you might need an electrician to figure that out if the information isn’t on drawings or labeled on each cabinet or receptacle. That could require short-term shutdowns of individual cabinets, but if your equipment is all dual-corded and is on different circuits this shouldn’t be a big problem.
There are two routes to take after you have your phases sorted. The first is trial and error, where you just move plugs to outlets on under-loaded phases until the balance gets better. There’s nothing wrong with this as long as it ends up improving the load balance. It just takes time and may require a number of equipment shutdowns before you get it right.
The second — far superior — way to attack a phase-balance problem is with actual readings. This means monitoring the power draws from each cabinet — or even from each receptacle — as well as at each major point in the electrical distribution chain from the UPS through any PDUs and all the way to the cabinets. That’s the major justification for the expensive power strips now used in data center cabinets, variously called PDUs (which confuses them with the big conventional power distribution units), electronic PDUs (ePDUs), cabinet distribution units (CDUs) and rack PDUs.
By any name, these sophisticated power strips enable you to read how much current you’re drawing on each circuit in a cabinet in real time, either directly from built-in digital displays or remotely with a network interface. Armed with this information, you can plan your re-circuiting on paper. And if you’re going to maintain phase balance going forward, power monitoring is essential to planning power availability before making equipment additions.
There is another factor that could throw a wrench in things. To improve energy efficiency and reduce copper wire usage, more data center equipment is on 208 V power in the United States. This is derived by connecting to two of the three phases of a three-phase circuit like the one in Figure 1. With this system, moving loads becomes a “two-out-of-three” puzzle. Every time you move a computer cord or cabinet connection, you will be changing load on two phases at a time instead of one. That means one of the phases will stay the same, the load on another phase will go down, and the load on the remaining phase will go up. Figure 3 shows what might happen when moving a 5 kW, 208 V load on a 100 kW UPS (33.33 kW per phase):
|UPS PHASE||ORIG. LOAD||% LOAD||LOAD TO MOVE||MOVED LOAD||NEW LOAD||% LOAD|
|Phase A||32 kW||96%||5 kW||27 kW||81%|
|Phase B||30 kW||90%||5 kW||5 kW||30 kW||90%|
|Phase C||15 kW||45%||5 kW||20 kW||60%|
Figure 3 — Effect of moving 208 V, single-phase loads on a three-phase UPS
We’ve improved the situation on Phase A and put more load on Phase C, but Phase B is still high. This means we need to find another 208 V circuit that’s on phases A and B and move it to a circuit on phases A and C. In other words, while 208 V operation is more efficient and cost-effective and is certainly recommended, it requires more careful power management to achieve phase balance and full UPS utilization. As a side note, this issue is one factor that favors going to 415/240 V European-standard power in data centers — 240 V circuits move one phase at a time, just like our 120 V circuits.
No matter what voltage you’re using in your data center, it’s important to keep the draw on all three power phases as close to the same as possible to get the most capacity and energy efficiency out of your expensive UPS. It may take a little effort, but it’s much better than the cost and operational disruption of a “fork lift” UPS upgrade.
ABOUT THE AUTHOR: Robert McFarlane is a principal in charge of data center design for the international consulting firm Shen Milsom and Wilke LLC. McFarlane has spent more than 35 years in communications consulting, has experience in every segment of the data center industry and was a pioneer in developing the field of building cable design. McFarlane also teaches the data center facilities course in the Marist College Institute for Data Center Professional program, is a data center power and cooling expert, is widely published, speaks at many industry seminars and is a corresponding member of ASHRAE TC9.9 which publishes a wide range of industry guidelines.