Designing a data center's power system consists of numerous decisions about the components in the power path. In...
most of the world, there are two primary voltage schemes (three-phase) available, which are based on either the North American 480/208/120 V (600/208/120 in Canada) or the 400/230 V system in used Europe and some parts of Asia. In all systems, much higher voltages are used to deliver power from the utilities to the site, but these are not part of this discussion. Also note that we are generically referring to the 400/230 V system (this is the midpoint voltage that represents 380/220 V through 415/240 V). While some data centers are exploring the use of direct current (DC) to improve overall efficiency of the entire computing ecosystem, alternating current (AC) power is still the predominant form of power in the data center. (Follow this link for more on the AC/DC data center debate.)
Rack-level power density and distribution
An alternative option
Several manufacturers offer overhead "buss tracks" that allow you to simply plug in a module that contains circuit breakers and receptacles or power whips (single- or three-phase). They resemble track lighting on steroids. This provides an extremely flexible alternative and allows you to deliver almost any level and type of power to any rack, without pre-planning. This is gaining in popularity in newer data center designs.
Rising data center power density is one of the big factors driving the re-examination of voltage choice to IT equipment and which voltage to use in distribution systems.
Here in the North America, the common use of 120 V worked fine when a rack used 1-2 kW per rack and a single 20 A circuit was all that was needed (two for A-B redundancy). With the advent of blade servers, which typically require 208 V or 230 V circuits and use five or more kilowatts as well as racks full of 1U servers, the new baseline is now 5 kW per rack.
Ten to 20 kW is not uncommon anymore, and even 30 kW or more is not unforeseeable (we can provide power at these levels, but cooling is a much greater challenge). Moreover, almost all IT power supplies are now autosensing and universal voltage-capable (100-250 V) to allow the same product to operate worldwide. In fact, they are also more efficient at 208 V or 230 V than at 120 V (or even lower at 100 V in Japan).
We can increase the power delivered to each rack by increasing the voltage (or amperage) and also by running three-phase power to the racks. The diameter of the cable determines its "ampacity," or the number of amperes it can safely carry (and its cost). The voltage that is used will determine how much power can be delivered at different voltages over the same conductor size (example: 12-gauge wire, typically used for 20 A feeds for distances of up to 50 feet).
Note that under North American Electrical codes, the branch-level circuit breakers are 80% rated, so that only 16 A can be delivered to the load.
Branch circuits: Single-phase power distribution
|16||120||1.9||L1 + N + G|
|16||208||3.3||L1 + L2 + G (across any two of three phases)|
|16||230||3.7||L1 + N + G|
Branch circuits: Three-phase power distribution
|16||120||5.7||L1 + L2 + L3 + N + G (120 V any phase to neutral)|
|16||208||5.7||L1 + L2 + L3 + N + G (208 V across any two of three phases)|
|16||400/230||11||L1 + L2 + L3 + N + G (230 V any phase to neutral)|
Note that by making three-phase power available in the rack, you will increase the available power by a factor of 300%, yet increase your cable conductor count and its cost by only 66%.
Moreover, by deploying three-phase 208/120 V power to the racks, you can supply either 208 V single-phase or 208 V three-phase power and also provide 120 V for older or specialized IT gear (that may only work on 120 V).
In fact, by running three-phase power, some rack-level PDUs (aka the rack power strip) can provide 208 V and 120 V simultaneously from the same strip.
Instead of hardwiring, also consider using three-phase connectors such as NEMA "Twist-Lock" L21-20 or L21-30 (20 A/5.7 kVA or 30 A/8.6 kVA, respectively) or the larger IEC 309 40-60 connectors (also called "pin and sleeve"), or even Russell Stove for higher power. This will permit you to change power strips as your equipment changes, without rewiring. While this is somewhat more expensive up front, it can save a lot of money in the long run by providing an easy and lower-cost solution to moves, adds and changes during equipment upgrades.
While on the subject of rack-level PDUs, please consider using units that allow remote monitoring to prevent overloads and also allow for energy management and capacity planning.
In the 400/230 V system, all output circuits are 230 V single-phase (from any phase to neutral and ground).
Floor or row-level power distribution
Depending on the size of the data center and the amount of power required (and power density), power can be distributed at 480 V or 208/120 V in North America. Assuming that you have larger installation, 480 V is the most common and preferred choice for the UPS and all major power distribution until it gets to the data center floor.
Once the 480 V power has been delivered to the floor or row PDU, it needs to be transformed down to 208/120 V for use by the computer equipment.The type of the transformer will impact its efficiency and the overall efficiency of the data center.
Transformer types and the K-Factor
No discussion of power distribution would be complete without a mention of transformer types and the "K-Factor." Check out this tip for more info on data center transformers.
In a 400/230 V distribution system, there is no transformer required, only circuit breakers to protect the branch circuits. European sites sometimes specify a transformer in the PDU to provide addition isolation and also to mitigate the effects of phase imbalances on the upstream UPS, especially if it is a transformerless UPS.
The North American 208/120 V distribution system also does not require a transformer, only circuit breakers to protect the branch circuits. However, data center designers will sometimes specify a transformer in the PDU to provide additional isolation and mitigate the effects of phase imbalances on the upstream UPS, which is especially true if is a transformerless UPS.
As noted in the discussion of the rack-level power, the higher the voltage, the lower the current required to deliver the same power to the load. The lower amperage will also lower the size and cost of the electrical switchgear, UPS, distribution panels and copper cabling used throughout the entire system. This can amount to a significant overall savings.
Current required to deliver 300 kVA to PDU at different voltages
Conversely, here is a chart to show the effect of a fixed current capacity (such as based on existing wire size) versus voltage.
Power delivery to PDU at different voltages*
* Example uses 400 A-rated circuits and feeder cabling.
This is useful if you are considering a voltage upgrade by retrofitting the distribution system and want to save money and construction time by re-using the existing cables and conduits to the PDUs. For example, converting existing feeder cables from 208 V to 480 V, you can deliver over twice the power over the same cable -- just make sure the panels and switchgear are rated for the higher voltage.
Safely hazards and voltage
In North America, we commonly use 208/120 V to end-user equipment using standard plugs and receptacles. It is also a common practice for electricians to add circuits to live 208/120 V distribution panels. The 480 V service has a much higher potential for an electrical arc to occur and is therefore not considered safe for plug-in equipment. At 480 V, the danger of Arc Flash is substantially greater -- electricians require additional safety gear to work on 480 V circuits, and the possibility of service interruptions is higher due to an arc occurring during electrical work in the panel.
Will European voltages work in U.S. data centers?
In Europe, only single-phase 230 V is distributed to plug-in devices via standard IEC C13- and C19-type receptacles and plugs, at up to 16 A. However, three-phase 400 V power is also commonly available via the larger IEC type 309 receptacles at up to 60 A. Also in Europe, 400 V work in the panel is commonly done (with appropriate safety gear), since that is the basis of all their power distribution systems.
Historically, which side of the ocean you occupy has dictated which voltage has been used in a given country and in data centers. However, the inside of the data center has become its own microcosm, sometimes independent of its location. It is clear that European data centers will continue to use the 400/230 V system since it is already native to their overall existing power system.
One of the major advantages of the 400 V European system is that there is no voltage conversion; therefore there are no additional transformers required for voltage conversion (other than the main utility transformer). This makes the entire power system potentially smaller and more efficient overall.
In North America, several vendors now offer 400/230 V products as a higher-efficiency alternative to traditional 208/120 V distribution systems. They typically use an "autotransformer" (which is smaller, lighter and more energy efficient than a traditional stepdown/isolation transformer) in the floor- or row-level PDU. This allows these PDUs to work with a standard 480 V UPS and 480 V distribution system that is carried to the PDU, and the PDU then outputs 230 V (single-phase) to the IT racks. It offers greater efficiency and a smaller footprint on the data center floor. Some vendors have created 400/230 V "touchless" modular PDU systems that shield the main buss, allowing circuit packs (breakers and cabling) to easily and safely be added and removed.
In a more advanced 400 V power scenario, the UPS would be fed at 400 V and there could be no transformer in the PDU. The main input power to the data center and all the power equipment's switchgear, generators, etc., would be 400 V; potentially it would be just like a European data center -- the North American high-utility voltage would be transformed only once to down 400 V (instead of 480 V). Afterward, there would be no need for a transformer to step down the voltage, theoretically avoiding all transformer losses and minimizing copper cabling losses while allowing the IT power supplies to operate at 230 V, at which they are the most efficient.
In so far as 400 V power taking hold in North American data centers, it may take many years or it may never take hold as a mainstream system. There is massive mindset change required by all those involved in the designing, building and operating of the data center. Moreover, the existing equipment has inertia and very few people or equipment makers may want to make such a major change in the hope of gaining a 2-5% potential increase in energy efficiency. Then again, stay tuned -- if energy prices continue to rise, it may cause everyone involved in the data center to examine every option.
About the author: Julius Neudorfer has been CTO and a founding principal of NAAT since its inception in 1987. He has designed and managed communications and data systems projects for both commercial clients and government customers.
What did you think of this feature? Write to SearchDataCenter.com's Matt Stansberry about your data center concerns at firstname.lastname@example.org.