This tip is part three of a series on using data center infrastructure management tools. Read part four: Measuring total energy use for an efficient data center.
For a data center infrastructure management (DCIM) platform, or any other energy-monitoring system, to gather and analyze energy usage information, it needs to have energy-monitoring hardware installed at each of the key points of the power and cooling infrastructure. This involves electrical work for the installation of energy-monitoring hardware.
For facilities-based systems, the most common energy-monitoring method requires the installation of current transformers (CT) and voltage monitors, also called potential transformers (PT), into the electrical panels feeding the equipment. These CTs can sometimes be “clamp-on” types, which can be retrofitted without interrupting the power to the load. The clamp-on CT is more expensive than the solid-core, pass-through type of CT, which requires disconnecting the load during installation if it was not originally pre-installed in the panel. For that reason, most retrofit installations tend to use the non-disruptive clamp-on type CTs.
However, some larger installations require a solid-core type CT and an interruption of the circuit or the entire panel, which is obviously not the favored choice for a data center. It is important to note that depending on the level of granularity expected, a solid-core type CT will determine the number of individual points to monitor and the corresponding quantity of CTs required.
For cooling systems in particular, it would be ideal to measure each subsystem component individually to help optimize the operation and efficiency of the cooling system. For example, in a chilled water system it’s best to separately measure the chillers, pumps and cooling tower fans.
In the data center white space, each computer room air conditioning (CRAC) or computer room air handler (CRAH) unit should be instrumented. If possible, even measure the reheat and re-humidification system energy separately. It is amazing how much energy can be saved by simply broadening the humidity set points. Some of the newer CRAC and CRAHs have built-in monitoring and logging, and have this information stored so that it may be harvested via Simple Network Management Protocol (SNMP) or Modbus.
Energy monitoring at the rack level
DCIM platforms are not just limited to the facilities infrastructure. The most common practice to measure the total IT energy consumed is to obtain it directly from the uninterruptable power supply (UPS) output readings (via SNMP or Modbus). While this approach does not include any downstream losses, it is typically the simplest and least costly method. Of course, measuring energy usage at the rack level and for individual devices is the most recommended way to measure energy.
Measuring energy consumption at the rack usually requires installation of the so-called “intelligent” rack power distribution units (iPDU). Originally, iPDUs were primarily sold to IT departments to prevent them from overloading the circuits and tripping the circuit breaker either inside the PDU or upstream at the power panel. “Managed” rack-level PDUs came into the market later, and allowed power to be cycled remotely (at the individual receptacle level) so that you could reboot a hung server without traveling to a remote site. This type of fully-managed PDU became more popular as many organizations moved out of local data centers and into colocation sites.
If the decision is made to do rack-level monitoring, then additional cabling and network ports for the energy-monitoring hardware needs to be reserved for each iPDU. This can represent a significant additional expense, especially if the iPDU uses ports on a high-end production network switch. Alternatively, a separate management network consisting of end-of-row, low-cost, stackable-type switches may be used for this power-management network.
Some vendors are also offering wireless connectivity as an option. For example, one vendor has a line of iPDUs that can communicate via wired Ethernet network or wireless with a plug-in USB-based wireless WiFi (802.x) transceiver to any industry standard WiFi access point. This allows the same product to be easily used as either a retrofit, or for new installations. This is especially cost effective since the same iPDU can be deployed for a wired or wireless network, or any combination thereof.
Other vendors offer proprietary mesh network-based communications and temperature instrumentation via a gateway, which also eliminates the need for cabling and network ports. There are also a number of vendors that offer mesh-type wireless power sensors, some of which can measure power inline at the rack, but must also be installed by first unplugging the existing basic PDU. These wireless power sensors simplify the installation and avoid the cost of a new iPDU and additional network ports or cabling.
Alternately, some IT vendors are offering systems that can forgo the need to have intelligent power strips at the rack level to provide the power measurement for the rack. Under this scenario, the DCIM platform communicates directly with the IT equipment’s own internal management firmware. In fact, most modern servers are capable of measuring, reporting and even controlling the amount of power used by the hardware, as well as its temperature, without needing external power-measurement equipment such as iPDUs. This valuable capability has been long overlooked by many IT administrators.
Some vendors offer software that can accept data from other vendors’ equipment, such as a UPS or CRAC, while others offer both hardware sensors and software. DCIM functionality is even available as a hosted service, wherein the sensor information is sent over the Internet to hosted systems and viewed over a Web interface. In some cases you also have the option to buy the DCIM software and install it your own local or remote server.
As previously noted, retrofitting energy-monitoring hardware may require de-energizing some equipment. Not a good thing, depending on the existing level of redundancy in the data center. In some cases, “hot work” is required to avoid shutting down an electrical panel or equipment. Some organizations do not allow hot work in the data center, since it poses a potential risk to personnel and introduces the possibility of an unplanned outage.
In some data centers, especially colocations, where the original design included energy monitoring and management, the floor-level PDU contains branch circuit-monitoring hardware and interface subsystems that can report usage per circuit. This report eliminates the additional cost and communications requirements necessary when you have individual iPDUs at each rack. However, it does not eliminate the need, or provide temperature monitoring at the rack level, in high-density installations.
To transmit the data collected from the energy-monitoring hardware to the DCIM platform, some form of connectivity is required. In the IT world, the common choice is SNMP over Ethernet LAN networks, which is well understood and widely implemented. It can be transmitted via the production network, but preferably it should be transmitted over a separate management network or virtual local area network.
In the case of facilities-related equipment and building management systems (BMS), it is the Building Automation and Control Network (BACnet), which was originally developed by ASHRAE and interfaced and transmitted via Modbus, commonly sent over dedicated serial communications lines. There is also the locally operated network,(LON) protocol, which can use unshielded twisted pair (UTP) cable similar to standard Ethernet cable, but without the Ethernet protocol. These are still the facilities departments' and cooling equipment manufacturers' protocols of choice. These systems tend to be vendor specific and propriety and, in many cases, make interfacing and multivendor communications difficult and expensive to implement. Recently, BACNet-TCP and Modbus/TCP has been slowly accepted by the facilities world, and they are being implemented to use standard Ethernet LAN IP-based networks via TCP/IP – but using the user datagram protocol (UDP).
Read part four of this series: Measuring total energy use for an efficient data center.
About the author:
Julius Neudorfer has been CTO and a founding principal of NAAT since its inception in 1987. He has designed and managed communications and data systems projects for both commercial clients and government customers.