As data center designs evolve, cabling installations must adhere to established standards, such as TIA, and judiciously incorporate innovative yet disruptive designs, such as Open Rack.
Multi-socket, multi-core servers have higher bandwidth needs than ever, hosting 10, 30 or even more virtual machines. The entire server deployment demands more flexibility and scalability as well; multiple server racks and supporting equipment deploy simultaneously. With pre-wiring and configuration, installations take hours rather than days and demand adequate space and mobility. Network virtualization technologies also affect how network devices are deployed, configured and managed.
There are four issues to consider for data center cabling: bandwidth, modularity, connectivity and the effects of virtualization on computing resource utilization.
A long-term cable plan
TIA-942-A-1 is a comprehensive document referencing a wide range of data center considerations. TIA's standards on optical and copper telecommunications cabling systems and switch distribution take into account data center performance, roadmap and budget.
The TIA standard defines four principal spaces for the data center network
1. An entrance room to segregate equipment from outside providers.
2. A main distribution area (MDA) connecting the entrance room to core routers and backbone switches via high-bandwidth backbone cabling.
3. Horizontal distribution areas with local LAN and storage-area network switches connected to the MDA via backbone cabling in branches, and to the racks and cabinets in the equipment distribution area (EDA) via twisted-pair copper Ethernet cables.
4. An optional zone distribution area between an HDA and EDA for configuration use, or free-standing gear.
Design cabling and switches to support all of the available connectivity options from any rack location in the facility: gigabit Ethernet (GbE), Fibre Channel or unified fabric like Fibre Channel over Ethernet. Otherwise, bandwidth and connectivity upgrades will be long and disruptive. TIA defines how to rack cables to separate optical and copper media and segregate power cabling. It dictates distances between cable types, bend radius limits and other considerations.
TIA recommends laser-optimized 50 micron multi-mode optical fiber for backbone cabling and twisted-pair for horizontal cabling. Data centers should choose the fastest feasible copper for deployments because Ethernet cabling is backward-compatible with earlier cable categories. Investing in better cables today prevents horizontal re-cabling later when Ethernet port speeds increase.
Network backbones start at 10 GbE, and should include a roadmap to 40 and 100 GbE, as growth will affect the cabling and switching infrastructures. However, not every data center needs to support 100 GbE, and therefore investments in higher-value cables should be justified against cost and roadmap.
The TIA-942 standard provides guidance on cabling distances and protection. For example, backbone optical cabling is typically limited to 300 meter runs, while copper horizontal cabling is restricted to 100 m or shorter runs. Data centers standardize on end-of-row (EoR) or top-of-rack (ToR) schemes. With EoR, cables run from racks to a row end where switching occurs. EoR creates different cable lengths -- racks farther away need longer cables -- but more servers per rack -- switching is concentrated at the end of the row or racks.
The ToR approach puts switching at the top of each rack, making racks uniform and swappable. The rack switch is interconnected to the greater local area network (LAN) using fewer high-bandwidth cables, such as optical fiber. It tolerates a transition to faster Ethernet standards better than conventional twisted-pair copper. ToR enables rack additions and changes with fewer re-cabling tasks.
Disruptive wiring ideas
Emerging concepts, like the Open Compute Project's Open Rack, are shaking up how data centers connect servers.
Open Rack's physical rack standard aims for operational efficiency in large data center deployments.
Rack implementations are segregated into 48 mm segments called an OpenU with three 13 OpenU zones -- 10 for servers and three for power distribution units. Power is distributed within the rack through bus bars, eliminating individual server power supplies.
Intra-rack connections target extremely high speeds, using optical cabling and Intel's new photonic connector. A two OpenU space at the top of the Open Rack design allows for a ToR switch.
Open Rack places data cabling and fault indicators on the front for easy access from the cool aisle.
TIA-942-A-1 is comparable to European standard EN 50173-5 and international standard ISO/IEC 24764 for structured cabling systems. The Building Industry Consulting Service International (BICSI) also supports cabling and data center standards with BICSI-002 on data center design and NECA/BICSI 568 on cabling installation.