You can trust your storage traffic to that pedestrian Ethernet network -- with gradual upgrades and replacements.
Fibre Channel over Ethernet (FCoE) is a means of exchanging Fibre Channel storage traffic over an Ethernet LAN. Encapsulating FC frames in Ethernet packets allows block storage traffic to share the wire with other LAN traffic. It simplifies the data center and lowers the cost of maintaining multiple networks.
The LAN/SAN shakeup
Storage and network traffic exist in different data center realms, each managed by different sets of IT administrators in specialized silos. Storage traffic relies on a dedicated Fibre Channel network, while local area network (LAN) traffic uses an Ethernet network.
Ethernet started as a relatively slow, low-performing, lossy network that exchanged limited amounts of client/server data. But as Ethernet speeds increase and reliability in moving data improves, organizations are rethinking separate networks.
Shared storage posed bandwidth and performance challenges, such as prioritized flow control, which Ethernet implementations could not support. A storage area network (SAN) requires technology that prioritizes data, avoids packet loss and meets other performance needs. Therefore, data centers segregated SAN traffic on Fibre Channel.
Two distinct networks are expensive to maintain and troublesome to manage. The silos of responsibility between LANs and SANs cause untenable provisioning and configuration delays.
Ethernet speeds increased and protocols improved to prioritize storage data -- data center bridging (DCB), converged enhanced Ethernet -- and reduce packet loss during traffic congestion. These LAN improvements enable storage traffic and application traffic to share the same network, eliminating the cost and management duplication in siloed LAN/SAN environments.
FC over Ethernet technology sends storage data over Ethernet LANs operating at 10 Gbps or faster while interoperating with existing FC SAN deployments. SAN data can share the line with near-field communication, common Internet file system, Voice over IP and other traffic types.
Benefits include network infrastructure unification, management simplification and cost savings.
Key network features for FC over Ethernet
Storage activities produce significant network traffic -- especially if many servers share backbones to a SAN -- so review the LAN's overall architecture and bandwidth capabilities before adopting FCoE. At minimum, LAN backbones and high-bandwidth segments should be lossless 10 Gigabit Ethernet (GbE).
FC over Ethernet traffic crosses network switches as it moves between servers and storage arrays. Make sure these switches support DCB (options include Brocade's VDX 6730, Dell Networking MXL 10/40GbE blade switch, Cisco's Nexus 5020 Switch, HP's StorageWorks 2408 FCoE Converged Network Switch and others) and offer some 10 GbE ports.
Pay attention to the individual network adapters on each server. With ordinary Ethernet network interface cards (NICs), the software handles FC over Ethernet encapsulation, which degrades NIC performance under heavy traffic. Replace each server's NIC with a converged network adapter (CNA) that supports onboard FCoE encapsulation. A 10 GbE NIC provides more Ethernet bandwidth for busy servers and reduces the NIC port count. CNAs include QLogic's 8200 and 8300 series adapters, Cisco's UCS M71KR-Q, HP's CN1000Q Dual Port Converged Network Adapter, Dell's Brocade 1020 and others.
Legacy storage arrays might only support Fibre Channel interconnects. Some new storage arrays support native FC over Ethernet traffic, while others support Fibre Channel to the switch then FC over Ethernet from the switch to servers. FCoE storage arrays include NetApp's FAS3170, EMC's CX4-960 and VNX-F all-flash array, Dell's Compellent SC8000 and others.
Cabling is often overlooked, but the cables between switches, network adapters and storage subsystems are important. Ordinary Category 6 Ethernet cabling may be inadequate for high-performance FCoE traffic needs -- especially on network backbones. Organizations can shift to optical cables (such as SFP+) for long-distance connections or specialized twinaxial cables for shorter top-of-rack FCoE switch connections. Tweak the twinax cabling to suit the particular SAN, switch or NIC vendor.
Deploying FC over Ethernet
The path from Ethernet to FCoE doesn't require a full rip-and-replace. FC over Ethernet is usually adopted in slow, carefully planned phases over one or more technology refresh cycles, eschewing expensive or disruptive upgrades.
The move to FCoE usually starts with replacing server network adapters with CNAs. While swapping out the FC host bus adapters and Ethernet NICs with a single CNA card works, many IT organizations wait for a server refresh cycle and specify new servers with native CNAs instead.
Replace the Ethernet switches deployed to each server with dual DCB or FCoE-capable products. This lets legacy servers interface to the existing LAN and Fibre Channel director, while new FCoE servers and switches interface with the FC SANs.
In most data center networks, you can gradually replace the LAN switches. Once FCoE proves useful in the enterprise, replace the remaining Ethernet switches with DCB or FC over Ethernet switches. The storage arrays and other storage subsystems (such as FC tape libraries) connect to the new FC over Ethernet switches and eventually will make way for native FCoE storage systems.
A phased approach is perfect for lab or proof-of-principle testing, allowing IT professionals to gather valuable experience with FCoE setup, configuration, management and performance monitoring. A well-planned adoption scheme and detailed testing that highlights the value of FCoE technology to the enterprise mitigates resistance and eases the transition.
Will IT, business practices need to change?
From a technological perspective, FC over Ethernet involves skill sets from both sides, so IT professionals with backgrounds in Ethernet and FC setup and configuration don't require additional training.
However, FCoE alters the organization and delivery of IT services. For example, the storage group owns the SAN and storage arrays, while the networking group handles the LAN. These groups exist in silos that are difficult to overcome.
For FCoE to work long term, SAN and LAN teams need to merge into a single uniform networking group with shared policies and procedures. Considerable political or social resistance from staff members will potentially slow, or even prevent, FCoE adoption.
FC over Ethernet is non-routable, so it is restricted to a single data center, though this may change as the protocol evolves.
International standard T11 FC-BB-5 outlines the methods to encapsulate Fibre Channel frames into Ethernet packets.
FC over Ethernet cannot wholly replace FC until these issues are resolved.