The largest data center operators find better performance and economy in custom servers. More enterprises are taking note of the benefits, for their outsourced workloads and in the trickle-down effect on server options.
Server vendors offer various system options -- CPUs, memory, disk and I/O -- within the context of the vendor's established motherboards and form factors. To get beyond these simple options, very large data centers customize the fundamental design
Custom servers only make sense for hyperscale data centers that benefit from incremental system improvements and can justify the economics. The typical enterprise data center hosting various applications will find complete server customization both financially and technologically unfeasible. However, with more outsourcing providers and large Web companies hosting more workloads in hyperscale data centers, the custom server niche is growing. This should generate more server options for enterprise data centers and help to spawn more specialized server designs including ARM-, x86- and even OpenPower-based system designs in the future.
Levels of custom servers
The actual level of customization can vary dramatically depending on the data center and its server vendor.
On the simplest end of the spectrum, a vendor might customize a server by installing a disk drive, PCI Express expansion card or other device. For example, a company might require stock servers with a specialized high-performance GPU card with oversized power supply to support the GPU.
On the other end of the spectrum, the data center's business provides a complete server design: chassis mechanical design, motherboard design, firmware code and other electronic or mechanical elements. The server vendor manufactures the design for sole use by that customer.
When every clock tick or packet translates to revenue, custom servers offer system performance advantages. When a data center operation handles 1 trillion requests each month, improving system performance by just a few percentage points can make a huge difference in computing capabilities. Customized servers typically form a homogeneous system environment, delivering far more detailed management insight and control over a huge number of servers.
Hyperscale data centers also benefit from controlling the server form factors. Physically smaller servers increase system densities. Custom server designs can incorporate rail-less or quick disconnect frames. For power efficiency and power resiliency, custom server designs can employ a quick disconnect DC rail system instead of individual system power supplies.
Who takes advantage of custom servers?
Full custom server design is not undertaken lightly. It requires a core of expert systems designers and technicians with a detailed understanding of the enterprise's technical computing requirements and how those computing requirements translate to server hardware and software. For example, if the business's computing requirements emphasize storage bandwidth, designers might incorporate high-performance storage elements such as SSD drives or PCI Express I/O accelerator components.
CloudFlare custom developed the G4 server with Intel Xeon 2630L processors, Intel SSD drives and upgraded 10 Gbps Ethernet network cards, along with enhancements to system cache. Taiwanese original design manufacturer Quanta is building the servers for CloudFlare. (Previous G3 servers were built by original equipment manufacturer Hewlett-Packard.)
Another company is keeping its custom server builds in-house. Web hosting company OVH builds its "2013 Reloaded" models in an on-site facility in Beauharnois, Quebec. The server combines high storage bandwidth and RAID support with several cache layers that accelerate page serving from memory rather than from disk access. The in-house turnaround time is so fast that new servers can be built in 15 minutes and in the rack in under an hour.
What goes into creating these fleets?
Even large enterprises rarely maintain the expert staff necessary to design, build and provision full custom servers, troubleshoot them and monitor performance to optimize the design. For customized server production to be economical, batches must be built by the thousands. New designs must roll into manufacturing roughly every three years. If your server demand isn't on the same scale as Facebook, Google, Rackspace or other top-level data center operators, custom servers won't be a viable option.
If you're operating at this scale, your server lifecycle is probably shorter than most enterprises because of the efficiency and performance pay-offs from new technology. But a new server design takes time with comprehensive benchmark testing. While server testing and validation is to fact-check vendor promises in most large data centers, custom server users take validation to a whole new engineering level. Benchmarking focuses on getting the most efficiency from the data center because the business is the data center.
CloudFlare's G4 server undergoes continuous optimization and tuning. The business constantly tests and validates new components, such as newer, faster SSDs, on refresh cycles as short as one year.
The two traditional server manufacturing options are OEMs and ODMs. Most major server vendors are OEM-capable, delivering modified or specialized systems based on existing products. While a high-performance 10 Gbps network card might not be in the list of regular options available for a particular system, the right customer could convince them to make it available. However, a unique design, such as the Open Compute Project's oddly shaped rack chassis, better suits ODMs, which source and assemble customized mechanical and electronic components.
This was first published in December 2013