Server benchmark testing succeeds or fails on the IT staff's approach to the process.
IT professionals have always relied on benchmarks to gauge performance in the data center. But server benchmarks are only as accurate as the people who use them and the measurement circumstances. Too often, the wrong benchmarks are deployed, they don't reflect real operating conditions, or they are deployed carelessly or for the wrong reasons. Follow these five key steps when
1. Lay out your server benchmark goals
Though you can evaluate diverse operating parameters, this kind of comprehensive benchmark process is time-consuming for an IT staff already burdened with heavy workloads and tight budgets.
Administrators must first recognize the problem they want to solve. It may be as simple as verifying a vendor's performance claims before purchasing a next-generation server, looking for underutilized system resources to consolidate, or even grappling with a suspected performance problem. With a clear goal in mind, you'll know what parameters to measure -- the CPU utilization may be interesting, but offers little value to resolve an issue with network I/O, for example.
2. Identify the right benchmarking tool
There are many server benchmark tools on the market, and each typically does a very specific job. Once you determine the parameters to benchmark, focus on selecting a tool that will best measure what you need to know.
The Standard Performance Evaluation Corporation (SPEC) produces a range of benchmarks routinely used in IT environments. For example, SPEC CPU2006 measures the performance of compute-intensive workloads, while SPECjbb2013 benchmarks Java application performance. Use the latest version of the appropriate benchmark to ensure current hardware is identified and measured accurately.
As a rule, avoid vendor-provided server benchmarking tools due to vendor bias; you can hardly be surprised if a benchmark tool from "vendor X" reports their server is better.
3. Perform server benchmark testing under real load conditions
Benchmarks only provide accurate information when the server is running under actual load conditions. It isn't always possible to deploy a benchmark in a working production server. Instead, benchmarks are often run in a test-and-development or other simulated environment that may not properly mimic CPU, workload, storage and network traffic patterns.
A classic case is benchmarking a CPU that isn't running any workloads other than the OS and benchmark. Of course it will have plenty of spare capacity then. Many organizations with formal server benchmarking processes in place use a test setup that includes transaction scripting or load generators to simulate real working conditions. Performance can be affected by factors like load balancing and even the placement of data on disks, so IT professionals must consider every element of the server's deployment when acquiring benchmark data.
4. Test under varied time and usage conditions
Most data center servers don't maintain a constant load. A Web server workload may see more network and disk access usage during business hours, while a payroll application may only be used for a few days each month. This means a single point-in-time snapshot of the server's performance is not an accurate picture. Run the benchmark continuously or run several tests as workload conditions change. This helps identify conditions that cause unexpectedly poor performance, as well as opportunities for improvement via upgrades or workload rebalancing.
5. Document test conditions and be consistent
Lack of documentation is a major oversight in server benchmark testing -- especially not recording test setups and load conditions. Without established parameters test-to-test, you can have unexpected differences between results when testing different systems or when testing the same system multiple times. By documenting the test setup, scripts, load generator data sets and other factors, administrators can achieve more repeatable, objective results. Reviewing the documentation can also help validate benchmark results.
Server benchmarks allow data center administrators to measure key attributes of system performance, make informed evaluations of performance over time, and compare different servers. Most benchmark tools are quickly and easily deployed, but meaningful results will depend on a goal or goals, selecting the right tools, testing under real load conditions (or as close as possible) and over a period of time, and maintaining copious documentation for future reference and validation.
This was first published in July 2013