Many data centers strive to be fully "lights out," but is that such a wise idea? Last week, that was the question raised by Mike Hagan, the senior vice president of operations at Lee Technologies, at an AFCOM
A lights-out data center means getting as many people out of the data center as possible to reduce staffing costs and eliminate the possibility of human error. But according to Hagan, at least on the facility and infrastructure side, it can be dangerous to pull out the troops.
Indeed, said Hagan, there are risks. "You have to get into the question of evaluation," he said. "It's not what taking the cost of the individual out would be. It's the cost of what not having him there is."
Hagan does have a pawn in the game; one of the major services provided by Lee Technologies is data center staffing. Still, Hagan's session –"Lights-Out Data Center: Don't Believe It" – was packed, and during the Q&A session, some members in the audience voiced concerns similar to Hagan's. (Of course, some of those naysayers may be data center facility staffers who fear being replaced by automation software.)
Hagan's colleague Don Denning, a facility operations manager for Lee Technologies, said that sometimes there is no replacement for a warm body.
Case in point, at one brand-new site that Denning helps manage, the building management system indicated that an air handler had shut off, so data center staff went to look at it and found that the air handler ran fine but the building management system was wrong. Had no one been on-site to identify the problem, the company might have overcompensated by running other air handlers harder or starting up a backup air handler, thereby wasting energy and taxing power capacity, he said.
Yahoo: Among the lights-out skeptics
Even among mammoth data center operators, there is some debate about lights-out data centers. While huge companies like Google have moved toward a lights-out data center as much as possible, Yahoo has taken a different course.
"We are believers that there is a significant risk reduction by having people in the data center," said Kevin Timmons, the company's vice president of operations. While Yahoo tries to minimize the number of people on the data center floor among server and IT equipment racks, "we've seen the benefits of having people in the data center 24 hours a day," he said.
Nevertheless, the imperative to reduce staff and go lights-out remains, and data center managers should prepare themselves for the eventuality. To that end, Lee Technologies' Hagan recommended cross-training IT and facility personnel. If an edict comes down to cut staff, a data center won't lose as much institutional knowledge as it would otherwise.
"If you have eight facility guys, and 12 IT guys, you may want to cross-train in case you have to eventually go down to 15 total," he said.
Hagan also recommended standardizing what a lights-out data center means throughout an organization. For businesses with multiple data centers, lights-out may mean something different in each location, and getting the different facilities to work together as single operations team can be a challenge. It can become especially problematic if staff from one facility is consolidated or moved to a different site.
"The biggest stumbling block we see is inconsistency in procedure," Hagan said.