Earlier this year, SearchDataCenter.com conducted a survey asking readers how they were coping with the economic downturn and one of the surprising findings was increased interest in data center automation -- tools to help IT shops automate menial processes.
More than 20% of survey respondents said IT automation tools are the spending priority in 2009 -- the second highest systems management priority, trailing only monitoring tools.
In the down economy, IT managers have been asked to meet increased demand with fewer resources (i.e., fewer staff) and that seems to push IT automation to the forefront.
Self-healing server monitoring systems
Kyle Rankin, senior systems architect at Quinstreet, is one IT pro who has integrated automation features into his systems management toolbox. "I've always been really big on automation. I hate doing manual, mundane tasks, so I always try to do the work up front to automate it ."
Rankin works in a data center with 2.5 full-time people supporting around 300 servers. He gets away with it by making every server as "vanilla as possible and automating the install process so it only takes us a minute or two of work per server to spin it up," he said. Rankin uses the Red Hat Kickstart feature to automate server installs.
Rankin also has some self-healing automation software installed to monitor the clock skew on his servers, using a homegrown script he wrote for the Nagios monitoring tool. If the clock gets too far out of skew, the monitoring program does a manual time sync before it alerts a system administrator about the problem. "Most of the time, I never end up having to get alerted because the monitoring script fixes it for me."
Rankin said this kind of automated, self-healing infrastructure is common in many organizations, but IT managers should be careful with it.
"I wouldn't trust anything too critical to self-healing systems because they can potentially be exploited to trigger on the wrong things," Rankin said. "I would have to really trust the vendor to implement something that would automatically fix IT problems. I probably wouldn't use it for anything critical -- only for trivial problems that you'd typically have a junior staff member or first-level support tech take care of."
Data center automation tools range from point products to big systems management suites from the Big Four -- Hewlett-Packard Co., IBM, CA and BMC Software Inc. Mississauga, Ontario-based Opalis Software Inc. is one of the best-known automation point product vendors, boasting customers such as Dow Chemical and Xerox. Opalis offers software to automate server provisioning, self-healing and data handling.
In 2007, HP bought the IT process automation vendor Opsware Inc. and has rolled out a suite of IT automation tools based on that acquisition. CA has a large portfolio of automation tools as well, ranging from job schedulers to automated load balancing and provisioning. According to CA, over the past six months sales of workload automation tools have been impressive, a trend that flies in the face of the current economy. And in June 2009, the company acquired IT process automation vendor Cassatt.
Business climate drives IT to automate processes
One of the factors driving interest in data center automation is the overall business climate. Mergers, acquisitions and layoffs are all factors that may prompt an organization to consider automating IT processes.
"One company will acquire another company, pick up its assets and lay off a good chunk of people," said Andi Mann, an analyst at Enterprise Management Associates (EMA). "With companies merging, there is a lot of activity around integrating access to customer records, financial data. A lot of that is done manually. But some companies are doing Web-based portals into back-end databases, extracting records, transferring into the other systems, loading across enterprise data environments."
The slumping economy has affected IT automation decisions in unexpected ways as well. Across the U.S., new car sales are down in the dumps. But used-car sales are up, and used-car research sites have experienced massive IT growth. These companies have turned to data center automation tools to deal with spikes in IT resource demand.
According to Robert Stinnett, a senior analyst at Carfax Inc., more used cars are on the market than ever before, and his organization has more data on them. This creates an avalanche of IT demand.
Automating data entry
"When we started, we only had title records," recalled Stinnett. "Over the years, we've gotten into maintenance and accident records. More and more sources of information are coming online. Even mom-and-pop repair shops are providing us with more records. We're integrating with partners like Autotrader and Cars.com. The amount of records our partners are sending is phenomenal."
In the past, people sent Carfax tapes, CDs, floppy disks with car history information, and uploading that data was a manual process. Today that process is handled by a Web service, API call. But whether it's Autotrader sending Carfax 1.5 million records to add to its database or Joe Consumer calling up a single report, Carfax needs to process these requests.
Carfax uses BMC's Control-M File Watcher and the Advanced File Transfer module to use FTP to move files both in and out of its databases. The tool recognizes the data request and uploads it, without a human involved.
"We're not getting rid of people; we're giving them more exciting jobs to do than load the databases by hand."
One of these more exciting jobs at Carfax is converting boxes to VMware virtual servers and developing a private cloud for its server infrastructure.
Carfax's private cloud is built on Control-M's node group functionality. Carfax's applications don't rely on single machines; they are built on nodes consisting of CPU, memory and disk resources that can be used by whatever batch job needs resources at the time.
"If a server dies, it's no big deal," Stinnett said. "Just like a light bulb, we simply screw in another one." This strategy also reduces the need for manpower. "When something goes wrong overnight, who cares? Shift the load over to something else and we'll deal with the failure sometime tomorrow."
Automating server deployment
Santa Monica, Calif.-based Edmunds Inc. runs four auto consumer research websites, including Edmunds.com, which has experienced the spike in used-car research. The sites serve 15 million visitors each month. The company has roughly 650 servers (primarily Dell boxes running Red Hat Linux) in production at a large data center in Los Angeles, according to Kenny Sexton, the IT operations lead at Edmunds.com.
Edmunds' IT department often has to react to news in the auto industry, rolling out infrastructure to launch new site functions and information quickly. For example, the recent Cash-For-Clunkers legislation prompted a new set of tools and site features on the Edmunds.com site.
These new applications are often developed in-house. Supporting new functionality requires the operations team to deploy new servers quickly. Sexton's team uses BMC's BladeLogic to roll out "gold standard" server images, so new servers are nearly ready for use when the provisioning process is completed.
Edmunds' development team has self-service provisioning for its sandbox. "Our internal customers can go in and get a mirror of a production Web server: a gold-template virtual machine with bits and pieces of the site that they can develop against," Sexton said. "Our goal is to have it be completely hands-off."
Bill Bradford, a senior systems administrator for an energy services firm in Houston, Texas, has also automated server deployment. "We're doing more automated installs and using scripts for post-installation setup, to cut down on system build time and, of course, get more systems built in less time with the same amount of staff."
The limitations of data center automation
While automation tools for tasks such as job scheduling have been around for years and companies increasingly use automation tools to streamline software installation, significant barriers to adoption persist.
The automated migration of virtual machines is one area that still needs work, according to Mann. "People talk about VMware's VMotion like workloads are just flying around the enterprise, but these changes need to go through a change management process," EMA's Mann said.
The other area that has a long way to go but shows a lot of promise is automated remediation. "A lot of tools will tell you what's going wrong, but in an ideal world, these tools would fix the problem once they discovered the problem," Mann said.
Enterprise Management Associates recently conducted a survey, asking IT managers if they trusted data center automation tools to take actions on problems independently. The survey also asked IT managers to compare tools they have today with the capabilities they would like to see in the future (see Figure 1).
Indemnification, support woes plague open source systems management
IT shops pick point management tools to cut staff, speed installs