But while Danley's group has had to troubleshoot problems within this distributed infrastructure, he never considered implementing an enterprise systems management tool along the lines of Tivoli . "Within the walls of a large IT world, there are dozens and dozens of management tools," Danley said. Going for another enterprise management system to fulfill his needs wouldn't have been practical or timely.
What Danley wanted was a tool to provide him with a view of all the various IT assets that run the relevant e-business components. So Danley turned to Splunk Inc., a provider of an eponymous open source tool that indexes IT-generated data such as logs, configurations, alerts, scripts and performance metrics such as power consumption.
Needless to say, whenever problems occurred, tracking down the root cause was a challenging exercise, as was determining ownership once the problem was detected. Resolving a help desk ticket regarding an integration problem might require a systems administrator to manually grep -- that is, write a command to search for a particular character string -- 75 files or more.
Here's how the process typically worked. Let's say a user has difficulty reconciling purchase orders received with those processed and opens a help desk ticket. The systems administrator then begins to gather logs and records from the various integration components, then searches them for any references to the purchase order number among the numerous files and systems involved in the transaction. Ultimately, resolving the problem can take two hours or more and involve multiple systems administrators and departments.
But with Splunk in place, a systems administrator simply plugs in the purchase order number in question into Web-based Splunk interface, which runs a canned search script to see where among servers the transaction got stuck; the problem is resolved immediately.
The index-and-search approach to IT data
Splunk co-founder and CEO Michael Baum likens his company's product to Google. Rather than search and index the Web, however, Splunk searches and indexes the data generated by servers, applications and network devices. As IT environments have become increasingly complex, Baum posits that the ability to search for relevant data is more useful than poring through the reams of reports IT administrators can get out of traditional systems management tools.
David Williams, the vice president of research at Gartner Inc. says that Splunk fills a gap not found in proprietary log management tools. "There are plenty of log file management systems out there, but most of them focus on logs from a security and compliance perspective," he said. They search log files looking for discrepancies that could suggest a security breach or compliance problem. "Splunk is really designed for IT operations in that it collects huge amounts of miscellaneous log file data from disparate sources and makes sense of it."
ITIL and free downloads seal the deal
As an IT Infrastructure Library (ITIL) shop, Motorola's help desk group's functions are structured based on ITIL precepts including incident management, problem management, availability management and service-level management. Danley was initially drawn to Splunk in January 2007 as a way to tackle those processes.
Splunk is free to download, which didn't hurt either. "The thing Splunk had going for it was a free evaluation. I got a limited usage with the download, and I could immediately evaluate the tool's potential," Danley said.
Once Danley had a look at the tool, he opted to buy it. For the initial setup, a Splunk representative worked on-site with Danley's team to index the log file information thrughout the relevant architecture. "It was real easy to config," Danley said.
Streamlining problem management
Currently Danley uses Splunk for the EAI infrastructure components found in 12 servers -- and collects 6,000 data points or sources. (A separate IT security group within Motorola uses Splunk proactively monitor the network by indexing data from network devices, firewalls and routers.) Additional component layers Danley hopes to load into Splunk include B2B, EDI and LDAP among others.
"I get my own little picture of all my IT data," he said.
Let us know what you think about the story; email: Megan Santosus, Features Writer .