Who controls big data implementation?

Technical challenges aside, perhaps the most difficult thing to navigate in a big data implementation is politics between data scientists, lines of business and central IT.

IT teams are very wary of investing in infrastructure for big data initiatives, said Andrew Warfield, CTO at Coho Data, a provider of scale-out storage infrastructure. Pitching them dedicated resources can be difficult.

"IT tends to have an enormous amount of suspicion about whether this [big data] stuff is going to be successful," Warfield said. As a result, individual lines of business often circumvent IT altogether, and buy, build and manage any big data infrastructure themselves.

Any department that opts to go-it-alone may get what's coming to them, but savvy IT professionals know that no good can come of this.

"I've talked to really terrified IT groups seeing 10- to 12-node clusters out there that are totally out of their control," Warfield said. "Without a strong systems administration competency, organizations just dipping their toes in the big data waters have a lot of trouble just even getting these tools up and running -- getting the data in, launching workflows, backing it up," to say nothing of the operational and financial benefits of an IT-centric approach.

Among savvy businesses, the goal should be compromise, said a VP for a telecommunications provider undertaking a big data initiative.

"Our main goal was to have an infrastructure that allows us to have some control over it, rather than having to go back to IT every time we need to make a change," the VP said.

"IT sets up the hardware, and then my team had administrative control," they added. That way, "we can control our assets on a virtual level, and let IT focus on the infrastructure."

Alex Barrett is editor and chief of Modern Infrastructure. Contact her at [email protected].