Despite challenges that can deter successful deployment, big data analytics remains a goal for enterprise IT.
As many as 73% of businesses have or plan to invest in big data within the next two years, according to " Big Data Investment Grows but Deployments Remain Scarce in 2014," a big data survey by Gartner Inc., a research firm in Stamford, Conn. The study of 302 Gartner Research Circle members explored global organizations' big data technology investment plans, stages of big data adoption, business problems solved, data, technology and challenges.
Up 9% from last year, the number of businesses investing in big data continues to rise. Companies don't want to be left behind and miss opportunities their competitors are taking advantage of, so they invest in technologies to support big data, said Nick Heudecker, research director at Gartner.
Only 13% of respondents reported their IT organizations put big data projects into production this year, but that's 5% higher than last year.
Organizations without previous interest in big data are not as skeptical, as only 24% of those polled voted against the use of big data technologies in their business -- down from 31% last year.
While investment in big data increased, real infrastructure deployment did not. In most businesses, unwillingness to try a strategic big data deployment and experimentation held back use.
"Businesses have trouble understanding the value proposition," Heudecker said. "They ask, 'How do I take insights from this data and move them into production?'"
Not all companies fail to understand the potential value from big data.
Magpul Industries Corp., a firearms accessories manufacturer in Erie, Colo. did not need IT or a big data deployment until about one year ago.
Now that the company is growing into an enterprise, it will implement big data technologies as soon as possible to aggregate Magpul's data sets, said Chris Steffen, director of IT, yielding business insights
Due to the broad definition of big data, Gartner's survey focused on ways companies use big data to improve customer experience -- mostly in healthcare, insurance, banking, retail, communications and media -- and process efficiency.
Steffen was involved in the big data implementation at Kroll Factual Data, a credit reporting company based in Colorado. As big data technologies evolved, more were implemented in its facility.
"Ten years ago, it was so CPU-intensive," Steffen said. "[Big data technology] took data sets and aggregated them into something readable and understandable." Now, those data sets can be broken down even further.
Steffen is considering different big data tools, like Microsoft Dynamics ERP and SMB cloud tools, and using his past experiences to decide what's best for Magpul to move forward with data analysis as quickly as possible.
Invest in key areas
Big data analytics allows organizations to address a wide range of business decisions and follow patterns and trends.
Organizations are progressively more willing to explore big data opportunities. Previously, they were unwilling to make business-driven decisions, instead of IT-focused decisions, Heudecker said.
"It's the newness of the technology and the skills required for tactical investment" that prevents big data adoption, he said.
A lot of the technologies aren't fully optimized, Heudecker said. Many businesses are experimenting and deploying big data in the cloud.
"The fundamental change in investment models is another challenge," he said. "If you require local storage to a large extent, then the cloud is attractive."
A 7% increase in big data projects illustrates organizations' progressive understanding of its value.
"We're starting to get insight into [real-time business intelligence] and pulling data across the system," Steffen said.
Before the inauguration of its IT program, Magpul had no idea of the true ramifications of a big data infrastructure, like aggregating data and breaking it down into categories, he said. For example, companies might collect a mortgage loan applicant's education, criminal and credit history, along with various other data points, but not all of this information is relevant to whether the applicant is a good candidate.
Gaining an understanding about what data to collect and process is just one part of solving investment, initiation and deployment concerns. For organizations without plans to deploy big data analytics, some challenges that hinder adoption include uncertainty about how it would best deliver value, how to outline a proper approach, and leadership difficulties, the survey showed.
When planning to invest in big data technologies, organizations must define the value, acquire new skills and abilities, define a strategy, find funding, and predict infrastructure concerns. For those that are beyond the planning stage, understanding risk and governance issues, data integration and infrastructure become prominent.
"Technology is changing so rapidly, you never know what your [true needs are]," Steffen said. "Structure your data [so you can] structure it another way later, when your needs are different."
Before implementing big data on a large scale, roll out the technology in tiny pieces, he said. Figure out what is important, and what isn't.
Volume vs. variety
Increasing storage and compute capacity allows an organization to deal with the mass amount of data; however, the variety of data sources poses more of a challenge. Most organizations analyze data from traditional sources. Input from multiple sources means acquiring various tools and skills to use the data, not just store it. Integrating social media data sources with other types can deteriorate profile and interaction analysis.
When asked which data sources they expected to add in the future, 30% to 40% of respondents said every data source. Either organizations lack a plan for the future, are afraid of missing out on opportunities, or are optimistic they can leverage challenging data sources easily -- because they can analyze big data on transactions and log data.