This article is part of an Essential Guide, our editor-selected collection of our best articles, videos and other content on this topic. Explore more in this guide:
4. - Big data, data center applications: Read more in this section
- Go commercial for big data analytics software
- Big data projects require big changes in data center hardware and software
- Legacy applications in the cloud: What's ahead?
Explore other sections in this guide:
Big data analytics was once an area of technology reserved for scientists, but corporations now need the technology to analyze huge volumes of data for a host of business reasons.
Some 42.6% of respondents to TechTarget's 2013 IT Priorities Survey said they plan to either continue or expand their big data management and analytics programs this year or initiate one by year's end. Another 19.4% said they plan to conduct an evaluation for a program to address big data needs. Only 25% said they have no plans to invest in big data analytics this year.
One company looking to capitalize on that demand has rolled out an updated version of its big data analytics software for collecting and analyzing big data that had largely been aimed at the intelligence community.
Chiliad Inc.'s commercial offering, called Discovery/Alert 7.0, is designed to cut the time and expense of consolidating big data by residing in close proximity to major data sources. Those data sources can also be connected as a virtual consolidated data center that can be more easily searched.
Chiliad said the advantage of its product is its search and analytical capabilities, which gather not just individual pieces of data, but through its "iterative discovery" capabilities, bring to the surface connections in that data that illuminate bigger picture trends.
"Many researchers need to quickly get to the gist of all the content they collect so they can figure out what really matters," said Ken Rosen, a Chiliad spokesperson. "Once they do that, they can refine their hunches and gain an understanding of an entire area, not just individual facts here and there."
This capability has proved useful to a number of federal agencies in law enforcement and intelligence that use data warehousing applications to not just vet individuals to determine if they are dangerous, but to dig deeper to see if they are part of larger conspiracies.
"With this system you don't just get a list of documents returned to you, but also all the concepts that are in those documents," Rosen said. "It's like having an assistant gather all the documents you need to search, then reading them to you, pointing out all of the most relevant passages, even passages that were not part of your original search."
Company officials believe the health care and medical industries would also be a good fit for the technology. While some observers agree the technology makes sense for those two markets, they say competing in the private sector will be a challenge for Chiliad.
One analyst believes that by eliminating the costs associated with consolidating data, the technology could be enough to at least win over the wallets, if not the hearts, of some commercial IT shops.
"Avoiding data consolidation is not just about saving costs, but about gaining better insight," said Benjamin Woo, managing director with New York-based Neuralytix Inc., a data management consulting agency. "Solutions like this can enable customers to discover the context inherent in their data by uncovering relevant concepts that may not have been implicit in a user's original query."
How Discovery/Alert works
Discovery/Alert 7.0, which can run on high-end servers down to pizza boxes, sends a query over a "thin Internet connection," waking up a single instance node that then indexes all local data regardless of its format. That node in turn communicates with other nodes in the network to search and collect the requested information, Rosen said.
"You don't have to move data anywhere or send it to the cloud. It simply wakes up and begins performing like a virtual data center. So right away, that first step of having to consolidate all your data as part of this whole process is already done," Rosen said.
Another advantage of being able to install the product close to data sources is it can better protect private or sensitive data because it eliminates the risks inherent with copying and moving data to other locations, company officials noted.
"We think being able to operate on things like Android devices is important from a value proposition standpoint because we can show that big data solutions don't necessarily require fat clients," Rosen said.
Available now, Version 7.0 pricing is offered on a tiered basis through subscriptions. For instance, Tier 1 covers two data centers and up to 10 TB of data for $60,000 per year, with Tier 2 covering five data centers and up to 50 TB of data for $200,000 a year.