This content is part of the Essential Guide: Emerging data center workloads drive new infrastructure demands

Build a big data and internet of things data center roadmap

Many data centers make the mistake of diving into new technologies, such as big data and internet of things, too quickly. Pacing yourself and assessing requirements will pay off.

From a data center perspective, big data and internet of things projects almost always stress the network and storage infrastructure. Planners need to carefully assess infrastructure requirements before their organization embarks on these sorts of large-scale, data-intensive projects.

Traditional business intelligence projects are founded in needs and understandings that differ from those of big data projects. A typical business intelligence effort starts with a clear idea of what questions must be answered; what data is available or must be collected to answer those questions; what results must be reported; and who within the organization needs those results. This type of project has been at the foundation of enterprise IT for decades.

Big data and internet of things (IoT) initiatives have a different focus. They ask: What are the right questions; what are the problems to address to better serve customers; and what products must be made available to retain current customers while enticing new customers to purchase products and services from the company?

This typically means that IoT and big data projects require different expertise, different levels of experience and different tools. As a result, such projects can be more difficult for an IT team to execute.

Start slowly with big data and internet of things

When powerful new technologies or new approaches to IT gain a certain momentum, there can be a rush to adopt them -- sometimes with little understanding of what is required to have a successful first experience. IoT and big data clearly fall into this category.

Resist the pull toward a ready-shoot-aim approach. This is especially true of an IoT project.

This thinking can drive organizations to invest heavily in an approach that yields much disappointment and little useful data. Failure could come from selecting inappropriate tools, incorrectly configuring systems that support those tools, a lack of necessary expertise or aligning with the wrong partners. Once burned, many decision makers blame the approach or technology.

There's certainly been no shortage of buzz about the potential of big data. And reports have banged the drum for IoT, too, pointing out how it will connect everything from our mobile phones to our automobiles to our household appliances. Suppliers of hardware, software and professional services have jumped in, each wanting to get their unfair share of the potential revenue that these technological approaches will produce.

Nearly all of the suppliers of systems, storage, networking, operating systems, data management tools and development tools have come forward with big data product and service sets. These same vendors are starting to offer ways to converse with and gather data from intelligent devices.

Integrating big data and internet of things

Before leaping into IoT and big data projects, it is wise for an organization's leaders to stop and assess what the business really needs. Assess the IT team's capabilities and expertise. Be realistic about where things could go wrong and what could be gained.

Organizations typically design big data projects to determine which questions to ask, rather than to address specific, previously known requirements. This means decision makers and developers must first want to determine the right questions to ask based upon operational, machine and other types of data already being collected; it's possible no one has taken the time to analyze that data. An IoT project could become a source of data that would feed into a big data undertaking.

Both IoT and big data typically rely on NoSQL databases that, in turn, rely on clusters of systems executing data management software, extensive use of network capacity and either shared memory or sophisticated data caching technology to accelerate the use of available storage media. An IoT project is likely to greatly affect data center networks and storage as well.

Most organizations have a wealth of raw data about their own operation, including information automatically collected by operating systems, database management products, application frameworks, applications and even by point-of-sale or point-of-service devices. Organizations could use the data to gain a clearer, holistic awareness of the strengths and weaknesses of procedures, products and training. Adding IoT to the mix offers the potential for a company to extend its understanding of customers.

Analyzing this huge and growing pile of data could, and often does, give enterprises clues to better grasp customers' needs. The business may also learn that it hasn't been collecting the right information to ferret out the questions that will address its own unique set of issues.

Resist the pull toward a ready-shoot-aim approach. This is especially true of an IoT project. Seldom has an organization had such a powerful opportunity to put off, irritate or offend individual clients.

An IT team must develop a clear understanding of its intentions, the tools it will use and the suppliers who will be part of that effort. Only then can a team attempt to capture and tame the big data beast or put IoT to effective use.

This requires an organization to properly configure and provision its infrastructure, a process that involves deploying the necessary amount of processing power, memory, storage and network capacity, as well as the proper software for development, ongoing operations, monitoring, management and security.

Each of these elements must be selected and provisioned with care. This process, however, isn't necessarily a case of more being better.

With IoT or other customer-facing projects, it would be wise to consider how customers will react to being online with a business all of the time. Performance, privacy and functional capabilities are all critically important.

Development tools for big data and internet of things

Each big data approach has its own set of development and deployment tools. The same can be said of IoT platforms. To be most effective, a company's development staff must understand these tools, know how to use them and understand how to build an optimal system.

It is likely that the people working on the big data project will use different tools than an IoT development team. The two teams, however, must communicate with one another. The IoT team will need to collect the appropriate data to support the big data initiative. A business that is new to these types of technologies would be wise to start with small projects and build up to larger ones as the staff develops experience and expertise.

Organizations must treat big data projects like the business assets they are. This means operational oversight by the IT administrative staff. It would be best to select monitoring and management tools that fit with the enterprise management framework and will provide data that is understood and useful.

An IoT project, because it directly faces customers, requires lightweight, responsive monitoring and management. If the tools are too heavy, customers are going to complain about your company consuming too much of their expensive data plans. Finding the proper balance of information gathered, features offered, overall performance and the volume of data sent back and forth can be tricky.

Many organizations find real promise in big data. IoT best practices are still emerging, so standards are not broadly available. In both cases, however, properly selected and configured components combined with technical expertise are key elements of a successful project. The proper configuration choices are driven by which systems are selected, supported OSes and the system, network and storage configuration deployed.

Often the most important factor, however, is embarking on the project with the proper mind-set. In the case of big data, the goal should be learning the right questions to ask rather than treating the project as just another business intelligence initiative. In the case of IoT, the project must be able to provide useful services to customers in exchange for permission to gather data to feed big data-based sales, support and business intelligence systems.

Next Steps

Big data belongs in the data center -- for now

Prep your infrastructure for big data

IoT cloud service battle heats up

Dig Deeper on Emerging IT workload types