Modern Infrastructure

Container trends fuel storage needs


Falling back on bad habits in a DevOps organization

In a DevOps organization, can you still set boundaries? How do you pick the right tools? Learn from these experiences before you make the switch.

There's a lot to love about DevOps -- in the abstract. In real-life implementations, a DevOps organization is subject to the same turf wars and poorly suited tools as traditional IT. It also opens IT up to a few new challenges, including isolation from peers and unstructured adoption paths.

"As [teams are] trying new tools and flows, nothing is perfect from the get-go," said Vijaya Kokkili, director of quality at CommerceHub, a support provider for e-commerce retailors based in Albany, N.Y. "We have stumbled on several issues, and still some we don't have answers for."

In the two years since her company adopted DevOps, Kokkili saw both the challenges of the transition and of the new reality.

"We found that we were trying to put so many standards in place that it makes it hard to get things done," she said. "It's not about this tool or that tool making it work," the teams realized, but about how you communicate and collaborate and refine how you're working.

A new kind of lonely

It's easy to fall back on bad habits in a DevOps organization, with a new twist.

Like most IT organizations, CommerceHub's functional teams of development, database, quality assurance, operations and other areas were physically segregated around the office. When they adopted DevOps, they created cross-functional teams with members from each group, while also retaining managers that oversaw functional areas. The cross-functional teams, however, remained silo'd on the inside. By itself, putting a few developers in with the IT operations tech, the database administrator, QA tester and project manager doesn't break down any silos.

"The QA engineer could be left out by developers -- they don't understand each other [or] speak the same language," Kokkili said, and there's no other QA person there. Training eventually alleviated this frustration, teaching each member how to share in the others' jobs.

Once communication barriers fall, newly collaborative IT shops must reinforce the aim of all these projects: a customer-focused culture.

Ohio State University's IT team implements ITIL with some lean and Project Management Professional concepts. DevOps is next. If ITIL is a framework of best practices to manage IT changes, "DevOps gets at changing the culture," said Bob Gribben, director of service operations there.

Meanwhile, a DevOps organization still has departmental goals. IT operations has uptime and utilization goals, for example; these are different than the product owner's goals for cross-functional DevOps teams. A customer- and product-centric culture doesn't mean abandoning IT metrics.

We share everything -- but that's mine

If DevOps is about shared responsibility for an application, can you still have boundaries? For instance, if everyone on a cross-functional team shares in incident response, how can you restrict the code developer from accessing production to address a problem? Should you even try?

Complicating matters is the fact that DevOps doesn't look the same from one IT organization to another. In one DevOps organization, the IT operations team retains 100% control over the production environment. In another, developers are responsible for infrastructure as code and ensuring stability in every release.

Preserving the security of the network and the company's information security represent barriers to agile, flexible development. As such, security, data and testing are often neglected pieces of a DevOps culture.

Security, role-based access control and encryption are vital components of DevOps application lifecycles, said Michael Grant, DevOps practice management lead at technology consulting firm Column Technologies Inc. Having tools in place that provide good visibility into spurious traffic make it easier for security teams to agree to open the ports or grant an application the access that DevOps environments require. Infrastructure monitoring tools that provide visibility into the network and applications' calls can also increase security, whether in a data center or on multi-tenant cloud resources.

Data is the underpinning of every enterprise app, and therefore, it should be a major component of DevOps processes. But database administrators don't have a natural place in the DevOps continuous feedback loop. Data as a service is the next step for DevOps, said Grant. Secure data as a service, he said, enables more accurate QA load testing prior to releasing code to production -- without overrunning the database team's infrastructure.

Test has an even harder sell to DevOps adopters, who don't always see the value of quality assurance and testing expertise.

"We had a product team with no QA person -- the assumption there is that dev can do test," said Kokkili. "But without the expertise, test always ended up with technical debt that caused issues in production."

In contrast, placing test and quality assurance engineers in the DevOps team can prevent the outdated mindset of future fixes and one-off workarounds for code problems in production that DevOps set out to fix in the first place.

When all you have is a hammer

Tools are a critical part of any DevOps environment, but for such an important issue, tool choice is a sticking point for DevOps shops.

"One team uses SalesForce to track issues, another uses Trello and another uses Jira -- but it's all for the same application," said Kokkili. "Higher-level visibility is difficult."

Developers are accustomed to grabbing tools that work -- whether that means it fits the company's compliance stature, reporting requirements or ops visibility needs or not, noted Chris Riley, a DevOps analyst at Fixate IO, in a podcast with SearchDataCenter. The goal for a DevOps organization is to standardize without blocking creativity for team members, and it is still a struggle to find common tools across functional positions.

Enterprise IT shops adopting DevOps tools often weigh open source versus supported enterprise suites, and usually come down on the side of enterprise suites, said one IT consultant who focuses on DevOps. That's because enterprises expect support, stability and usability features like GUIs that many open source tools don't have. Meanwhile, traditional infrastructure management and monitoring tool vendors such as CA and BMC are developing DevOps products, but with mixed results. A compromise may be tools such as Puppet that follow the Linux open source model and draw from the upstream community for a controlled, supported product.

Are we there yet?

Success requires continual improvement and refinement, rather than an initial push for certain changes without an overarching plan. DevOps, like ITIL before it, promotes this feedback loop, but too often, IT shops expect to arrive at a final destination, said Doug Tedder, principal consultant at Tedder Consulting.

We saw the same pattern in ITIL IT service management (ITSM) implementations. Folks bring in incident management and a rudimentary change management process, but don't understand how processes interact or their goal, said Tedder. "Some folks have over-engineered the ITSM processes [...] That's not ITIL, that's the people building the process."

"It takes effort to want to change," added Ohio State University's Gribben. For example, "[training on new tools is] the cost of taking someone away for five hours a week -- for something that could save you 25 hours a week."

He also noted that it's human nature to focus on different tools, or more tools, rather than to address processes

"The C-suite says 'Wait we already bought you a tool' [so why do you need another one?] And it runs out of steam," Tedder said. If the IT department isn't serving continuous improvements that align it to the business, no amount of tools or processes or working groups will save them from shadow IT and outsourcing.

CommerceHub's Kokkili advocates doing like they did, and easing into DevOps, rolling it out with just one product team for the first year and building up acceptance with trainings, presentations and conference attendance. "Fail fast" might be a good mantra for application updates -- not so much for the people involved.

Meredith Courtemanche is a senior site editor covering data center and Microsoft technologies. Contact her at [email protected].

Article 4 of 6

Dig Deeper on Data center budget and culture

Get More Modern Infrastructure

Access to all of our back issues View All