A standard joke in World War II was military intelligence was an oxymoron – an impossible combination of opposites....
Today, substitute “military” with “business” and you’ve got the same joke. Some see the state-of-the-art, fancy processes of agile development as impossible to combine with stodgy mainframes, and with system administrators’ (sysadmins) attempts to manage the performance of legacy mainframe applications (apps) in particular.
Never fear! As it turns out, vendor tools and user best practices have recently arrived to allow you do just that. Before we get too excited, let’s see where we stand.
The unstoppable force of agile development
I have written extensively on agile development, and I see no need to repeat those highly positive analyses here. Suffice it to say that agile development is now on most software developers’ tongues; scrum and agile business intelligence (BI) – even continuous integration of tested modules into a large-scale new-version implementation – are steadily gaining ground in software development organizations of all stripes.
In the last four years, as agile developers learned how to scale with people and code size, vendor tools from past application lifecycle management (ALM) suites have become more pertinent to agile developers’ processes. These tools have also expanded their range so agile ALM now means not only continuous back-and-forth between testing and coding, but also coordination between operations and developers.
The immovable object of mainframe performance
During those same four years – especially the last two – upper management discovered a fact that has been blindingly obvious to sysadmins for many years: App performance, not just app uptime, matters!
For a long time, IT could just concentrate on keeping key apps running. However, today’s architectural complexity and the difficulties of scaling legacy apps have brought the chickens home to roost. Now, without good app response times, performance problems and incipient outages result in angry customers and unproductive employees at best, or additional slowdowns and outages from trial-and-error fixes at worst. These outages or slowdowns make your key applications unavailable for long periods. They impact or even halt sales and production cycles. They sometimes inflict crippling damage on the organization’s bottom line.
This problem is particularly important for a mainframe-based data center, and the legacy apps inside it. Not only do its legacy apps typically help run the business, it is increasingly home to Linux-based apps that are almost as critical and almost as impactful on global online customers when they slow to a crawl. And that’s not to mention business intelligence apps – no oxymoron here.
The result of increased corporate attention to legacy applications is that application performance management (APM) tool suites are finally gaining traction as targets for corporate IT investment. Meanwhile, vendors have engaged in an APM tool arms race to handle new architectural combinations. There are obvious choices like IBM Tivoli, CA and Gomez/Compuware and the not-so-obvious like Precise Software, whose tool provides an extra measure of analysis and the additional ability to suggest good fixes. It’s one I definitely recommend for use somewhere in the overall architecture. The result is tools that have made great strides in the key APM needs of sysadmins: monitoring that understands all software levels, sees apps across cloud and other networks, grasps the effects of workload competition and composite applications and uses its knowledge for quicker, deeper root-cause analysis and better fix suggestions.
The unstoppable force meets the immovable object
Agile development is likely applicable in Linux on your mainframe, though implementation on the z/OS side is likely to be difficult. Let’s go from there.
The idea is to integrate the sysadmin’s performance problem data into an agile development organization’s rapid-reaction bug fix processes and next-release efforts where possible. Hopefully, development teams will finally hear sysadmins’ screams of anguish over slow-performing, bottlenecking new releases and will deliver quick fixes. The sysadmin, in kind, gives agile developers valuable information about how the app is coming across to the customers in the real world after release. Let’s take those as the key goals and break them down.
The first step is to identify and implement an ALM plan that can span both agile development and mainframe admin – taking for granted the use of some sort of mainframe APM tool. This is not as hard as it sounds. The main ALM tools do indeed apply to some extent to both Linux and z/OS and their Linux sides to agile development on both the mainframe and other major platforms. (The main ALM tools run on and apply to development work on zLinux, other Linux, z/OS and other operating systems – Windows, UNIX, etc. – with a few exceptions. The parts that work on all of these platforms differ in significant details, but the differences are vanishing.) REXX and COBOL don’t fit neatly into agile development, but many efforts center around web services interfaces for composite applications and front-end legacy z/OS apps. In those areas, the proportion of Linux/web functionality tends to outweigh the amount of new mainframe assembler code. The new ALM should span almost all of your needs with relative ease with the expectation that you jury-rig the rest.
Once the desired ALM plan is in place and initial experience has been gained, we can go back to those key goals:
- Identify the root-cause code for the really important performance problems the sysadmin sees, using an existing or acquired APM tool.
- Trace when possible back to specific code in the app under the responsibility (right now) of a particular agile developer by having the sysadmin notate the code in the operational version stored in the ALM system and then getting it to ID the present developer of that code.
- Integrate the bug report into the ALM system’s continuous integration effort, specifying fixes for both the operational and current development versions. This is a tricky process which requires the developer to make a fix in the current development version such that it ripples back to the operational testing version via the testing team and doesn’t disturb the developer further.
- Intercept the newly fixed operational version and feed it back as soon as possible either to a pre-operational tester or directly to the sysadmin for quick isolated testing and replacement of the old operational version.
- Normalize the process once this identify/trace/integrate/intercept routine has been established in a few cases.
In most cases this will give you far faster operational bug fixes to satisfy the sysadmin, and a surprising amount of information for the agile developer on nasty performance surprises the customer won’t like and can be erased in the next weekly race for a prototype.
Best practices to make agile development work with your mainframe
You may have noticed a certain amount of hand-waving in my description of the agile developer’s role in this. If you’re a sysadmin you’re probably thinking, “What I want is the developer to fix my performance problem immediately, not lose it in a haze of versions and testing hand-offs.” That’s where best practices come in – agile best practices.
You see, with agile you don’t necessarily get what you initially think you want, but what you ultimately need. In particular, one agile best practice is constant refactoring, which not only makes code easier to change in any direction, but also reduces the likelihood the code in the current version differs greatly from the operational version. That means ripple back is much more straightforward, and you get your bug fixed in less than a week instead of in nine months.
The second agile best practice is “let the thousand agile flowers bloom.” In other words, instead of telling the developer to fix that bug or else, encourage the developer – or the tester, who can also bug-fix previous versions with agile ALM tools – to add more and more agility options with performance-tuning capabilities within the confines of the weekly prototype race. The very essence of agile is minimizing so called technical debt – the overhang of things that must be fixed later. Again, it’s the paradox of agile: we emphasize the maximum of change and in the process minimize technical debt. This will minimize fixes and improve quality. Forget time, cost and quality and we wind up doing better on all three than with traditional methods. In this case, that bug is fixed by the end of the week, if not before –hey, it’s continuous integration.
Then there’s the strangest best practice of all: improve the agility of the sysadmin. You see, the same ALM tools that give the developer insight into the sysadmin’s findings also give the sysadmin and his APM tool insight into the developer’s available libraries and processes. A smart sysadmin will be able to use those resources – with the support and guidance of the ALM tool – to fix bugs in his apps without the need of – or loosely coupled with – the development process. And that means the sysadmin can fix things without reference to the development team at all; with the ALM sending that information to the development team, the fix is inserted in their work where necessary – almost magically. It takes a bit to set up, but it should be worth it for both sides.
The bottom line for mainframe shops
Enough science fiction. Let’s return to immediate jobs and immediate payoffs. To review, you want to beef up your ALM and APM tools to handle mainframe/sysadmin/agile developer interaction. In the short term you want to see faster bug fixes for the sysadmin and more operational information for the agile developer. For maximum results, you can go for the best practices we mentioned. If you do, you should see an additional jump in customer satisfaction, increased sysadmin productivity and lower technical debt. After that, what’s next?
Well, why are you asking what next? The point of agile is flexibility to meet the unexpected, to seek out and embrace change instead of feverishly trying to anticipate it, and to boldly go – oops, science fiction. If you have to ask anyway, maybe you can look around for some oxymorons that agile can fix. Perhaps business intelligence, though that oxymoron may take a bit more time.
ABOUT THE AUTHOR: Wayne Kernochan is president of Infostructure Associates, an affiliate of Valley View Ventures. Infostructure Associates aims to provide thought leadership and sound advice to vendors and users of information technology. This document is the result of Infostructure Associates-sponsored research. Infostructure Associates believes that its findings are objective and represent the best analysis available at the time of publication.