Techniques to extract and analyze CMF mainframe performance data

Administrators can use DFH$MOLS and Rexx to process mainframe performance data then import in Excel to do the analysis.

For mainframe administrators, it’s important to ensure  the “big iron” and every application running on it is performing at its peak. There are tools that can provide system details, but administrators need a way to analyze and visualize CICS performance data. This tip shows you how to tap the mainframe and interpret that data with Microsoft Excel.

IBM mainframes under z/OS and z/VSE include a transaction server called the Customer Information Control System (CICS). This transaction manager is designed to handle and track enormous processing loads. An element of CICS, the CICS Monitoring Facility (CMF), creates multiple record types to track CICS and system performance. Performance records are interesting to anyone tuning an application or dissecting a problem, but mining CMF data can be cumbersome. The records have a self-defining layout that is difficult to parse without complicated programming. What’s worse, the records can’t really be interpreted without a CMF dictionary record.

You can find tools to process CMF records, but even shops that don’t have these tools can still mine and visualize performance records using in three steps:

  1. Use the CICS utility DFH$MOLS to “unload” the performance records.
  2. Write a short script to convert the unloaded records into a comma separated value (CSV) file.
  3. Load the CSV file into Excel which can process and graph the data.

Using DFH$MOLS to unload mainframe performance data
According to the CICS Operations and Utilities Guide, the DFH$MOLS program included with CICS is used to selectively print CMF records, but DFH$MOLS can also “unload” CMF records, transforming them into predictable, fixed-length records. This makes the data easier to process and manipulate.

A simple script can tackle the process for you. Here’s a sample Job Control Language (JCL) script to unload CMF data with DFH$MOLS into a fixed length record file:

//SYSIN     DD  *

The UNLOAD control card at the end of the script tells DFH$MOLS to write the unloaded CMF records to DD card CMFOUT and use the system’s local time for timestamps. You must specify SORT for UNLOAD requests so that DFH$MOLS processes dictionary records before any performance data which lessens the chance of running into a U109 ABEND as described below. DFH$MOLS supplies the dataset control block (DCB) values for CMFOUT so they aren’t specified in the JCL.

Remember that DFH$MOLS has some quirks. For example, it doesn’t issue non-zero return codes. Instead, it uses an abnormal software termination (ABEND). Sometimes the ABEND is accompanied by a message. Check the CICS utility guide to get the full meaning of any messages.

One of the most frustrating aspects of DFH$MOLS is that you must supply dictionary records to interpret the performance data, and failure to do so will result in an U109  (no dictionary established) ABEND. Luckily, you can create dictionary records using IBM’s DFHMNDUP utility. Here’s another sample JCL:

//          UNIT=DISK,SPACE=(TRK,1)
//SYSIN     DD  *

Now you’ve created a CMF dictionary record for region CICSPROD. The MCT parameter specifies the suffix for the monitoring control table (MCT) used by the region to tailor performance records. This is so DFHMNDUP can load the MCT from the job’s STEPLIB concatenation.

For flexibility, a job using DFH$MOLS contains two steps. The first step creates a dictionary record for the CICS records you want to unload, and the second step invokes DFH$MOLS to unload them. The two sample scripts above cover both steps.

An alternate strategy for advanced users would be to use DFHMNDUP to create a permanent dataset of dictionary records for every CICS in the Sysplex, then any job using DFH$MOLS can reference the dataset for the requisite dictionary records.

Processing mainframe performance records with Rexx
The format of the unloaded records is almost as mysterious as the original. Although IBM supplies DSECT DFHMNPDA in library SDFHSAMP to describe the record layout, CICS lacks a high level language version of DFHMNPDA. Anyone wanting to use the unloaded records must either write an assembler program or create his or her own structure in COBOL, PL/1 or C.

Fortunately, you can also use REXX, which works well with character data, to create a quick script to process the unloaded records. Each record should be read as one long string so the program can use the built-in substring (SUBSTR) and conversion functions to pick off interesting fields. Since the DSECT starts at zero and string positions at one, the beginning of each field in the SUBSTR call should be one more than the offset listed in DFHMNPDA. For example, if the offset for the total CPU field (PDRCPUT) is 1,784, the SUBSTR function should use 1,785. Below is a sample Rexx code fragment:

numeric digits 20
“execio 1 diskr cmfunld”
do while(rc = 0)   
cics = substr(inputRecord, 1, 8)   
cputime = c2d(substr(inputRecord, 1785, 8)) / 4096) * .000001)   
cpucount = c2d(substr(inputRecord, 193, 4))   
csvout = cputime”,”cpucount”   
“execio 1 diskr cmfunld”

The numeric digits statement at the beginning sets the maximum arithmetic variable width; failure to set this value will cause errors in Rexx’s conversion routines. Also worth noting, CMF datasets tend to be too large to use Rexx facilities like queue or stem variables, so the second line indicates that the records should be read one at a time. The cputime line shows how to convert time fields for CICS/TS 4.1 and later. Note that it is divided by 4,096 and multiplied by .000001 to get the microseconds. Finally, the cpucount line converts to a count field. After picking out the interesting fields the output should look something like this:

“Start time”,”CPU Time”,”Response Time”,”Transaction Name”,”CICS”

I chose this file layout because:

  • Excel turns the top line into column headers so it’s helpful if the transformation program includes them in the first row of the file.
  • The double quotes around a field tell Excel to format them as character strings. Number fields without quotes should import correctly. Since Excel has trouble understanding time fields with lots of decimal points, it is best to insist that they be treated as character strings.
  • Excel looks for end-of-line terminators (ASCII characters “carriage return” and “line feed”) at the end of each row. This means Excel may interpret a fixed block file as one long line in a single row spreadsheet. To get the file into the correct format, insert these characters yourself or use a TSO command like OCOPY to convert the dataset into a Unix System Services (USS) file. The best solution is to write the file directly to USS.

Importing mainframe performance data into Excel
Once you’ve downloaded the CSV file to a workstation, it’s time for Excel’s data import dialog. Click on the data menu and select “From Text” on the ribbon. This opens a window where you can select the file.

Figure 1: Use Excel’s text import wizard to load data from the .CSV file in a suitable format.

The text import wizard appears as in Figure 1 and asks how to interpret the file. Select comma delimited fields. You may have to play with the “file origin” selection to make Excel correctly interpret the double quotes in the data.

Figure 2: In Excel’s Text Import Wizard, you may need to override the formatting preferences.

The screen shot of the Text Import Wizard in Figure 2 shows you how to override Excel’s formatting preferences so the date field appears as plain text.

Figure 3: Once the .CSV file is imported, use Excel’s analytical capabilities to evaluate and visualize the mainframe’s performance.

Once the import completes, you will have a spreadsheet of data as in Figure 3 that is ready to be graphed and analyzed with Excel’s processing and visualization capabilities at your command.

About the expert: Robert Crawford has been a systems programmer for 29 years. While specializing in CICS technical support he has also worked with VSAM, DB2, IMS and assorted other mainframe products. He has programmed in Assembler, Rexx, C, C++, PL/1 and COBOL. The latest phase in his career finds him an operations architect responsible for establishing mainframe strategy and direction for a large Insurance company. He lives and works with his family in south Texas.

Dig Deeper on IBM system z and mainframe systems