There's good reason why most of us get that sick feeling in the pit of our stomach everytime we're about to hit the delete button. Bad things can happen. Ravinder Singh still gets that feeling 10 years after he hit the button. It didn't end in disaster, but it did result in some lost sleep. Here's his story in our latest installment of Mainframe Mishaps.
In the early 90's, I was working as a production control head in one of the leading financial institutions in Malaysia. One day during banking business hours, we started receiving complaints from the banking branches of a problem in one of the loan systems. We immediately notified our in-house application programmer who told us to close all loan files in CICS. After a short discussion with the application programmer, I was told to delete the journal file.
Requires Free Membership to View
When you register, you’ll also receive targeted alerts from my team of editorial writers and independent industry experts with the latest news, tips, and advice to help you do your job more efficiently and effectively. Our goal is to keep you informed on the hottest topics and biggest challenges faced by IT professionals today working with data center technologies.
Cathleen A. Gagne, Senior Editorial Director
|
 |
||||||||||||||||
|
|||||||||||||||||
It was around 11:30 am so all transactions for the day from 9:30 am to that time would be deleted. So I followed instructions and went ahead with the IDCAMS delete define job. Upon deleting the file, the application programmer called me and said not to do it but it was too late. Now, we had to recover the deleted transactions. In our organization, we were also recording all CICS transactions on 3420 tape. So to recover these transactions, I had to perform a FORREC, which I could only do after banking hours, i.e. at 11:00 pm.
So that night, I went to the office and as usual the operators brought down CICS to continue with batch processing for the other applications except loans. There was a total of five tapes altogether which I had to read in order to recover the loans transactions.
Everything was fine until I reached the third tape which gave me an I/O error. I could not go further from there. I was a little shaken until I realized that my organization also subscribed to the IBM DITTO utility. I used this DITTO utility to read the tape. After reading back all transactions from all five tapes, we began executing the batch job when there was a synchronization check between the online master file and batch master file.
This was the acid test. If it synchronized, it meant that all transactions were recovered, if not, I was in deep [trouble]. Luckily for me it synchronized and we were able to proceed with the batch processing and made the online start up time the next morning at 6:00 am.
It was one hell of an experience that I still remember after a decade or more.