Can you tell me about some of the disasters your company has weathered? Everybody has heard of the Oklahoma City...
Bombing. We had to survive it. Then there was 9/11. Then there was the tornado that ran its finger down the city. Our executives are aware of the impact of disaster.
Things like the Oklahoma City bombing, 9/11, a tornado -- you may avoid it 20-30 years. Eventually you're going to get hit. But our biggest risk in DR isn't the big stuff. You lose a grid of power and you're out 72 hours. What is the biggest challenge of disaster recovery planning?
The basic principles and problems are the same as they were 30 years ago, but they're much more complicated and you have fewer people to deal with them. We intend to double our premium [to $1 billion] and reduce our expenses [1% per year] by 2010.
Companies don't address disaster unless they have several million dollars to address it with. We don't have a huge budget compared to some of the larger companies. Our IT staff is 175 -- it's not huge. So carving disaster recovery out of a smaller budget is really tough to do. How do you cover disaster recovery on a tight budget?
We have added incrementally to our disaster recovery plan --facilities, staff and technology. We're constantly in the process of assessing and reassessing. We find things that don't work and we fix them before the next test. Success might mean we can do 80% of what we wanted to do and isolated the 20% we couldn't. What kind of disaster recovery testing do you perform?
We run two tests a year, and after every test we make major modifications. Our DR staff consists of two people -- but 50 to 100 employees are involved in every disaster test. The tests are 40-hours straight, timed, with an IT auditor watching.
We don't replicate the entire systems in the test. We take what sustains the business and that's what we recover. Some things in our recovery plan don't come back up for two weeks to 30 days. It's not that they're not critical -- they're not the core of the business.
We actually have four hour objectives during the test. Every four hours, a member of the team reports on where we're supposed to be. If we're not there, why are we not there? We're in a forty hour timed event from ground zero. It's an objective for the business to be back operating within 40 hours. How much does full-blown disaster recovery testing cost?
Too much money. It's not just the 40 hour test with SunGard. I also have the disaster bunker -- running the servers and facility cost a lot. What is the cost of preparing for a test? Typical preparation takes my staff on and off a month. Analyst time, system analysts, executives, six months out of the year with meetings and test activity going on. What's the cost of all of that personnel time? I don't want to add it all up. We just consider it part of the cost of doing business. We have a [DR] budget of just under a million.
There are 139 disaster recovery plans put together to cover what we do. My operating procedures for disaster recovery are in three binders. It's great to have it online, but when you're down, what don't you have? Internet. [My employees] want it in their hands. They need something while they're doing the work on the computer. It has to be instantly available. Putting it on paper allows them to have it instantly. I carry a paper phone list with me. What if my cell phone isn't with me? What are some things people don't think of in disaster recovery planning?
One of the things we've added is to test the call list. We're going to call the entire chain of IS and all of the business units to make sure all of the phones work. I have applications that haven't failed in years. Has anybody ever called that phone number on there? If our systems are running correctly, that call list is untested.
One of our top priorities in a disaster is human resources. You have a disaster, the last thing you want to have happen is not to be able to take care of employees. During the Oklahoma City Bombing, a lot of people don't want to go back to work. Is food on the menu in your disaster recovery plan? You want to have a full kitchen, catered food service, television, copier and fax service.
It's exhausting to go through a disaster. Your executives and cash flow will wear out. There has to be great vision for a company to survive a disaster.
Let us know what you think about the story; e-mail: Matt Stansberry, News Editor