In these days of extremely tight budgets and ever-increasing energy and transportation costs, who would waste money? You have to look at your spending from various angles to see where you may have wasted dollars. For example, refusing to spend money on technologies that can reduce your disaster recovery (DR) deployment and testing costs actually wastes money. This is the first of the top
Not virtualizing your data center: Virtualization can save you money in DR maintenance and testing. I have talked to many customers who have leveraged virtualization to build DR solutions, even with applications that are not readily consolidated. In other words, they have implemented a 1:1 consolidation ratio just to gain the benefits of virtualization's mobility to simplify DR.
Consolidated virtual infrastructures can save even more money. Let's look at a real life example from a CIO of a medium sized business I spoke with. Like many companies diving into virtualization, they had virtualized the bulk of their applications and services in order to gain the benefits of consolidation. The CIO had not thought that he would see any additional savings and was very pleasantly surprised to see a dramatic reduction in DR testing time and personnel resources.
Before they virtualized, their DR tests (performed twice a year) would take three to five days to complete with seven to 10 IT staff involved. After virtualizing, the testing time dropped to one to two days and used only two IT staff. How much money does this save? Assume that a fully burdened IT engineer costs $175,000 per year and works 260 days per year. Best case savings for this company (10 staff for five days twice a year down to two staff for one day twice a year) is $64,616 per year and that does not include opportunity costs -- the work that was not being done by the extra IT staff that was busy performing the DR testing tasks. To be fair, let's look at the worst case savings (seven staff for three days twice a year down to two staff for two days twice a year) that would still be a savings of $22,884 per year. Bottom line: Leverage virtualization to avoid wasting money on DR testing.
Not maintaining your DR plan: You have spent considerable time and money on disaster recovery costs and your business continuity plan. You have the plan implemented, or so you think. You have performed an initial test or two. But business needs and tightening budgets drive you down the path of justifying putting off the next DR test until the current critical project is completed. By the time that project is mostly completed, the next critical project is looming over your head. Again, you delay your DR testing to focus on meeting the demands of the business. This vicious cycle continues and continues. Does this sound familiar?
I've seen this too many times, even in large mature IT organizations. Putting off your plan maintenance and testing means that you spent your initial DR investment in vain. You wasted all that money -- thousands to millions of dollars. Without ongoing testing and maintenance, history has proven time and time again that recovery is bound to fail, or at least take longer than expected as the staff wades through hundreds of small issues that would have been resolved through regular plan testing and maintenance. Bottom line: Maintaining and testing your plan is a critical project; without maintenance, any money spent on DR previously is mostly wasted.
Lack of CEO oversight and board involvement: I've heard too many stories where business continuity and DR is driven from the CIO and not the CEO. All of the companies that have experienced a disaster where the DR plan was driven by the CIO have had one thing in common: The IT department is up and running, but the rest of the business is broken and in a disaster state.
DR and business continuity are the safety nets for the health of the whole company, not just the IT department. DR planning, testing, and maintenance should be driven by the CEO and board of directors as a top priority. It represents protecting all corporate assets. When DR is not being driven by the CEO, all the money and time spent on DR by the CIO is in vain. It is wasted as the overall corporation remains in a non-functional state in the event of a disaster.
On the bright side, I talked to a CIO who had a CEO that drove DR across their company (a large distributor with five warehouses and a central data center). This CIO wasn't that excited about DR personally, but kept the plan going as a result of the CEO's continued vigilance. When Internet communications were cut to the central data center by an impatient back-hoe operator, the plan operated within parameters so that orders did get out by their deadlines. Some minor issues with the plan were discovered and repaired, and in the post mortem, it was determined that the plan worked. It had included the whole company, and it was the business units and employees that kept the business running while the data center was off-line. Bottom line: DR and business continuity are only effective if driven by the CEO and board of directors across the whole company. Any money spent on DR for IT alone without DR for the rest of the company is wasted.
Over protecting your environment: I have not seen this sin committed as often as the preceding mistakes, but I have seen it nonetheless. Executives tend to believe every service and component is critical and needs protecting at top levels -- especially at companies new to building business continuity plans. Handing them the bill for what they are asking usually brings a level of sanity back to the business unit head.
I have seen companies implement a recovery data center with site-to-site replication and hot, ready-to-go servers. They place all their services and applications into the hot site system regardless of system criticality. As a result, they protect systems at high costs that require much less aggressive protection at much lower costs (e.g., simple weekly tape backup stored off-site).
A detailed business analysis of each system can avoid such mistakes. Let's look at a slightly different example that can illustrate this point. I talked to the CIO of a large enterprise and he shared an interesting story. The sales division came to him asking for a customer account licensing tracking system for a very special class of customer. The company offered various levels of product licensing with the bulk of their millions of customers buying their standard product. The standard product license was already automated in an account tracking database system. This special class of customer license had some specific needs that could not be met by the general system.
The CIO put out RFPs and received proposals to retro-fit or add a completely new system with redundancy and protection for $2 to $3 million dollars. Then he asked the sales division how many customers are in this special case, and learned that it was currently about 400 customers. He then asked how fast that special licensing customer group was growing and what they expected it would be in three years. The answer, maybe ten percent in three years.
Armed with that business information, the CIO proceeded to purchase two locking steel file cabinets and hire two administrative assistants to manually track these customers. He arranged with the corporate records office to have copies of each customer record securely shipped to the corporate off-site archives for data protection and business continuity. The savings were between $1.5 and $2.5 million dollars. Bottom line: Look at the business aspects of what you are doing. Spending money on aggressive DR for all business processes may be wasting money.
Let us know what you think about this tip. Was it useful? Email Site Editor, Matt Stansberry.
This was first published in July 2008