Editor's note: In the last of our five-part series on rebuilding your data center after a disaster, Carrie Higbie has tips on how to make your redundant site redundant—including checklists for hardware, software, circuits, cabling infrastructure and data.
Congratulations! You can consider yourself an expert in disaster recovery, business continuity and lack of sleep. But one step still exists. Making your redundant site redundant again will require a bit of planning, a certain amount of expense and the final phase in resuming normal operations. ((Content component not found.)) It may be that a redundant site is a new service, but the tasks will largely be the same. It is not as simple as working this plan backwards, but rather working through the remaining tasks in a methodical manner. If you didn't have a business continuity and disaster recovery plan, now is the time to develop one using the steps previously taken and working in the lessons learned through this disaster. Be prepared to do some serious typing and in about 90-180 days going through a "mock" disaster to assure you haven't missed anything. If you need a fill in the blank disaster recovery plan, feel free to call Siemon and one can be provided for you free of charge.
If you are new to redundant sites after a disaster, or if you have rebuilt your primary site after both primary and redundant sites are lost, evaluate the process in terms of downtime, lost productivity, lost revenues and lost opportunity. This will help define your budget. Remember that during your outage, your competition was likely introduced to your customer base. This may have an even greater impact on future business. However an efficient plan for continuity, including your redundant site, may be enough to sway some of them back. It can be a great marketing tool to regain customer confidence.
Evaluate your hardware, software
In order for a redundant site to be effective, it must be a true reflection of your primary site. This will include hardware, software, circuits, cabling infrastructure and data. If circuits were temporarily increased to address additional communication needs, they can probably be restored to their original configuration provided that the original bandwidth is adequate for back up needs. Additional circuits that were added for specialty communications may be able to be turned off for an additional monetary savings.
Step one is to evaluate your new hardware and software in comparison to the newly procured equipment and applications that are now functioning at your primary site. Since applications are replaced by newer versions and hardware is upgraded, in order to assure smooth operations and mirroring, you will need to bring your redundant site in line with your primary site.
Using the budget guidelines that you developed in part 2 of this series; now is the time to begin vendor negotiations. Many vendors have trade in and upgrade programs. These will provide a savings to your company over just buying replacements.
If you are using a value added reseller for your hardware and software, you will probably want to stick with the ones that worked with you already. Even if the products are a bit more expensive, the valuable lessons you learned together can decrease the implementation expenditures and assure that any change orders are minimal. If, on the other hand, their services were problematic, this is a good time to evaluate capabilities of new implementers or maybe even new outsourced redundant services from other vendors. Solicit the input from your recovery team to determine overall satisfaction and areas for improvement. This will allow you to make this process a smooth one.
Evaluate cold site capabilities
Some companies develop consortiums and cold site services to each other. If you do not plan on having a live redundant site, this is a good time to evaluate cold site capabilities, locations and functionalities. In a cold site scenario, it is imperative to maintain backups as well as inventories and configurations for all hardware and software. It will take more resources to do this, but in some cases may provide a savings over mirrored site services, as you do not have the expenditures of monthly circuit fees, dual hardware maintenance agreements, etc. However, as you have probably figured out by now, you will be down for a longer period of time during an outage as the restoration occurs and maybe even as new hardware and software are ordered.
Installing new software on older platforms can have mixed results. While it may seem like a savings at first, troubleshooting and other problems may nix that savings in a hurry. You can certainly test the capabilities, but be aware, that in the future, every software upgrade will carry an additional testing burden for the redundant site over and above the testing at your primary site. Penny wise and dollar foolish in the long term are never a good combination.
What can you reuse?
It may be possible to reuse components of older systems within or connected to newer systems and save a bit more. For instance, you may already have high speed network cards that are costly but could be reused within new server boxes. The same can hold true for storage switches and other components that are the same as your new site. This is where your initial budget and comparison charts will help that was developed in Phase II.
Verify that older communications hardware such as switches and routers will work properly with any new services that may be added to the redundant site. Look at your security systems to be sure that all security services are an exact mirror of your primary site. This step is critical as maintaining two security configurations will be a nightmare in the event of another outage. More than likely, your new security settings are not going to be in sync as users tested the new system for functionality and new operating systems have varied requirements. However, if like software and hardware are in use, the new security parameters can be replicated over the links to your redundant site.
While the hardware is on order, two sets of backups should be performed on the redundant site and stored in two separate locations. These should be clearly marked "do not delete" and should be maintained for the longest time allowed by your policies, but at a minimum 3 years or as required for tax purposes. Remember, there may be some once a year or once a quarter processes that did not get tested during the 30-day testing phase like end of year reports and tax returns.
A new inventory of hardware and software for the primary site and redundant site should be archived at the redundant site and an alternative site such as a bank vault or stored online at an outsourced web site after being encrypted and password protected. If your company has multiple sites outside of the same local area, this can be stored at other site locations and your site should reciprocate for their sites.
Once all services, hardware and applications are in sync, you are not quite off the hook. You will want to plan a mock disaster within 6 months using your updated business continuity/disaster recovery plan to make sure that nothing was omitted from the new documentation. Lastly, you will want to schedule a quarterly review with your disaster recovery team to be sure that contact information, software and hardware information, etc are maintained on a regular basis. You may have learned that your vendors suffered the same outages as you did and new vendors were selected, for instance, so in your new plan, you may want a single vendor and two alternatives listed in your plan moving forward. All of these considerations will assure that, heaven forbid, there is a new disaster that your business will have true continuity!
Carrie Higbie has been involved in the computing and networking industries for nearly 20 years and has taught classes for Novell, Microsoft, and Cisco certifications as well as CAD/CAE, networking and programming on a collegiate level. Carrie currently works as the Network Applications Market Manager with The Siemon Company, where she provides liaison services to assure harmony between active electronics and networking infrastructures. She participates with the IEEE, TIA and other consortiums and works to further educate the end-user community on the importance of a quality infrastructure.
