Problem solve Get help with specific problems with your technologies, process and projects.

Lessons from Katrina: Part 1, Rebuilding a critical infrastructure

Rebuilt your data center after a disaster: Networking expert and hurricane veteran, Carrie Higbie shares tactics on how to rebuild your critical data center infrastructure after a disaster. This is the first, in a three-part series.

Editor's note: As a resident of Mississippi, Carrie Higbie is no stranger to hurricanes—she has experienced at least four. As a networking infrastructure design expert, it's her job to help companies prepare for a hurricane and rebuild in the aftermath. Katrina, however, is the first to affect her personally, having made its way up to Higbie's home about 90 miles north of New Orleans. Despite having no power, Internet access or running water, the experienced, and totally unflappable Higbie has written the first in a series of articles for on rebuilding your IT infrastructure after a disaster. Even if you're one of the fortunate and your data center is still standing, there are lessons to be learned and tales to be told. So read on.

In Katrina's aftermath, three states in the southeastern United States are currently in need of an infrastructure for power, telephone, Internet and other critical services. Food, water and shelter are certainly needed, but to get businesses back on their feet, and to bring the economy back into full swing, an infrastructure needs to be present. The businesses that had redundant sites outside of the hurricane area will need connectivity back to their redundant sites in order to function. It may seem like an impossible task, but that is certainly not the case. The secret lies in taking one step at a time.

Every company should have a business continuity plan. This plan is more in-depth than a disaster recovery plan, as it will include additional details surrounding business operations during critical outages. It will include a disaster recovery plan as well. Depending on the degree of interruption, the disaster recovery team will determine what steps in the business continuity plan to put in motion. Obviously, full interruption of services will dictate that a company starts in disaster recovery mode.

One task at a time

Much like networks have specific tasks for each layer, rebuilding an infrastructure can be approached one task at a time. The first and most important thing is to break out your business continuity plans and methodically move through them step by step.

Read on

Part 2: Operating from a remote site

 Part 3: Replacing active components

The first portion of the plan should include the contact information for each member of the disaster recovery team. No plan will be successful without full participation at all levels. The team should divide the tasks into manageable and actionable items from a control center. If the plan was implemented and tested, this will prove to be easier than shooting from the hip. Many business continuity and disaster recovery plans fall short of the facilities piece. This article will help end users understand what components need to be evaluated within a company's infrastructure.

The next step is to assess damages and the state of the critical services networks. Coordination with local power companies, telephone providers and Internet providers will be necessary and, in times of large outages, may take some time. Typically, services will be restored in an order of greatest impact first. The provider companies will assess damages and determine which areas will require minimal time to restore services to the largest number of premises. Companies on outlying areas may see longer times for service restoration. It may be necessary to find temporary quarters if the outage will extend longer than can be tolerated.

Rebuild, repair, replace

Whether a redundant site exists or not, working to restore the primary site will still occur. Depending on the level of destruction, an assessment team will need to determine if rebuilding a site, repairing an existing site or securing new facilities makes the best business sense. Assuming that a company plans to rebuild or restore the steps to restoring internal infrastructures will be similar. Following the example from the service providers, first an assessment is in order. Assuming that your structure is intact, it is time to assess your copper and fiber infrastructure. This step should be performed during the period of time when your active electronics are on order to replace damaged equipment.

There are two factors that will cause adverse reactions to cabling and infrastructure components, including connectivity. One would be water from firefighting, and the other will be flood and/or storm waters. If the water used to fight a fire was clean fresh water, the damage will be less than in the case of flood and storm waters. In the aftermath of a storm, however, even the water used to fight fires may not have the same clarity and purity levels as before the storm. Water will render most copper connections useless unless they were encased in an industrial rated connector or enclosure. As water carries current, if the connections were live or are energized while wet, the damage will be greater. Arcing can occur from contact to contact and, in some cases, may actually fuse the conductors together.

The damage you can't see.

In areas where flood waters have encroached on structures, several factors will determine how much of the infrastructure may need to be replaced. Salt water is corrosive to copper connections and salt particles may harm fiber connections as well. Salt can coat the connection points whether visible to the human eye or not. In some areas, flood waters are toxic and contain other corrosive materials such as gas, diesel and a variety of chemicals. These chemicals can cause a leaching effect for the cabling jacket materials and could degrade the jacket material over time. This can include the overall jacket material, as well as the jackets on individual conductors. Should the conductors become exposed, a short can occur that could damage any active components within the channel. If Power over Ethernet is utilized on the channel, low voltage leakage can also occur.

Any portion of the infrastructure that was submerged for any period of time should be replaced. While they may perform in initial tests, the corrosive nature of the contents of the flood water could cause the infrastructure components to fail over time. Replacing one cable at a time is more expensive in the long run than actually replacing all components. In areas where flood waters did not actually cover the cabling, but the cabling was exposed to the vapors of the corrosive water, the same may hold true. For instance, in a five-story building where the lower two floors were submerged, the vapors may harm the third floor, but is not likely to harm the fourth and fifth floors if they were adequately sealed from the lower floors. This would include complete firestopping in the riser system. Crowded risers that do not have adequate firestopping material will be affected. If the firestopping material is not rated for such exposure, it may not provide protection from vapors and the riser system will need to be replaced.

What's left to salvage?

The only cabling materials that may survive from a flood would be industrial-rated cable jacket materials, provided that it can be verified that no substance has come in contact with the jacket that it is not rated to withstand. Also, it would be advisable to determine air quality from fumes and vapors in this scenario. The same can be said for industrial connectors. Be aware that many of the ratings for cabling and conduit in an industrial setting are limited to time of exposure, length of immersion and depth of immersion. You will want to verify with any manufacturer what these limits are. If they have been exceeded, it will likely need to be replaced.

If the cabling was encased in sealed conduit, it will be necessary to verify that all seals were kept intact and that no seepage occurred into the conduit. The next step would be to evaluate the conduit and make sure that it was not exposed to corrosives that will degrade it over time. Conduit tends to corrode in shears. The sheared off metal creates sharp edges that may harm enclosed cables.

The best method is to plan on replacement unless the factors mentioned can be eliminated from concern. As mentioned, even if the systems function in the early days of the aftermath, degradation over time will be more costly to repair and harder to detect. Faulty systems will surface in retransmissions, bit errors and eventually failure. Failures can harm active components on the network. All cabling both copper and fiber will need to be assessed up to the point of carrier demarcation. Carriers will handle the demarcation points, but will typically require a call from the customer in order to evaluate the actual in-building connections at the demarcation point. As the cabling is designed to be a 10- to 15-year investment and will support many iterations of active devices, it is better to be safe than sorry. A solid infrastructure will aid in recovery efforts.

Carrie Higbie has been involved in the computing and networking industries for nearly 20 years and has taught classes for Novell, Microsoft, and Cisco certifications as well as CAD/CAE, networking and programming on a collegiate level. Carrie currently works as the Network Applications Market Manager with The Siemon Company, where she provides liaison services to assure harmony between active electronics and networking infrastructures. She participates with the IEEE, TIA and other consortiums and works to further educate the end-user community on the importance of a quality infrastructure.

Dig Deeper on Enterprise data storage strategies

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.