"We survived Ike."
That was the first thing I heard when I called my friend in Houston on Sept. 15, 2008. Actually, Roger Scott, CEO of Houston-based Cytel, formerly Cypress Telecommunications Inc., did a little better than just survive
Cytel prospered because it had thought about disaster recovery (DR) planning in advance and carefully chose its primary place of business. Cytel stages its DR services out of one of the best possible places -- a telephone carrier colocation (colo) facility, sometimes referred to as a carrier hotel for service providers.
My company is based in Dallas, which is 230 miles from Houston. We were lucky to experience very little wind and hardly any rain during Hurricane Ike. The fact it struck so close to us, however, reminded me that even hundreds of miles inland from landfall, we still could have been vulnerable if the storm had taken a different course.
Colocation facilities are ready to weather a disaster
Ike scored a direct hit on Houston, with the eye of the storm passing right over downtown. It was a strong Category 2 storm, with winds around 110 miles per hour when it made landfall near Galveston, Texas. At the time of this writing, Galveston is still only a shadow of its former self, and the damage will take years to correct. Downtown Houston is a few miles inland but still took a pounding, with thousands of blown-out windows in its many high-rises. The 75-story, 1,002-foot-tall J.P. Morgan Chase tower, for example, spewed out not only glass but also -- according to various news articles -- insulation, furniture and computers. The downtown area of America's fourth-largest city was in shambles.
Cytel was affected as a disaster recovery and a telecommunications provider. Cytel started out in the long-distance business about 20 years ago, became a competitive local exchange carrier (CLEC), and most recently became a DR provider. Throughout the years, Cytel held firm to its telecom carrier roots, a move that saved the company considerable anguish while recovering from Ike. Cytel operates out of a hardened colocation facility at 2 Shell Plaza in Houston. Its data center is located in the downtown area most affected by the storm, yet the company survived. There are two main reasons for its success: its disaster-ready facility and its location in an area with an effective drainage infrastructure.
Like many colocation providers, Cytel's facility employs backup power, generators and batteries. (This includes 48 V power -- the kind of data center power that generally runs telecommunications equipment like private branch exchanges (PBXes) and multiplexes. Don't forget it in your plans!) Colocation facilities often contain many other useful features, such as redundant power grids, backup generators, rack space, raised floors and good physical security. These features paid off for Cytel when Ike struck.
Initially, only two power grids in Houston survived Ike. One was the grid that served the downtown Houston Medical Center area and the other served the Cytel colocation facility. The rest of the city was dark, and this makes sense. Since both grids were downtown, chances are that the electric distribution facilities were buried. However, while fiber optic facilities work when completely immersed in water, the sensitive electronic equipment that powers them do not. Flooding disrupted buried copper facilities during hurricanes Gustav and Katrina in New Orleans, and when the water rose to a point where equipment was immersed, the fiber optic facilities failed too. Even so, it can be expected that major downtown areas have more effective storm drains and infrastructure for carrying away floodwater. In the suburbs and outlying areas, the issue of aerial cables and a relative lack of storm drainage are more pronounced. Consider the fact that, keeping with their carrier roots, most colocation facilities are located in urban areas, adjacent to primary telecom hubs. That's another reason to think about staging recovery technologies out of a colocation facility. The infrastructure serving these facilities is more likely to drain water, lest you have to tread water.
Carrier colocation offers quick disaster recovery
Oftentimes, a sprit of service exists among telecom providers. When disaster strikes, many competitive rivalries among carriers have been known to go out the window. Virtually all carriers perform commendably and honorably when lives and property are at stake. One finds the largest concentration of carriers in colocation facilities -- another great reason to consider them. There are a lot of people who can help you in a carrier hotel, and they have a demonstrated history of doing so. This was the case with Cytel and others affected by Ike.
Once Cytel employees returned from home (remember, your responders will always go home first -- take that into account in your recovery plan), they had all customers back online almost immediately. For example, Cytel restored a large exchange server for a major company only 12 hours after the storm in order to give that company email access. (This was more than acceptable -- consider the fact that a carrier's customers go home first, too. By the time the customers implemented their recovery plans, Cytel had them back online.) While this customer did not have phone service at its primary location due to lack of power, email access allowed the company to coordinate its recovery. Without the restoration of its email exchange server, this would not have been possible.
One moral of this story the importance of the 4CI principle (Command, Control, Communications, Computers and Intelligence) in any disaster. You can find a number of articles on 4CI at http://www.4ci.us. Suffice it to say that if you cannot communicate with your responders, you cannot recover. Since nobody can be sure which communications media (wireless, fiber, copper, IP, etc.) will survive after a disaster, a sound strategy is to move in with a good reputable colocation provider.
How do carrier colocations enhance a DR strategy?
Colocations, or "carrier hotels," are exactly what the name implies. They are generally well-hardened facilities and, almost without exception, allow carriers to connect with one another under day-to-day or emergency operations. They can be indispensable in a disaster. A typical colocation can house one or more of the following technologies:
- Cable and fiber optic connections to the incumbent local exchange carrier (ILEC), such as AT&T, Qwest or Verizon.
- Connections to all major long-distance companies
- Connections to competitive local exchange carriers, or CLECs
- Connections to wireless providers
- Connections to wireless Internet service providers (WISPs)
- Connections to Voice over Internet Protocol (VoIP) carriers
- Connections to satellite communication providers
It is prudent to plan in advance so your organization can put any of these features to use depending on the circumstances of the disaster. If connections to the ILEC go down, for example, connections can be quickly purveyed from a CLEC or long-distance provider. If everything goes pear shaped, like it did during Katrina, satellite and VoIP might still survive. You will obviously need a place to connect to these services, and again, colocation fits the bill. Satellite is often the only link left to a 21st Century communications infrastructure after large disasters such as hurricanes, earthquakes and tsunamis, and it's particularly effective when combined with other technologies like IP.
IP dates back to a time when U.S. war planners envisioned scenarios in which every AT&T primary central office would be in the upper atmosphere due to a nuclear attack. IP was designed so packets of information could bypass lost hubs and get through on the facilities that survived in outlying areas. This technology is equally useful today when major communications hubs are affected by disasters or terrorism. Even better, the technology allows voice to be carried, hence the term Voice over IP, or VoIP. After all, voice is really a form of data. Data is data too. Any questions? (Seriously, the same bit stream is used to carry voice or data, whether it's 64 Kbps, 1.544 Mbps or any other speed.) As it was stated in an old chicken commercial, "parts is parts"; in the case of telecom capacity, "bits is bits," and these bits can be used to carry voice or data. In this regard, carrier colocations are literally the Kentucky Fried Chicken restaurants of cyberspace.
When you sum it all up, it is possible to maintain 4CI on both a data and voice basis by cobbling together surviving transport communication facilities and employing IP. There is no better place to do this than in a colo. Interestingly enough, this is precisely how the military planners of the past envisioned IP would be used, except now you can carry voice data over IP. Today, civilian organizations demand military-level disaster recovery.
Summary and Take-Aways from this Article
To summarize, in our own "spirit of service," here is a quick checklist of considerations on what will happen after a large disaster impacts your organization. Pay particular attention to the services that can be accessed in a colocation.
- Paging systems, including two-way paging, should work because they are often satellite based. This includes Blackberries.
- In general, satellite services shouldn't be impacted. If you don't use this technology day to day, you should at least know where to connect to a satellite carrier (e.g., in a colo). However, if satellite is your primary technology (for example, if your organization is a TV network), you must plan for a backup in the event of a satellite outage.
- WISPs should be back on the air relatively quickly since there is no licensing requirement and the equipment is inexpensive and portable. WISPs often beat the phone companies in establishing Internet connectivity to affected areas. If you have access to the Internet, you have access to IP. If you have access to IP, you can restore 4CI in terms of voice and data. WISPs can often be found in colocations.
- For reasons explained earlier in this article, there is a strong probability that anything traversing a cable (aerial or buried) will be affected. Check into route diversity on cable facilities. Many different providers of wireline services can be found in a typical colo.
- Electric power will be impacted if the disasters in New Orleans and Houston are any indication. Plan for backup power, including 48 V for telephone and PBX equipment. Also plan for a reliable fuel supply for your generator. Why make these necessities into big capital expenses when they come with the monthly rent in a carrier hotel?
One final tip: If your organization is an essential service (hospital, government, etc.) look into Government Emergency Telecommunications Service (GETS), Wireless Priority Service (WPS), and Telecommunications Service Priority (TSP). These government-sponsored programs allow for priority when phone lines are saturated and wireless frequencies fill up after a major disaster. Again, this helps you maintain 4CI. One big surprise in the wake of Hurricane Katrina was how few qualifying agencies and essential services had actually signed up for these restoration priority programs. Don't wait until you need them -- look into them now.
In conclusion, we are happy to report that Cytel and its customers are alive and well due to effective use of colocation facilities and some well-thought-out advanced planning. Remember, colocation providers house all sorts of useful communications and, if employed properly, can leave you "room to live" after a major disaster by providing immediate access to surviving telecom infrastructure. Once basic connectivity is established (WISP, CLEC, satellite, etc.), today's technology allows even more flexibility. For example, with IP and VoIP provided by these facilities it is possible to restore 4Ci (voice or data) and thereby recover your organization, regardless of circumstances.
We hope you enjoyed this article. Keep in mind that for a reasonable cost, a good colocation provider will pay handsome dividends to your disaster recovery plan. Until next time, best of luck and happy DR planning.
ABOUT THE AUTHORS:
Leo A. Wrobel pioneered carrier colocation in 1986, when he became the first person in the U.S. to put a computer disaster recovery center inside a telephone central office. Leo is active author and technical futurist, and has published 10 books and over 400 trade articles on many technical subjects. He is presently CEO of Dallas-based and b4Ci Inc. You can contact Leo at email@example.com or by phone at (214) 888-1300.
Sharon M. (Ford) Wrobel conducts extensive publishing and regulatory research as Vice President of Business Development for b4Ci Inc. Sharon was a major content contributor to Leo's book Business Resumption Planning Second Edition and a co-author of Leo's latest book, Disaster Recovery for Communications and Critical Infrastructure. She published over a dozen trade articles in 2008 alone. You can email Sharon at firstname.lastname@example.org.
What did you think of this feature? Write to SearchDataCenter.com's Matt Stansberry about your data center concerns at email@example.com.
This was first published in February 2009