Sempra's primary 50,000-square-foot data center is located in Rancho Bernardo, Calif. Its secondary 15,000-square-foot facility was used mainly for test and development, but Data Center Manager of the Year finalist William Stewart believed that there was an opportunity for the company to move its DR operations there.
"We started to develop plans over the last five years that the site in Los Angeles would be multipurpose – test development and training, as well as production failover," he said. "So we started bringing disaster recovery and business continuity in-house instead of outsourcing it."Disaster recovery contract with SunGard
Sempra had a large contract with SunGard to provide disaster recovery services in the event of a problem at its primary data center. Wintel and Unix servers were backed up in SunGard's Phoenix facility, while its mainframe workloads were supported in a SunGard facility in Philadelphia.
In addition to the cost of the contract itself, Sempra spent a lot of money sending staff to Phoenix and Philadelphia for DR testing. It also couldn't recover its main systems as quickly as it could by doing data replication at the Los Angeles data center.Making the case for L.A.
Stewart decided that moving DR operations to the Los Angeles-area data center could save the company serious money. But there was concern about having both data centers in southern California, which is a potential magnet for natural disasters.
So Stewart performed an in-depth analysis of the risk factors involved and considered moving the DR site in-house and to its Los Angeles facility. He found that the primary site in Rancho Bernardo has the fewest risk factors in the state, while the Los Angeles data center is not located on the San Andreas fault.
Still, the Los Angeles data center was outdated and required significant renovation. And because California heavily regulates its utility companies, Stewart had to appear before a state committee to make the case for changing the company's DR site location.Renovating the L.A. data center
Located on the second floor, the Los Angeles data center needed some overhauling to prepare it for DR operations. Some areas of the raised floor were made of wood, which is a fire hazard. Other portions had cracks in the seams, and Stewart was concerned about how much weight the floor could handle. So Sempra replaced the raised floor.
Next, the power and cooling infrastructure was outmoded and also needed to be replaced.
Security was yet another major problem. Surveillance cameras weren't configured effectively, and the security doors did not function correctly. In addition, the site did not have so-called zoned security, so secure entrance was an all-or-nothing proposition. If someone were allowed entrance to the data center, he had access to all locations inside the facility. So Sempra installed mantraps to prevent tailgating and installed additional security cameras.
Stewart said Sempra also focused on virtualizing its x86 Hewlett-Packard Co. servers with VMware and its IBM Power-based servers with AIX, which has saved money on costs and management, he said.Ending a costly contract
In January 2007, Sempra began the work, and by November of that year, it began phasing out its contract with SunGard in November Stewart retained the contract for another quarter as a backup, but now Sempra has concluded its contract completely. In all, Sempra has saved $700,000 a year in its move to the Los Angeles facility, $500,000 of which comes from termination of the SunGard contract.
Other cost savings came from reducing the travel budget (no more on-site DR tests are required in Phoenix and Philadelphia) and by reducing tape backups (sent to an off-site Iron Mountain Inc. facility) through data deduplication.
"It was an exciting, successful project," Stewart said. "My budget is reaping the benefits, and those are hard-dollar savings. The IT organization likes it, and the executive management likes that."Editorial note: Supporting engineering firms and vendors for the project included: CA, IBM, Hitachi Data Systems, and Technology Services and Safety Solutions.