Even data centers with plenty of backup power go dark once in a while. It happened to data center colocation giant...
365 Main Inc.'s San Francisco building when generators didn't fire up this week.
According to a statement from Miles Kelly, vice president of marketing at 365 Main, at 1:49 p.m. on Tuesday, July 24, 365 Main's San Francisco data center was affected by a power surge caused by a PG&E Corp. transformer failure in a manhole under 560 Mission St.
The 227,000-square-foot facility first opened in 2002 and is located a short walk from the city's financial district. The building serves as a colocation facility to CNET Networks Inc., Esurance, Sun Microsystems Inc. and many others.
While the facility has installed backup electrical infrastructure to protect against power surges, an initial investigation revealed that certain 365 Main backup generators did not start when the initial power surge hit the building. On-site facility engineers responded and manually started compromised generators, allowing stable power to be restored at approximately 2:34 p.m. throughout the entire facility, according to Kelly's statement.
As a result of the incident, certain customers' continuous power was interrupted for up to 45 minutes. The company confirmed that three of the eight colocation rooms were directly affected, and impact on other colocation rooms is still being investigated.
The colocation company frequently reports providing customers with 100% uptime and boasts 24/7 availability via two complete backup systems.
In the unlikely event of a cut to a primary power feed, the state-of-the-art N+2 electrical system instantly switches to live 3,000 horsepower backup generators to "keep the data center continuously running and to shield tenants from costly downtime," the company reported in a recent press release.
In March the company announced it had provided craigslist Inc. with 100% uptime since the networking site became a tenant of 365 Main's San Francisco facility a year ago.
Craigslist, which receives more than five billion page views per month, was one of the companies affected by this outage, as were Technorati Inc. and LiveJournal, though 365 Main could not confirm that these companies were affected due to confidentiality agreements.Finding the cause
365 Main is working with Hitec Power Protection, Valley Power Systems Inc., Cupertino Eliectric Inc. and PG&E to further investigate the incident and determine the root cause that prevented certain generators from starting.
In the short term, all generators in the San Francisco 365 Main facility, which are currently operating normally, will continue to operate on diesel until the cause of the event has been identified and corrected. The company will not switch back to utility power until PG&E ensures that utility power is stable, the statement said. Generators are currently filled with more than four days of fuel, and additional fuel has already been ordered, the statement said.
For affected customers, stable power resumed approximately 45 minutes after the power surge. The outage affected between 20% and 40% of 365 Main's customers. Because of confidentiality agreements with customers, the company did not disclose information on the specific companies affected.
Though 45 minutes seems like a short period of time, once power returns to affected areas, customers must also bring their own equipment back online. Depending on the complexity of a site's operations, the process of bringing a company's facility back online can take minutes, or it can take hours.
"We will apply knowledge gained in this investigation to all 365 Main facilities to help prevent this type of incident from happening again," 365 Main's Kelly said in a statement. "We sincerely apologize for the impact this has had on our customers' operations. We understand the seriousness of this issue and will provide more details as they come available."
Let us know what you think about the story; e-mail Bridget Botelho, News Writer.
Also, check out our news blog at serverspecs.blogs.techtarget.com.