On Saturday, May 31, the hosting company the Planet suffered an electrical explosion that affected about 7,500...
customers, 2,600 of which experienced extended downtime into Tuesday afternoon and Wednesday morning.
On Saturday afternoon, an electrical explosion took place in the underground conduit that brings utility power into one of the Planet's data centers in Houston, where the company is based. The resulting electrical fire destroyed the electrical gear where the utility service enters the building as well as the transfer switch and main distribution panel that feeds power into the first floor of the data center.
When the Houston Fire Department arrived on the scene, it ordered all power to what the Planet refers to as its "H1 facility" to be shut down, including the company's on-site backup generators. Shutting down power resulted, of course, downtime began.
"We worked closely with the fire department, and they did not allow us to use our on-site backup generators for safety reasons, which we understood," Doug Erwin, chairman and CEO of the Planet, said in an online audio message to customers posted on Monday evening. "As a result, our team jumped immediately into action and called in help."The Planet suffers prolonged downtime
The Planet facility restored power in phases: 5,000 customers with 6,000 servers on the second floor -- called Phase 2 -- saw power restored on Monday, while 2,600 customers with 3,000 servers on the first floor -- Phase 1 -- had to wait until Tuesday. Some in Phase 1 are still waiting.
As soon as power was fed back to the site, the Planet used one of its backup generators to return power to Phase 2. But given the extent of the damage on the ground floor, the Planet had to use temporary generators to power the Phase 1 servers. The company did so, but then suffered another setback early on Tuesday morning at approximately 2:30 a.m., when internal sensors within one of the temporary generators shorted out, causing another shutdown in Phase 1.
The Planet was unable to replace those sensors, compelling it to bring another temporary generator on-site on Tuesday afternoon. In the meantime, the company offered to physically move customers' servers to its other Houston data center, which it ended up doing for about 350 servers.
Tonight, Erwin said he will post another message online talking about service-level agreement refunds.
"We are very proud of our data centers, and we will put our money where our mouth is," he said in a second message delivered on Tuesday night. "We cannot control any act of God or freak accidents, but we surely can control our actions and work efforts."Users mixed on downtime fallout
For customers whose servers went offline, things got pretty unpleasant – especially for those in Phase 1.
"I know that the CEO of the Planet has been posting a lot of 'good news' recently," one customer, Robert Nagle, wrote early yesterday on a Houston Chronicle blog. "But for smaller noncommercial websites, the news has been mostly bad. The data center only deals with hosting services, and our hosting service has been giving us no news or confirmation of yet another delay. They are pretty much at the mercy of the Planet's prioritizing."
Another user wrote on the Planet's online forums: "Day four of watching my business crumble before my very eyes. This has gone far beyond ridiculous. 10 years of work down the Planet's drains. I give up."
Others commended the Planet for keeping its lines of communication open from the outset. Beginning with the company's update on Saturday evening and subsequent, frequent updates.
"First let me say one thing," a user wrote. "I think The Planet (is) doing a great job. The explosion at H1 is a disaster and what they are doing is incredible to get the data center up and running and I thank them for it."
This same user, however, also noted that customers need to be prepared for this sort of problem, and be ready with backup servers at other sites or otherwise be prepared to quickly secure them. Shortly after the Planet's power went down, this user ordered a virtual dedicated server from Go Daddy, a move that got his business up and running again in about a day. He said he's probably going to hold onto the GoDaddy server for backup and order a new one at the Planet's Dallas facility.
"I think I'm going to cancel the H1 data center server, just because I think I would do the Planet a service – one less server to worry about in that data center."