"We lost all city utility as far as water pressure," said Russ Burlew, the data center facility manager. "It took three and a half days to repair that line, and we were without water. We had vendors come in with water trucks and pumps to keep our tower full."
"When everyone talks about Tier IV [data center], they talk about power and cooling redundancy," Burlew continued. "But they don't think a lot about water. But in our case, without water, you don't have cooling." After the water main line broke, Burlew began looking for a backup water supply. And eventually he and his data center staff took a page from the book of the old Texas oil men: You drill for it.Digging a hole to get out of one
That's right; Experian dug a well. It hired a contractor to drill 1,200 feet into the ground to hit the Woodbine Aquifer, which runs through 17 counties in north central Texas. Burlew said that in the case of another emergency like the breaking of the water main line, the McKinney data center -- which is a production data center for the company -- can switch to well water indefinitely and automatically.
Drilling a well was costly, at a price tag of about $150,000, according to Burlew. But that pales in comparison to the cost of its data center not being up and running.
"If we lost cooling and operations stopped, we would make up that money in seconds," he said.
To avoid downtime, Burlew and his team have automated the process of switching from city water to the well using controls at the chilled-water plant. Experian has also taken precautions to ensure a graceful cutover by first shutting down the valve to the city water so that, as Burlew put it, "we won't back-feed well water into city water."Permits, valve controls, capacity and filtering
The absence of regulations meant that Experian didn't have to submit a permit to drill the well. "It's an open range to drill as you see fit," Burlew said. Not every part of the country has that luxury, and even that area of Texas might not in the near future. Burlew said the 17 counties served by the Woodbine Aquifer are in negotiations to form a consortium that would regulate access to the aquifer. Luckily, the Experian well has already been dug and, should regulatory changes be made, will be grandfathered in.
But there was concern about whether the well could supply enough water to the facility: 30 gallons of water per minute, according to Burlew. Luckily, Experian just made it; once it drilled the well, it determined that the well could supply 32 gallons of water per minute.
There was also concern about the cleanliness of the well water, and whether it would need extensive filtering.
"There is a fine silt at the bottom of the aquifer that you'll see at initial startup," Burlew said. "That's what we're concerned with. If that goes through the chiller veins, it could scratch up the tubing, causing lost performance at the chillers and be costly."
The drilling company said the silt is common in well water, and that it would clear up after a few months of pumping. Nevertheless, Experian sends water samples to a lab to assess water quality and how it might affect the chillers. Since the mechanics of the well were completed only a couple months ago, it's still early in the process, and Experian will have a chance to install a filtering system if they need it.The unknowns of well drilling and other backup-water options
Robert McFarlane, a data center design expert and a principal at New York-based Shen Milsom & Wilke Inc. , said that well drilling is an uncommon way to get backup water for a data center, because there are so many question marks.
"The problem with well water is that, until you reach it, you don't know what condition it's in, and what kind of minerals are in it," he said. "If it's more like spring water, then it can be great."
In addition, unless you have assurances that there is a water supply somewhere underground, it could be like digging to China: You'll never get there.
"The thing about wells is you can drill, but it doesn't mean you're going to hit something you want," he said. "How many holes do you have to sink before you know you're going to hit what you need?"
Experian's, however, it knew it could tap into the resources of the Woodbine Aquifer.
McFarlane said a more common approach to backup water is to dig a retention pond, which Experian also considered. For those data centers that have the highest demands for availability, those retention ponds usually are large enough to supply water to chiller plants for a week or so. At that point, the company will need to decide whether they'll be able to bring that primary site back on or transfer to a backup location.
"It's all part of the whole disaster recovery plan, and every organization treats it differently," McFarlane said. "The decision is based on how long you think you're going to have to run and how long you think it will take to solve the problem."