Three summers ago, a 76-inch main line that feeds water from Lake Lavon to 29 cities and towns in Texas broke....
That was a disaster for credit services giant Experian, which relied on city water to supply the chillers that cool its McKinney data center.
"We lost all city utility as far as water pressure," said Russ Burlew, the data center facility manager. "It took three and a half days to repair that line, and we were without water. We had vendors come in with water trucks and pumps to keep our tower full."
"When everyone talks about Tier IV [data center], they talk about power and cooling redundancy," Burlew continued. "But they don't think a lot about water. But in our case, without water, you don't have cooling." After the water main line broke, Burlew began looking for a backup water supply. And eventually he and his data center staff took a page from the book of the old Texas oil men: You drill for it.Digging a hole to get out of one
Drilling a well was costly, at a price tag of about $150,000, according to Burlew. But that pales in comparison to the cost of its data center not being up and running.
"If we lost cooling and operations stopped, we would make up that money in seconds," he said.
To avoid downtime, Burlew and his team have automated the process of switching from city water to the well using controls at the chilled-water plant. Experian has also taken precautions to ensure a graceful cutover by first shutting down the valve to the city water so that, as Burlew put it, "we won't back-feed well water into city water."Permits, valve controls, capacity and filtering
But there was concern about whether the well could supply enough water to the facility: 30 gallons of water per minute, according to Burlew. Luckily, Experian just made it; once it drilled the well, it determined that the well could supply 32 gallons of water per minute.
There was also concern about the cleanliness of the well water, and whether it would need extensive filtering.
"There is a fine silt at the bottom of the aquifer that you'll see at initial startup," Burlew said. "That's what we're concerned with. If that goes through the chiller veins, it could scratch up the tubing, causing lost performance at the chillers and be costly."
The drilling company said the silt is common in well water, and that it would clear up after a few months of pumping. Nevertheless, Experian sends water samples to a lab to assess water quality and how it might affect the chillers. Since the mechanics of the well were completed only a couple months ago, it's still early in the process, and Experian will have a chance to install a filtering system if they need it.The unknowns of well drilling and other backup-water options
Shen Milsom & Wilke Inc.
"The problem with well water is that, until you reach it, you don't know what condition it's in, and what kind of minerals are in it," he said. "If it's more like spring water, then it can be great."
In addition, unless you have assurances that there is a water supply somewhere underground, it could be like digging to China: You'll never get there.
"The thing about wells is you can drill, but it doesn't mean you're going to hit something you want," he said. "How many holes do you have to sink before you know you're going to hit what you need?"
Experian's, however, it knew it could tap into the resources of the Woodbine Aquifer.
McFarlane said a more common approach to backup water is to dig a retention pond, which Experian also considered. For those data centers that have the highest demands for availability, those retention ponds usually are large enough to supply water to chiller plants for a week or so. At that point, the company will need to decide whether they'll be able to bring that primary site back on or transfer to a backup location.
"It's all part of the whole disaster recovery plan, and every organization treats it differently," McFarlane said. "The decision is based on how long you think you're going to have to run and how long you think it will take to solve the problem."
Let us know what you think about the story; email Mark Fontecchio, News Writer. You can also check out our Data Center Facilities Pro blog.