How to Flow with Power Outages in the Data Center
Credit: Dan Videtich
Scheduled power outages go much more smoothly now that the Kansas Department of Labor has upgraded its backup batteries, says Danny Kuckelman.
Whether they're caused by tornadoes ripping down power lines or are simply the result of an overtaxed electrical grid, power failures are inevitable. The question is how well organizations are prepared for them. A backup power solution can make all the difference between a minor incident and a major catastrophe.
Last July, the Kansas Department of Labor decided to stop tempting fate and upgrade the uninterruptible power supplies and batteries in its primary call center. Some of the UPS gear was more than 10 years old, says Danny Kuckelman, director of technology services for the agency. Worse, that equipment could provide only 10 minutes of backup power in an emergency.
That meant citizens calling the department during an outage to inquire about unemployment benefits or request records regarding a workers' compensation claim might encounter a dead line. The agency's aging interactive voice response servers could also be fried by a power surge or spike, putting the call center out of business for days until replacement servers could be found.
Sudden outages were a disaster, and even scheduled ones were a bit of a nightmare, says Kuckelman. In the past, when maintenance needed to cut power to the building, he'd have to assign three or four employees to go in, carefully take all the servers offline, and then bring them back up when the power was restored — a process that could take two or three hours.
So last summer, the Kansas Department of Labor spent about $30,000 on two APC Smart-UPS VT 30 kVA units and an extended battery cabinet. With one upgrade, the call center went from a hodgepodge of legacy UPS gear spread across six racks down to one rack and a single enclosure. More important: Backup power increased from roughly 10 minutes to up to three hours, says Kuckelman.
"Three or four weeks after we had the first APC unit installed, our maintenance group needed to take the power down for about half an hour," Kuckelman says. "The utility company came by to disconnect our building, but all our equipment stayed up. I had two staffers onsite in case I needed them, but I didn't. It was fantastic."
Besides providing ample reserve power, Smart UPS equipment also makes it easier for Kuckelman to monitor power loads and battery life, and receive email or text alerts when the power drops. The upgrade went so well, he says, that the department is currently upgrading battery backup power in two more of its data centers.
"It's a great feeling to walk into the data center, look at the front of the UPS, and know your systems are protected. If you experience a power outage, you're not going to lose equipment or drop service levels," says Kuckelman. "It's peace of mind."
Larger Loads
Global increase in data center power consumption between 2011 and 2012
SOURCE: DatacenterDynamics 2012 Global Industry Census (DCD Intelligence, 2012)
One of the biggest problems with power is that data centers need so much more of it than they used to, says Gartner Research Director Nik Simpson. Many just aren't keeping up.
"Ten years ago, a typical rack load would be from 3 to 5 kilowatts," he says. "Now the average is around 8 kW with peaks from 15 to 20 for a rack full of blade servers. That's a lot of equipment to pack into a small space. The older the data center, the more problematic this becomes."
Like the Kansas Department of Labor, Florida's Sumter County Clerk of Courts was trying to run a modern virtualized data center relying on backup power systems that were more than a decade old.
"Our UPSs were not only overdue for an upgrade, they were also running out of capacity," says Brian Berry, director of technical services for the Clerk of Courts in Bushnell, Fla. "We needed a scalable system that could support the new power demands of our virtualized workloads, with enough inherent redundancy to increase the protection of our IT systems."
As a result, Sumter County decided to replace its aging UPS with a 24 kW Eaton BladeUPS rack system and Eaton Advanced Managed enclosure power distribution units. "The Eaton BladeUPS system was ideal for our situation," Berry says. "They're incredibly scalable, so we can expand capacity in the future."
What's more, the ePDUs allow the Clerk of Courts to monitor and control power consumption at the outlet level, which ensures servers receive a constant power supply and the IT department can shut them down safely if necessary.
Going Green
Number of radios the New York City Department of Information Technology and Telecommunications distributed to agencies during Superstorm Sandy
As data centers grow more sophisticated, they're also learning how to use power more efficiently and save taxpayer money in the process. Agencies like California's Department of General Services are helping to lead the way.
As part of a statewide mandate to go green, the Department of General Services is consolidating its Sacramento data center, moving from 36 server racks spread across 2,500 square feet of raised floor to six racks in 600 square feet. By reducing the amount of space it needs to cool and using in-row cooling gear from APC, the department can trim its annual electricity bill by around $70,000, says CIO Jeff Funk.
Despite moving from 60 percent virtualization to 95 percent, Funk estimates the new data center will draw almost 50 percent less power than the old one. Just the same, he is making sure the agency has backup power to spare by installing one 250 kW APC backup unit.
"We didn't have to spend millions of dollars to upgrade our data center. We just looked really hard at how we could gain efficiencies in each area," Funk says. "We have headroom to grow if we need it."
Lessons Learned
Last October, Retirement Systems of Alabama opened a state-of-the-art data center in a historic building in the heart of Montgomery. RSA, which manages $32 billion of assets for state employee and teacher pension funds, wanted to create a showplace for other companies looking to colocate there, says Michael Blevins, a software analyst. Here are some of the organization's power tips:
- You can't be too redundant. The data center features two 1.6 megawatt UPS systems, two emergency generators rated at 2.5 MW apiece, and two sets of automatic transfer switches, making it 100 percent redundant, says Blevins.
- Have a backup plan. The center has 4,000 gallons of diesel fuel on hand — enough to run the generators for nearly 24 hours at 100 percent load — and can be refueled within four hours, says infrastructure engineer Ryan Lambert.
- Don't forget to test. RSA does live tests of its UPS and its generators once a week, says Lambert. "The biggest mistake people make is to assume power will be there when they need it," he says. "Without regular maintenance and testing, you're setting yourself up for problems."
