Power Struggle

When you get to the point that you can't power up new servers or cool down the existing ones you have, it's time to move your data center, according to Bruce Bounds, director of IT for the city of Monroe, N.C. "We knew we had definitely outgrown the ‘large closet' in which we had our servers and other infrastructure when, as soon as we plugged in our new virtualized servers, the room's backup UPSes started alarming," Bounds says.

City Systems Administrator Josh Hyatt was eager to design a new data center and immediately sketched out the energy requirements for the room's 32 physical servers as well as the heat they were likely to generate. Rather than prioritizing certain servers and racks for backup, Hyatt says he and Bounds drafted power and cooling specifications with the assumption that all the servers were mission critical. After all, they house applications for the city's electric, garbage and sewer billing and management, GIS systems, aquatic center operations, and the airport.

It became apparent early on that the new data center, which is housed in a different area of the same building, would require a doubling in the amps delivered to the room, to 400, and a doubling in air conditioning to 16 tons.

"Our goal was total redundancy," Hyatt says. However, the city faced one large obstacle in achieving that goal, in that it supplies its own power. Therefore, it would be impossible to shield against a power outage in the typical way of bringing in feeds from different substations. Instead, the IT team equipped each rack with battery-powered 208-volt APC Symmetra UPSes. The combination of 12 KVA and 24 KVA backup power supplies ensures that the city's staff and 37,000 residents can sustain an outage without an interruption in services.

Rack 'Em Up

In addition to the backup battery packs, Hyatt focused on ensuring that each rack is properly powered and cooled. He installed APC metered rack power distribution units (PDUs) to monitor usage on each rack. "The PDU alerts us when a rack is running hot so we can balance out the equipment before we suffer a failure," Hyatt says. For instance, when the city deployed new mobile surveillance DVRs, one rack jumped two amps, so they had to immediately lessen its load.

Hyatt and Bounds also watch the air conditioning patterns closely. They installed a two-stage wall-mount air conditioner so that if one unit goes down, the other kicks in. Temperature is monitored via a web interface that displays room humidity, temperature and overall heating and cooling trends. "We're very focused on airflow and power capacity across the whole room so there are no hot spots and no chance that if one system goes down, it will take others with it," Hyatt says.

Nik Simpson, research director for Gartner's Data Center Strategies  group, says that as state and local governments move from distributed server closets to consolidated data centers, they have to work harder to protect power and cooling. "You'll probably have higher rack densities in the consolidated data center. Where you previously only needed 3kW of power for a rack, you'll now see 10kW or more, and you need to be able to support that," he says.

Simpson also advises IT organizations to group high-density racks together rather than mixing them with racks that have less demanding power and cooling requirements. He says that solving the power and cooling problem for one concentrated group of systems will be easier than buying high-tech cooling such as refrigerated rack doors to deal with hot spots scattered around the data center.  

In Hubbard County, Minn., IT management and support consultant Tom Hankins follows this advice. While every piece of electronic equipment that might be susceptible to the voltage variations that happen during the county's frequent brownouts is equipped with a backup power supply, the capacity of that power supply is contingent upon the critical nature of that application.

He uses a mix of UPSes from Tripp Lite and APC. "For instance, the GIS team has a UPS with a long enough lifecycle that they can save their programs and not lose work -- about 10 to 15 minutes," Hankins says.

Meanwhile, the county's 5-year-old data center, which houses servers that support law enforcement and other mission-critical services, has multiple UPSes, multiple circuits and multiple feeds from different substations to avoid a single point of failure. It also has a large diesel generator that automatically comes online in case of a total power loss.

With aging power infrastructure throughout the city, Hankins said it was imperative to build redundancy throughout the enterprise. "The redundancy we've set up across the county is so transparent that in most cases, when there is a power outage or drop in voltage, users aren't affected," he says.