How Counties Avert Outages with UPS Systems

This equipment proves key to maintaining uptime in the data center.

Uninterruptible power supplies may not be sexy, but they keep computers and networks of all kinds running without a hiccup during a power failure. In data centers and emergency operations centers, where the cost of downtime ranges from highly expensive to life threatening, UPSs are nothing short of critical.

But when the data center goes dark, the UPS often is the chief culprit, according to a Ponemon Institute study released last fall. Survey respondents frequently attributed the cause of downtime to UPS failure or a UPS exceeding capacity. That’s why UPSs and the legion of batteries that power them rank as a key consideration in rolling out a new data center.

In the Eye of the Storm

Ray Wilson, fire marshal for Hood County, Texas, had been in his newly appointed role as emergency manager for just two weeks last May when he found himself staring down the eye of a violent tornado.

“Our area wasn’t scheduled as being high risk, but tornados started popping up,” says Wilson. Concern mounted as a storm spotter sent word that he had seen a tornado forming to the northwest of Granbury, the county seat. “The first emergency call that we had for that area was for a laceration, nothing more than that. But when the ambulance responded to the scene, it was total devastation,” recalls Wilson. Houses were swept clear down to the slab. Several mobile homes were taken off their mountings and had rolled or blown away. By the time it was over, six people were dead.

Wilson-Solomon

A $500,000 investment in power gear helps Hood County, Texas, maintain continuity of operations, say Ray Wilson and Jackie Solomon.

Besides being situated in a tornado zone, the Granbury emergency operations center was delicately positioned due to a high-profile neighbor: a nuclear power plant. “We keep constantly refining the scenarios of what we would do in different types of emergencies,” says Jackie Solomon, IT director for Hood County. “The EOC was beginning to age. The room was built to hold 15 or so people. With an average drill we might have 30 people,” says Solomon.

“During the tornado, we outgrew the room and had to take over the sheriff’s training rooms for days. They were not set up for those types of operations,” says Wilson. The good news: During the event, the overstretched EOC did not lose power. Even better: When the tornado hit, Hood County was already planning to build a new EOC and data center with state-of-the-art equipment, including UPSs.

Ironically, the plans for the new building and equipment had to be finalized just a few weeks after the storm. Solomon helped spec out the requirements for the UPSs, selecting Schneider Electric’s APC Symmetra PX 40kW models. He made sure the EOC and data center would each have their own dedicated gear.

While cost was an important consideration, it was not the top priority, notes Solomon. “We needed to think about what was going to be best for the whole county,” he says. Hood County was able to move a Schneider Electric 22 kVA UPS that was already in use, saving taxpayer money. The total bill for the cabinets, electrical work, emergency power, battery backups and installation was just over $500,000.

“That was pretty reasonable,” says Solomon, considering the importance of having continuous power to the data center and EOC.

New, Improved Batteries

power supply market

SOURCE: “Global Uninterruptible Power Supply Market 2011–2015” (TechNavio, December 2012)

The configuration of the UPS plays a role in its reliability, according to Kaushik Venkateshwaran, lead analyst for the energy, engineering and automation division of TechNavio, a research and advisory firm. The safest approach is “n+1,” he recommends. That means that for however many units the space requires, the organization implements an additional one for failover.

Sarasota County, Fla., didn’t stop with n+1. IT Director Ken Watson says the county aimed for many layers of redundancy when building its new data center in 2010, achieving a configuration of a little more than n+2. “We designed two of everything but we didn’t stop there. If our power consumption grows and we need to run full time on two units, we have built in enough room to add a third UPS,” says Watson. “It could happen at some point.”

The county spent $1 million to $2 million for Emerson/Avocent UPSs. Watson appreciates the improved battery technology. A monitoring system sends an alert if the battery is on its last legs. Previously, he had to wait for quarterly maintenance to tell him if a new battery was needed. “We were burned by that in the past. A couple of bad batteries kept us from transferring from street power to generator power. This takes away all the guesswork,” he says.

Price is always a consideration when buying equipment, but not always the trump card. For Fidel Deforte, infrastructure and communications technology manager for the city of Cape Coral, Fla., ease of management ranks high. Many UPS vendors require their own engineer to come onsite to change batteries or perform repairs. By contrast, the highly modular design of APC’s UPSs makes it literally a snap to change a battery, says Deforte.

“They have a modular chassis, so the batteries and the modules that make up a cabinet are all able to be replaced by the user. Pull it out, put a new one in,” Deforte says.

Most important, this doesn’t void the warranty. Deforte says his city was able to squeeze more life out of its batteries after the local real estate market tanked in 2008. “The batteries were supposed to be replaced anywhere from three to five years out, but we went seven years,” he says. “We had experienced five years of budget cuts and complete removal of capital. We were forced to push our refresh out.”

Ultimately, Cape Coral came through the downturn just fine — with no unplanned downtime.

Purchase Criteria

Buying an uninterruptible power supply can be tricky. TechNavio Lead Analyst Kaushik Venkateshwaran identifies the most important features to look for:

Reliability and availability: Because downtime brings serious repercussions in the form of monetary, data and productivity losses, reliability is critical. The estimated average costs incurred during downtime are in excess of $4,000 per minute.

Accuracy and efficiency: The higher the efficiency, the lower the power consumption will be, ensuring longer lifetime and reduced operating costs.

Scalability: Expansion of infrastructure directly impacts the capacity of the existing UPS system.

Other attributes: Compliance with regulations, user friendliness and around-the-clock technical support are other important capabilities.

Lesson Learned

Fidel Deforte has experienced firsthand just about the worst thing that can happen to a data center manager on the job: total silence. Several years ago, a local utility in Cape Coral, Fla., was doing work when street power cut off.

“If the UPS notices there is no power coming from the street, it will switch the load to the generator,” says Deforte, infrastructure and communications technology manager for the city of Cape Coral. But before the switch is complete, the batteries have to carry the whole data center for up to 10 seconds. He remembers well the day when that didn’t happen and the city’s old data center went down for a few hours.

“The worst thing is going into a data center when it’s completely quiet. It is supposed to be humming with all the power sounds,” says Deforte. “That was one of the worst feelings I have ever had.”

Now, he says, “we constantly test everything. We go through the gamut of losing power and transferring back once a month to make sure everything is working.” That diligence has paid off — it’s been more than three years without a hiccup.

Trevor Paulhus
Apr 04 2014