Mark Ramsey, manager of IT operations for Charlotte County, maintained system availability amid air conditioning repairs by seamlessly moving virtual servers to another site.

Streamlined Servers

Charlotte County, Fla., rolled out server virtualization to avoid a repeat of outages it suffered during Hurricane Charley.

Charlotte County, Fla., rolled out server virtualization to avoid a repeat of outages it suffered during Hurricane Charley.

"Living in hurricane country, as we do, we fully expect to have to use our disaster-recovery plans," says Mark Ramsey, manager of information technology operations for Charlotte County, Fla. Virtualization of blade servers is one way Ramsey ensures that the government's critical applications are available no matter what the weather throws at them.

Charlotte County's experience during Hurricane Charley in 2004 drove the migration. As the storm approached, Ramsey's team loaded 10 to 12 1U servers into a van and carted them from an unhardened data center to a hardened data center: A windowless room in the jail that, being above sea level, should be able to take a Category 5 hurricane.

The IT group quickly brought the servers back online. Unfortunately, the facilities people at the jail had inadvertently thrown the wrong circuit breaker, disabling the cooling system. The room temperature climbed to 120 degrees, and after several hours the servers overheated and stopped working.

After the storm, Ramsey decided it would be a good time to consider virtualization. "We didn't want to ever again have to rush servers by truck from one location to another," he says. Along with strengthening the county's disaster-recovery stance, Ramsey wanted to boost server utilization and reduce costs. As a result, the county replaced 80 servers with eight IBM blades and partitioned them into virtual servers using VMware Infrastructure 3.

Virtualization not only eliminates the need to physically move servers in the event of disaster, but it allows Ramsey to move applications from one server to another without having to touch the physical machines. Because IP storage is a better fit for his network gear and staffers' skills, Ramsey switched from a Fibre Channel storage area network to an Internet Small Computer System
Interface (iSCSI) to reduce costs and protect his shared storage (see sidebar).

It's All Good

Linda Yarchenko, director of Charlotte County's Information Technology Department, sees nothing but benefits from the move to virtualization. "It will save us money while at the same time protecting our data and applications," she says. She was able to pay for the rollout with the money she had budgeted for upgrading the previous servers, some of which were at the end of their lifecycles. Insurance payments from the loss of equipment during Hurricane Charley helped, though she doesn't count those dollars because the server virtualization project kicked off before the insurer calculated and released payment.

Virtualization provides Charlotte County a number of advantages. For one, it offers a uniform presentation of hardware to the operating system. As long as the hypervisor is installed on a physical server, you can install any supported OS on top of it no matter what type of network interface card, processor, memory, drives or other component the physical machine has.

Pain Points
The Charlotte County IT department solved a number of challenges by deploying server virtualization and an Internet Small Computer System Interface (iSCSI) SAN.

  • Aging equipment: The Fibre Channel SAN gear was at the end of its life and the servers had been in operation for more than four years.
  • Increased server failure and recovery times: Hardware failures were on the rise and accounted for more than 1,000 staff hours of lost productivity in 2006. Average time to recovery was four hours.
  • Growing storage demands: Shared storage would have to be augmented to meet new demands from business units. IT estimated the county would need between two and three terabytes of additional storage.
  • Limited electrical capacity: Further server farm growth would force the county to expand its existing electrical, UPS and generator capacity.
  • Continuity of operations: IT needed new processes to ensure the survivability and increased availability of the county's data assets.
  • Underutilized resources: In most cases, processor, memory, disk and network resources were being utilized at only 5 percent to 10 percent of total potential. Storage utilization approached only 70 percent.

"During Charley, I had to determine which physical servers in the hardened center could support which operating systems. With virtualization, I can decouple my dependence on the type of hardware when considering moving applications from one physical server to another," says Ramsey, a Certified Chief Information Officer.

What's more, virtualization eliminates the need to physically move anything during a disaster. As long as the data is available through shared storage, Ramsey and his team can simply send the virtual server over the network to any other virtual machine. Generally, the move will be between the hardened and unhardened data center, but it could also be to a disaster-recovery center in another geographical area if Charlotte County decides to use this approach.

Ramsey says the VMware VMotion tool makes it easy to move virtual machines. "All we'd have to do is use VMotion to move the server from one place to another, and we're back in business with almost no disruption of service," Ramsey says. If the underlying physical server is near capacity and has to take on extra work, the virtual machine might run in a degraded state. But, says Ramsey, "It's better to have limited performance than not have access at all."

Brian English, programmer and network administrator with Charlotte County, says virtualization allows him to carve out pieces of physical servers to use for testing and development. "We can create whatever environment we want without affecting production," he says.

Virtualization has also allowed the county to reduce its server farm more than tenfold. Like most organizations, Charlotte County used to buy a new server when it added a new application. The problem was the average server didn't see more than 3 percent to 5 percent utilization, says Ramsey. Now, adding capacity is a simple matter of moving one or more virtual servers to another machine without even taking down the application.

Server Specs

Ramsey was able to collapse his server farm of 80 1U servers to eight newly purchased IBM LS41 blade servers, which run about 122 virtual machines. The servers sport quad dual-core AMD Opteron processors, 32 gigabytes of RAM and four network interface cards each. Half are housed in the hardened data center and the remainder are located in the nonhardened center. In general, the hardened center holds more of the enterprise applications, while the nonhardened center holds more departmental applications.

If you put all your eggs into six baskets and one of those baskets breaks, you lose a lot of eggs. But Ramsey was able to actually improve availability. If any of the host machines goes down -- for example, because of a failure of its power supply or because of a malfunctioning processor -- VMware High Availability monitors and detects failures of the physical server, the virtual machines and the OS installed on the virtual machines. It automatically moves the affected VMware instances to the surviving physical servers and restarts virtual machines as needed. VMware DRS automatically examines the loads on the physical machines and distributes the virtual machines appropriately to achieve optimal performance.

Seeking more security, the county recently built a state-of-the-art public safety facility with concrete reinforced walls, situated 20 feet above sea level and outfitted with redundant
air conditioning.

But even the best solutions sometimes bring unexpected problems. A few months ago, Ramsey walked into the new hardened server room and was shocked to find a thin layer of dust that had settled on all the equipment. As it turned out, the air-handling equipment had a defective pulley that was grinding the belts into dust.

In the past, a problem like this might have required Ramsey to carry some servers back to the nonhardened data center. This time he was able to move the virtual servers over the network to the nonhardened center, giving facility-management people the time to solve the problem. When the equipment was fixed, Ramsey was able to quickly and transparently move the servers back to the hardened site.

If necessity is the mother of invention, then Hurricane Charley helped Charlotte County find a better way to protect its data center. But the solution went well beyond just providing high availability.

SCSI Implementation

One of the core requirements of Charlotte County's server virtualization project was shared storage. The county's legacy shared storage was nearing the end of its lifecycle.

Manager of IT Operations Mark Ramsey used this opportunity to move from a Fibre Channel storage area network to an Internet Small Computer System Interface (iSCSI) SAN from LeftHand Networks. This allowed him to repurpose his expensive Fibre Channel gear for other network jobs and tap the extensive IP network he had in place. He didn't want a SAN that required proprietary hardware, which would tie him to a single vendor's products. Another benefit of the move is he can tap his staff's IP skills.

LeftHand Networks SAN/iQ software provides storage clustering, which allows Ramsey to create separate virtual pools of storage. Ramsey says one of the key strengths of the SAN/iQ software is the ability to span his critical data across his two data centers.

Do the Math

Deploying virtual machines rather than separate physical machines was a no-brainer for Charlotte County.

-> Physical environment $673,562

122 IBM x3650 @ $5,000 = $610,000
122 MS Windows Server 2008 standard licenses @ $521 = $63,562

-> Virtual environment $292,128

8 IBM LS41 blade servers @ $19,000 = $152,000
16 VMware VI3 enterprise licenses @ $5,000 = $80,000
32 Microsoft data center licenses @ $1,879 per physical processor = $60,128

-> Manpower savings $48,394

Building 122 physical servers would have taken six hours each, conservatively costing the county
$51,240 in staff time. Using VMware's template and cloning features, IT was able to build a new server
in 20 minutes, which works out to about $2,846 in manpower.

Alex McNight
Jul 07 2008