Streamlined Servers

"Living in hurricane country, as we do, we fully expect to have to use our disaster-recovery plans," says Mark Ramsey, manager of information technology operations for Charlotte County, Fla. Virtualization of blade servers is one way Ramsey ensures that the government's critical applications are available no matter what the weather throws at them.

Charlotte County's experience during Hurricane Charley in 2004 drove the migration. As the storm approached, Ramsey's team loaded 10 to 12 1U servers into a van and carted them from an unhardened data center to a hardened data center: A windowless room in the jail that, being above sea level, should be able to take a Category 5 hurricane.

The IT group quickly brought the servers back online. Unfortunately, the facilities people at the jail had inadvertently thrown the wrong circuit breaker, disabling the cooling system. The room temperature climbed to 120 degrees, and after several hours the servers overheated and stopped working.

After the storm, Ramsey decided it would be a good time to consider virtualization. "We didn't want to ever again have to rush servers by truck from one location to another," he says. Along with strengthening the county's disaster-recovery stance, Ramsey wanted to boost server utilization and reduce costs. As a result, the county replaced 80 servers with eight IBM blades and partitioned them into virtual servers using VMware Infrastructure 3.

Virtualization not only eliminates the need to physically move servers in the event of disaster, but it allows Ramsey to move applications from one server to another without having to touch the physical machines. Because IP storage is a better fit for his network gear and staffers' skills, Ramsey switched from a Fibre Channel storage area network to an Internet Small Computer System
Interface (iSCSI) to reduce costs and protect his shared storage (see sidebar).

It's All Good

Linda Yarchenko, director of Charlotte County's Information Technology Department, sees nothing but benefits from the move to virtualization. "It will save us money while at the same time protecting our data and applications," she says. She was able to pay for the rollout with the money she had budgeted for upgrading the previous servers, some of which were at the end of their lifecycles. Insurance payments from the loss of equipment during Hurricane Charley helped, though she doesn't count those dollars because the server virtualization project kicked off before the insurer calculated and released payment.

Virtualization provides Charlotte County a number of advantages. For one, it offers a uniform presentation of hardware to the operating system. As long as the hypervisor is installed on a physical server, you can install any supported OS on top of it no matter what type of network interface card, processor, memory, drives or other component the physical machine has.

"During Charley, I had to determine which physical servers in the hardened center could support which operating systems. With virtualization, I can decouple my dependence on the type of hardware when considering moving applications from one physical server to another," says Ramsey, a Certified Chief Information Officer.

What's more, virtualization eliminates the need to physically move anything during a disaster. As long as the data is available through shared storage, Ramsey and his team can simply send the virtual server over the network to any other virtual machine. Generally, the move will be between the hardened and unhardened data center, but it could also be to a disaster-recovery center in another geographical area if Charlotte County decides to use this approach.

Ramsey says the VMware VMotion tool makes it easy to move virtual machines. "All we'd have to do is use VMotion to move the server from one place to another, and we're back in business with almost no disruption of service," Ramsey says. If the underlying physical server is near capacity and has to take on extra work, the virtual machine might run in a degraded state. But, says Ramsey, "It's better to have limited performance than not have access at all."

Brian English, programmer and network administrator with Charlotte County, says virtualization allows him to carve out pieces of physical servers to use for testing and development. "We can create whatever environment we want without affecting production," he says.

Virtualization has also allowed the county to reduce its server farm more than tenfold. Like most organizations, Charlotte County used to buy a new server when it added a new application. The problem was the average server didn't see more than 3 percent to 5 percent utilization, says Ramsey. Now, adding capacity is a simple matter of moving one or more virtual servers to another machine without even taking down the application.

Server Specs

Ramsey was able to collapse his server farm of 80 1U servers to eight newly purchased IBM LS41 blade servers, which run about 122 virtual machines. The servers sport quad dual-core AMD Opteron processors, 32 gigabytes of RAM and four network interface cards each. Half are housed in the hardened data center and the remainder are located in the nonhardened center. In general, the hardened center holds more of the enterprise applications, while the nonhardened center holds more departmental applications.

If you put all your eggs into six baskets and one of those baskets breaks, you lose a lot of eggs. But Ramsey was able to actually improve availability. If any of the host machines goes down -- for example, because of a failure of its power supply or because of a malfunctioning processor -- VMware High Availability monitors and detects failures of the physical server, the virtual machines and the OS installed on the virtual machines. It automatically moves the affected VMware instances to the surviving physical servers and restarts virtual machines as needed. VMware DRS automatically examines the loads on the physical machines and distributes the virtual machines appropriately to achieve optimal performance.

Seeking more security, the county recently built a state-of-the-art public safety facility with concrete reinforced walls, situated 20 feet above sea level and outfitted with redundant
air conditioning.

But even the best solutions sometimes bring unexpected problems. A few months ago, Ramsey walked into the new hardened server room and was shocked to find a thin layer of dust that had settled on all the equipment. As it turned out, the air-handling equipment had a defective pulley that was grinding the belts into dust.

In the past, a problem like this might have required Ramsey to carry some servers back to the nonhardened data center. This time he was able to move the virtual servers over the network to the nonhardened center, giving facility-management people the time to solve the problem. When the equipment was fixed, Ramsey was able to quickly and transparently move the servers back to the hardened site.

If necessity is the mother of invention, then Hurricane Charley helped Charlotte County find a better way to protect its data center. But the solution went well beyond just providing high availability.