Downsizing the Data Center

The city of Chesapeake, Va., was on the verge of making a costly technology mistake until Peter Wallace arrived.

In late 2007, the city's new CIO met with department heads and his IT staff and discovered that his first task was to combat the age-old problem of server sprawl. The data center, filled with 135 servers, was out of space, and its cooling equipment was overtaxed, threatening the IT department's ability to support new applications. Before Wallace's arrival, city leaders were considering expanding the data center by adding a second floor. They had also earmarked $350,000 to replace four-year-old servers whose hard drives were utilized only 15 percent to 35 percent.

"I told everyone we had to do something better, and that the budget to replace servers gave us that opportunity," Wallace recalls. "It was obvious we needed to leverage our servers more."

Wallace solved the problem with server and storage consolidation. With the $350,000, he purchased networked storage to consolidate the city's data. He then bought new blade servers and virtualization software and sent engineers for server virtualization training.

Today, Chesapeake has reduced its server count to 40, and by the project's end, the city will cut it to 20. Server virtualization saves the city $203,000 in hardware costs per year and $2,000 on its monthly energy bill.

Faced with budget constraints, many state and local governments like Chesapeake are embracing data center consolidation to operate more efficiently and cut costs. Server virtualization software such as VMware's vSphere, Microsoft's Hyper-V and Citrix's XenServer lets IT departments maximize the resources of each server and easily deploy new servers when needed, while networked storage helps agencies better utilize storage capacity and more easily manage and back up data.

Combined, the technology consolidates the amount of hardware used, which saves on data center space and reduces power consumption. The result is cost savings and an easier-to-manage data center. "If done correctly, you reduce infrastructure and management costs, improve server utilization and have faster, more efficient disaster recovery," says analyst Laura Didio, of Information Technology Intelligence Corp.

Improved Disaster Recovery

Two years ago, administrators at Livingston County in Michigan tapped server virtualization and consolidation because it was the best choice available to them.

At the time, county departments wanted their applications to run on their own standalone servers, says Network Manager Greg Jolliff. But the data center had reached its maximum power capacity. To add more servers, the county would need to add a new electrical circuit, which could cost as much as $40,000.

The IT staff sold county administrators and the board on virtualization. "We told them we could consolidate a lot of those servers and offer them redundancy," Jolliff says. Because multiple applications would run on each server as virtual machines (VMs), departments would lose their own separate servers, but in return, reliability and uptime would be improved.

Using VMware ESX, the county consolidated 70 servers to seven physical servers with each server running between 10 to 15 VMs. IT purchased HP DL380 rack servers with about 40 gigabytes of memory.

At the time of implementation, VMware engineers told the county that any application could be virtualized. But other vendors were more cautious, saying some of their applications hadn't been tested on VMs yet.

Since then, VMware's virtualization software has proved itself, Jolliff says. Today, Livingston County has migrated nearly all its applications to VMs, including high-transactional applications, such as Microsoft Exchange and SQL Server databases.

"Back then, we were a little leery because it was a question of whether you can virtualize big SQL databases, but it works fine," Jolliff says. "People now have more trust in it."

To improve continuity of operations, Livingston County houses four servers in the main data center and three servers in a secondary data center several miles away. If one server goes down, VMware's management software will immediately move the VMs to the six remaining servers to prevent downtime. And if all four servers in the main data center go out, the other servers at the secondary site will take over, Jolliff says.

The county's storage area network was antiquated, so for the virtualization project, IT bought a new EMC SAN for each site. For increased redundancy, the main SAN replicates its data to the secondary SAN, he says.

Easier IT Management

Server virtualization simplifies the deployment, maintenance and management of servers, IT administrators say. If Jolliff needs to bring down a server to perform maintenance, such as adding more memory, he simply puts the server in maintenance mode or uses VMware's VMotion software to drag and drop the VMs over to another server, and once he's done, he moves them back with no downtime.

That's a huge benefit for Alan Lantz, system administrator for the city of Rogers, Ark., who used to come into the office on weekends to shut down servers and perform maintenance. Lantz deployed virtualization using Citrix XenServer a year ago and today has 80 VMs running on six blade servers.

"I can migrate the VMs back and forth between servers during the week, and users are not aware that I'm doing my upgrades," he says.

Scott County in Minnesota, which started with 40 physical servers and reduced the number to four production servers running 52 VMs, has saved half a full-time position because setting up virtual servers is so much faster than setting up a new physical server from scratch, says Infrastructure Manager Todd Croy.

Setting up a VM from a golden image takes about an hour, while ordering a new physical server, waiting for it to arrive and setting it up in the rack was a two- to 10-day process, Croy says.

Deployment Strategies

In Chesapeake, Wallace initially considered hiring consultants to implement virtualization, but decided the $150,000 to $250,000 cost was too high. Instead, he sent two network specialists to a six-month virtualization training course for $30,000.

When the pair completed their training, they began virtualizing small applications such as help-desk ticketing. When their initial forays were successful, the IT team developed a game plan to migrate the city's enterprisewide software, from financial to public safety applications. They would migrate an application over the weekend, monitor and troubleshoot it for several weeks, and if it was problem-free, they'd move on to the next set of applications.

"After we saw the early successes, we got brave and didn't tell the users we were cutting them over," Wallace says. "We did it at night, and they'd come in the next morning and had no clue their applications were now working on virtual machines."

For the project, Chesapeake purchased VMware vSphere, HP blade servers and a 96-terabyte EMC Celerra NS40 storage device, which allowed the city to centralize storage and improve storage utilization.

IT staff typically run eight virtual servers on one physical server. About 60 VMs are currently running on nine blade servers with another 10 VMs in sleep mode. If certain applications see a spike in usage, VMs in sleep mode will automatically spring into action and handle extra processing requirements so users don't see a degradation in performance, Wallace says.

The SAN, combined with virtualization and Vizioncore's vRanger backup software, is also speeding the backup process. Rather than having a tape drive connected to each rack server, the city now uses faster disk storage to back up data. As a result, backups that used to take an entire weekend now take just eight hours, Wallace says.

The IT staff still has a few applications left to migrate. The city plans to replace Novell GroupWise with Microsoft Exchange and will install it on a VM this year.

Some applications, such as fax servers, must remain on their own physical servers, so the city will continue to have some servers running standalone applications.

Thanks to virtualization and consolidation, Chesapeake and Livingston County have solved their data center space and power problems. In fact, Wallace plans to turn part of his data center into new office space. "We have so much space that we can build a NOC and have people proactively manage the city's network from one central point," Wallace says.