Securing Virtual Servers

Robert Maley, CISO for Pennsylvania, is a fan of server virtualization. The technology helped him increase online services without expanding the data center footprint.

But as enthusiastic as Maley is about the virtues of virtualization, he’s equally cautious because of potential threats to the physical server, the virtual machines and the hypervisor (the abstraction layer that sits between the physical server and the virtual machines). “If you have a single physical server hosting 20 virtual servers and that physical box becomes compromised, all 20 of your boxes are in danger. You have to be careful,” he says.

Though there have been no published attacks on virtual environments to date, as server virtualization deployments soar, government IT managers expect the threat vector to rise. “There has been a great deal of speculation and concern regarding the possibility of hypervisor malware and hypervisor weaknesses,” says John Matelski, CIO and director of IT services for Gwinnett County, Ga., in Lawrenceville.

For instance, researchers have described one possibility, called “Blue Pill,” in which a piece of malware could be hidden in the hypervisor and used to overtake the host operating system to gain control of its attached virtual machines. There is also fear of a “virtual machine escape,” whereby a virtual machine breaks out of its isolation and maliciously interacts with the hypervisor.

Yankee Group Senior Analyst Phil Hochmuth says the best way to combat these threats is to bring security to the forefront of any virtualization plan. “Security has been an afterthought because everyone’s been so caught up in virtualization’s coolness and cost savings. It has to be a primary factor in virtualization deployment plans,” he says.

Here are five tips to make your virtual environment more secure.

1 Deploy asset-tracking and configuration- management tools.

While virtual machines are much easier to deploy than physical servers, Patrick Hale, deputy director of infrastructure services at the Michigan Department of Information Technology in Lansing, says this can also be a curse. “Virtual machines can pop up like dandelions. By the time you actually notice them, it’s too late. You have to get and keep an accurate inventory of your environment,” he says.

As the DIT has consolidated the state’s 29 data centers down to three, they’ve relied on virtualization to migrate data. Hale says the department has avoided virtual server sprawl by using a combination of auto-discovery tools, agent-based query tools and a standard server build to capture and maintain an inventory of guest operating systems in the virtual environment. The team also keeps a close watch over virtual machine subnet address assignments so that they can easily detect rogue VMs.

“Virtual machine sprawl is not as harmless as it might sound because each virtual machine requires its own operating system and application licenses,” Matelski adds. “If you start spawning machines excessively, you can eat away at your hardware cost savings.”

Ted Ritter, an analyst at Nemertes Research, says virtual machine sprawl also conflicts with compliance mandates that require control over data. “With some compliance rules, you have to be able to show where your data physically lives and you have to demonstrate separation of duties. Virtualization makes that difficult,” he says. Asset-tracking and configuration-management tools ensure that applications that are supposed to be separate don’t end up on the same machine as the virtual environment grows.

2 Delineate trusted and untrusted environments.

Placing virtual machines in untrusted locations such as “demilitarized zones” can also be tricky, according to Hale. “We’ve put a lot of time and money into achieving Payment Card Industry compliance in our DMZ. These requirements get stricter every day, so I’m not in any hurry to introduce complexity into an environment that will in all likelihood require even deeper scanning of the database and application layers.”

Matelski says most problems arise from the loss of tight control over separation of duties that occurs in a DMZ. “There is an opportunity to introduce vulnerabilities through misconfiguration,” he says. For instance, someone could accidentally place the virtual network interface card of a virtual machine in the wrong trust zone, breaching the isolation between trusted and untrusted networks. This is also where a virtual machine escape could bring down your network.

To guard against this, Matelski maps out which virtual servers will reside on which physical hosts and establishes the level of trust for each of those server resource clusters so he knows which can safely reside in the DMZ. He then performs regular security audits on the virtualized resource clusters within the DMZ.

3 Segment your virtual traffic.

Experts say it’s critical to isolate management of the virtual environment to protect your network. “We have created separate networks to manage the virtualized environment and restrict access to only those who manage the servers,” says David O’Berry, director of information technology systems and services for the South Carolina Department of Probation, Parole and Pardon Services in Columbia.

Neil MacDonald, vice president and Gartner Fellow at Gartner, agrees that isolation is necessary. “In the event of a denial-of-service attack, if your management and security control plane are separate, you’ll still be able to communicate with your box,” he says.

4 Monitor traffic between virtual machines.

In physical networks, security tools can monitor traffic throughout the enterprise. Not so with virtual networks. Data can travel between virtual machines on the same box, never hitting the network. This invisibility causes serious problems for IT teams trying to detect security threats, according to Matelski.

“Because attacks can spread easily between virtual machines, IT teams need to scan traffic going in, out and between virtual machines for vulnerabilities and malicious code. They also have to make sure that the virtual environments are scrubbed or secured as they are reallocated,” he says.

Matelski uses an intelligent switch architecture that plays traffic cop to data sent between virtual machines and sends it to the appropriate security applications. He says this design also allows for patch management, intrusion prevention and network access control between virtual machines.

5 Automate patching and updates.

As your virtual environment continues to grow, so does the challenge of keeping your virtual servers current with patches and updates. Pennsylvania’s Maley says it can quickly become impossible for an IT team to manually patch and update servers.

Machines that have been left offline can also pose a threat to network security if they’re brought back up without being checked for proper security status, notes Gartner’s MacDonald.

O’Berry says automated patching tools keep down the TCO of his virtual environment. “You have to find ways to be more diligent without expending so much time watching things as to make it not worth the effort. One exploit on a single box can jump you right into a tragic mess quicker than if things were still separated,” he says.