How to Kill the Walking Dead Servers
They live among you. Sometimes they’re nearly undetectable for decades until mysteriously resurrected from the grave. These forgotten servers are your (or your predecessor’s) progeny, and they’re hogging precious power from the server room or wiring closet.
Worse, zombie servers increase an organization’s attack surface and profile, and consume valuable compute power, storage and backup resources. These boxes are unlikely to be properly updated, so they have cracks that can be easily pried open. Here’s how to go about finding and retiring rogue or unwanted servers.
Ping Away
Ping network segments and link-list the host. Surprise! You’ll find zombie servers this way. Don’t forget to ping each virtual LAN and logical network segment. Alternatively, you can also identify strange hosts by looking in your DNS tables.
Analyze Packets
One of the most popular zombie hunting tools is the Wireshark protocol analyzer. It requires a desktop or notebook with a network interface card set to promiscuous mode. Add Wireshark to search for IP addresses and then map them to your inventory. This works well for flat networks. If your network is partitioned by virtual LANs, you’ll need to have access to and examine all possible VLAN addresses.
Running Wireshark across the host’s card will pick up a lot of traffic sortable by IPv4 or IPv6 addresses. Link these to known hardware assets, and prepare to be frightened by unknown traffic and hosts. Find, identify and kill them where necessary, and measure power consumption before and after the hunting spree. AOL once saved $10 million in energy costs by decommissioning zombie servers.
Search Hypervisors
The downside to simple IP address link lists is that they won’t identify zombie hosts hidden by network address translation addresses, or hosts on unsearched VLANs at the top level. Worse, hypervisor or container hosts may contain multiple levels of IP addresses with NAT address ranges behind or inside them. Dive deep into Hyper-V, VMware, XenServer and other operating systems to find all of the hosts.
VM and container hosting platforms have different characteristics and may represent singular IPv4/IPv6 address ranges, with perhaps many hosts behind a single address. You’ll need management software and the correct training to understand what hosts are inactive or forgotten.
Citrix XenCenter, VMware vSphere/vCenter and Microsoft Systems Center all need to be accessible to ferret out the zombies on hosted platforms. For container hosts, the core platform must be accessible in lieu of management frameworks that identify zombie infrastructure allocations.
Walk Around
Zombie servers can be alive without responding to messages or pings, yet still draw power from the data center. Many IP addresses can be easily masked or be fully hidden by VLAN configurations, misconfigured routers or just accidentally detached from the network. In order to find hosts that are disconnected from network resources, conduct a physical examination of assets and check off the inventory list.
Every data center and network operations center makes interesting discoveries about lost, misconfigured and disconnected zombie equipment ranging from lowly experimental containers and servers through to discoveries such as Voice over IP servers, high availability stacks and whatever the last IT regime did. It’s time to hunt them and save money.
