Patch Management Primer

Runing software patches to fix bugs and improve performance is a constant in the data center. Each week manufacturers offer patches and service packs to make systems more secure. This works, of course, only if patches are applied in a timely, consistent fashion. Here are five tips for guiding your IT staff through patch management.

Tip 1: Understand that hackers are lurking.

Patching is a race between the IT department and the bad guys. The starting gate — if luck prevails — is the announcement of the patch. But too often, hackers know about a vulnerability weeks or months in advance of the patch and may be working on an exploit while IT staff is still largely unaware.

The saddest refrain in stories of security breaches is: “Oh, yeah, the manufacturer released a patch for that a while back, but the organization hasn’t installed it yet.”

Don’t be that victim. Patch as soon as possible.

Tip 2: Automate, don’t try to patch by hand.

Theoretically an IT staff can do patch management manually. In practice, that’s too time-consuming, complex and error-prone for anything other than a small operation.

There are a lot of different choices in automated patch management. All of the good ones will track patches as they are released by multiple manufacturers and apply them consistently to the organization’s systems and networks. The key is to pick a patch management program that is appropriate for the organization — and use it regularly.

Tip 3: Run tests on an offline network.

Remember that not all patches are perfect as released. Some of them will work smoothly and unobtrusively, and some will have their own flaws. Those patches may cause problems with the organization’s IT environment, breaking scripts, causing applications to fail or even crashing the system.

Test new patches on a guinea pig system before going live in a production environment. This is obviously a question of balance: making sure the new patches won’t cause the organization problems, but installing them as quickly as possible. Once you’re certain that the patches won’t damage anything, get them into place pronto.

Tip 4: Be prepared to roll back patches.

What if the patch interferes with the organization’s systems? It happens more frequently than most of us like to admit. In spite of testing, sometimes a flawed patch gets through, and soon afterward the help desk is flooded with user complaints.

When this happens, systems must be rolled back to their previous, unpatched state while the IT staff tries to fix the problem. Have a standardized method for doing this so the IT staff can roll back quickly to a known good state while they repair the system.

Tip 5: Monitor and log your patches.

Of course, the patching job isn’t done until it’s documented. If the IT staff doesn’t keep track of everything it does, it’s likely to have difficulty managing a slowly emerging problem.

Compliance is another reason for keeping good records of all the organization’s patch activity. Increasingly, an IT staff must demonstrate to skeptical outsiders that it routinely patches its systems.

Keep track of patches applied on all the organization’s systems. Maintain this information in a log file and database so that a record is available of which patches have been applied to which systems. It’s also important to monitor the systems to verify that the patches have been applied. Just because the system was programmed to apply a patch doesn’t mean it was successfully applied.

In most circumstances, patch management software will have features such as monitoring, alarms and a permanent log file to help the IT staff with this kind of housekeeping. Patch management requires a lot of little nit-picking details, making it an ideal candidate for automation. Automate patch management as much as possible, but verify that the patches have been applied and are properly documented.