Responding to a Breach

No one wants a data security breach, especially during an economic downturn when the pressure is on to reduce costs and increase efficiency.

Given today's budget realities, it's unrealistic to expect a huge funding increase in response to an incident, so how can you plan for such an event?

Start by understanding the two primary goals of incident management. The first goal is to reduce the negative impact of the incident. This means reducing direct costs such as downtime, fines and fees; and indirect costs, such as a damaged reputation with citizens and less credibility with oversight agencies.

The second goal is to learn from your mistakes. Take the necessary time to determine what could have been done differently that would have reduced the impact of your most recent incident.

Several years ago, an IT manager at my higher-education institution discovered what he thought was a security breach on a server that handled highly sensitive information. He called the manufacturer, and together they ran several diagnostic tools on the system to determine if they had a software bug or if the server had indeed been breached. Unfortunately, only upon confirmation of the system compromise did he contact me for direction. I had him immediately remove the system from the network and told him not to take further action until I could mobilize my staff.

We then took forensically sound images of the system using proper chain-of-custody and preservation techniques. Because the data was very sensitive, we mobilized our critical incident team to receive further guidance. The team decided to hire external forensics firms to ensure there was no possibility of a conflict of interest with respect to the analysis of the breach.

The analysis determined that it was impossible to prove that sensitive data was not accessed or copied. The system had not been appropriately preserved prior to the tools run by the manufacturer, and the IT manager that had modified important time stamps. Plus, our institution had not been collecting network flow data that would have helped determine if any of the data had left our network. We then made a public disclosure of the security breach, with a fair amount of negative press and angst within our community.

Although we made some correct moves during this incident, we also made mistakes. The institution had to pay some steep fines, and the unit operation was closed for one day and had to operate offline for nearly a week while the system was redeployed on entirely new equipment in a much more secure configuration.

As a result of this breach, the security office implemented a new network flow capture service and launched a revised training program for IT staff covering the appropriate procedures for responding to suspected security incidents. We learned some hard lessons, but the good news is that we are now much better prepared should we experience another serious incident.

There are five distinct areas that need to be addressed for a complete and effective incident management program. The general elements of a good incident management program are:

Policy: An enterprisewide policy devoted to IT incident response and management is essential. It should require reporting of incidents, including to whom and in what timeframe; set expectations for the containment of problems and responsibility for them; provide authority for making decisions about internal and external notifications; and outline the entire communications process.

Training: All individuals should be well-informed of their responsibilities and know what procedures to follow when incidents occur. This includes end users, IT administrators, the organization's management team and IT security personnel. Policy, tools and procedures have little value if no one is aware of them -- or of how to use them. The purpose of end-user training is to inform people about policy and ensure that they understand how and when to report incidents.

Operational preparedness: Spell out the nuts and bolts of responding to an incident, a responsibility that typically falls within the information security department, but which could also be managed by technical operations or networking. Your organization needs to have services deployed that detect incidents quickly and have equipment and tools on hand with clear procedures for analyzing what happened. You'll also need the tools and expertise to do a forensic examination, even if your organization does not typically involve legal counsel or law enforcement when security incidents occur.

Being prepared is the key to effective incident management. There is a brief window of opportunity for analyzing what happened from network logs and other data sources. You need to have personnel who are trained and experienced in what must be captured and preserved, and arm them with the appropriate tools. Systems owners are typically impatient to get back into operation, so if you have tools and procedures in place to preserve the necessary information for a later analysis, they will be more cooperative.

Procedures and decision-making: In the heat of an incident, the event will be managed much more efficiently if you have developed and practiced your procedures. Know when to pull together a critical incident team for informing as well as decision-making, and outline the factors that lead to various decisions. At a minimum, you need participation from security, IT, legal, public relations and the affected unit. Depending on the incident's severity, you may also need representation from senior managers, human resources, privacy officers and law enforcement.

Quality improvement: After the dust settles, gather your team and discuss how the entire incident response played out. How was the incident detected? What tools would have hastened the analysis and response? What decisions were made, and were they appropriate? How might the costs of the incident have been reduced? What policies are unclear or missing?

There is always room to learn from your experiences and improve your ability to manage incidents. But it's not entirely about response preparedness and decision-making. Equally important are the factors, procedures, controls or services that could have prevented the incident from happening in the first place.