How 3 Local Governments Mitigated Ransomware Attacks
It was a normal May workday for one employee in Westland, Mich., a thriving suburb of Detroit. An email that appeared to be from human resources dropped into the employee’s inbox.
“It looked like it was a form that needed to be opened and responded to,” says Dan Bourdeau, the city of Westland’s CIO. The public safety employee did what many people in the same situation would do: read the message and clicked on the attachment. Instantly, the would-be form, which contained ransomware, instantiated itself on the employee’s PC, locking down the machine and spreading to a file server on the network.
In November, in Madison County, Ind., a late-night ransomware attack disabled the city’s infrastructure for nearly a week. While the attack is still under police investigation, officials believe ground zero was a networked server set up by a vendor.
“There were a couple of ports that were enabled that were a security risk,” says Lisa Cannon, the county’s IT director. “We think they got in with brute force.”
Her team contained the attack within four hours, but every file was encrypted. The ransomware hit 300 PCs and about 40 servers. The county has cybersecurity insurance, which paid the ransom. It took some time to obtain the keys, however. During the four days that Cannon and her team waited for the files to be unlocked, they assessed and worked on the infrastructure.
“We immediately started putting things in place so this wouldn’t happen to us again,” she says.
These experiences are far from unique, especially at the state and local level, and figuring out how to mitigate ransomware is complicated, explains Brian Calkin, vice president of operations for the Center for Internet Security’s Multi-State Information Sharing and Analysis Center (MS-ISAC).
“Ransomware is 100 percent opportunistic,” Calkin says. “States, in part, are large enterprise networks. For most organizations, it’s going to come down to a business decision.”
That, and a good offense. For instance, in Westland’s case, although it was a zero-day attack, the organization’s anti-virus solution picked up on a pattern and isolated two infected devices, containing everything in less than six minutes. Luckily, no files on the desktop required recovery.
“Our plan for security is isolation, assessment, reporting, evidence preservation, recovery and forensics,” Bourdeau says. “On the file server, all of the files were in fact encrypted, but we were able to get them back within eight hours from our backup solution — a Barracuda appliance that replicates to three secure locations in the United States and one overseas. We were completely recovered in less than a business day. We didn’t have to pay any ransom to anyone, and the ransomware’s actual impact was limited.”
Security Solutions in Action
One reason Westland didn’t face a complete shutdown and pricey ransom — hackers demanded $25,000 per device and Westland has 350 endpoints — was that it utilized what Bourdeau calls an onion strategy. His IT department of five employs a combination of email whitelisting, strong anti-virus and malware protection, backup and recovery technologies, and human processes.
When Sendio email security didn’t catch the phishing email, Trend Micro OfficeScan endpoint protection recognized that the ransomware’s behavior wasn’t normal and generated feedback to isolate the affected devices. The city’s backup solution made it possible for Bourdeau to wipe the devices before restoring all of the lost files, minimizing the impact.
While anti-virus programs and firewalls are key defenses in any cybersecurity toolkit, Gordy LaChance, the IT manager for Janesville, Wis., credits a packet-filtering device for saving his organization’s files from being completely infected.
When employees there received a phishing email containing ransomware, it contained the name and logo of a legal firm that conducts business with the city. Someone not only opened it, but also sent it along to several colleagues who also opened it, releasing the payload. The ransomware launched, but because it couldn’t call home, the files were never encrypted. LaChance and his team still needed to work quickly, however, to contain the issue and get operations back online.
“Recovery was very painful,” he says. “We moved those files and, out of an abundance of caution, air-gapped our network for six hours. We then pulled all of the files from the location and deleted them.”
All told, the team deleted several million affected files. LaChance was able to restore all of the lost files from a recent backup.
Back in Madison County, the first change in recovery efforts was shutting down the vendor server that started everything. Once the server’s ports were disabled, Cannon says her IT department examined everything else that was going on, assessing the organization’s servers, anti-virus protection and firewalls — including Cisco Systems’ Advanced Malware Protection and Adaptive Security Appliance, along with Trend Micro OfficeScan — before making changes proactively.
“While we were checking all potential areas of risk, we used that process as a learning tool for some of the lower level support people,” she says. “We also beefed up our web filtering and content filtering too. We used to be very lenient. Those days are gone.”
Cannon implemented a strategy to limit and isolate devices that engage in risky behavior, such as frequenting social media sites. That is important, she says, because Facebook in particular has become a popular entry point for ransomware. As a result, when the county’s probation and drug support departments require social media platforms as investigative tools, they now access those sites from PCs set up on a segregated network.
While it’s important to assess the network, hardware and software after a ransomware attack, organizations also need to revisit employee awareness and ongoing training.
“If you make the investment in technology but haven’t made the investment in people to review it all, it’s a wasted effort,” says MS-ISAC’s Calkin. IT employees should be active in industry forums and pay close attention to security alerts from vendors and other organizations, he says. They should also undergo frequent training to receive updates on the latest security threats, mitigation and remediation efforts.
Sharing information also helps to reduce the risk of infection. The 2016 Deloitte-NASCIO Cybersecurity Study, “State Governments at Risk: Turning Strategy and Awareness into Progress,” found that collaboration both within and between state and local governments is “becoming central to state government strategy.” These collaborative efforts include working with MS-ISAC, the U.S. Department of Homeland Security’s fusion centers and other IT organizations.
The Human Connection of IT Security
Protection from ransomware goes deeper than the IT department, though. For instance, in Westland, Bourdeau and his team were notified by both the security software and the employee who made the mistake. The employee felt comfortable calling because Westland makes it a practice to avoid embarrassing people for falling victim to a socially engineered attack.
Post-infection, the city started quarterly evaluations of the security structures inside Active Directory so that Bourdeau knows who has permissions and who doesn’t. He beefed up the organization’s already strong education process and reasserted that the organization won’t discipline anyone who opens ransomware.
“We don’t want employees to hide anything. I want them to feel comfortable calling IT if they have any problems. I want them to know they are going to be received and helped,” Bourdeau says.
Michael Kaiser, executive director of the National Cyber Security Alliance, a public/private partnership dedicated to cybersecurity, says Bourdeau’s strategy is one that every organization should emulate.
“Part of the culture of cybersecurity is creating an atmosphere where employees feel that reporting cyberbreaches is a good thing,” he says. “You have to use education and show people how to use critical-thinking skills to make the best choices they can make, but don’t punish someone if they make a mistake. And that’s what it is. It’s a mistake.”
Today, the city of Westland engages its employees two to three times each month (as opposed to three or four times per year) and shares information about the most recent social engineering programs. He has also created training videos to further engage employees.
“We’re shamelessly copying social media and social trends to deliver this critical business information,” he says.
