When the small city of Sammamish, Wash., was hit by a ransomware attack in January 2019, Stephen Schommer, a recently retired public utility IT director, volunteered to help.
“I have done dozens of tabletop exercises but never experienced ransomware firsthand, thank God,” Schommer recalls. After a few days of assessing the situation, Schommer was a little stunned by what he found. Every city server was encrypted and held for ransom, although the city of 65,000 had no intention of paying the cybercriminals. But the city’s 120 employees couldn’t do their work, and there was no IT director in place.
Sammamish had purchased an endpoint malware solution but only deployed it to 20 percent of its endpoints and a few of its servers. The city had an 8-year-old firewall with no advanced threat protection. “I realized there wasn’t any organizational structure to this,” Schommer says.
State and local governments that don’t have an organized approach to endpoint security may suffer from such ransomware attacks. Maggie Brunner, the National Governors Association’s program director for homeland security and public safety, says that there are some real haves and have-nots at the local level, depending on size. The combination of a holistic endpoint security solution and knowledgeable IT personnel on the ground can make or break a cybersecurity response. Cities and states discover that hiring critical leadership in-house or centralizing management across the enterprise can make a big difference.
“Many municipalities outsource IT, so there isn’t always someone at the local level who can tackle information security,” Brunner says. Increasingly, governors realize they have to think beyond the state networks and data to involve critical infrastructure, businesses and individual citizens. “With the drastic increase in ransomware, they are asking what they can do to help their county and municipal partners,” she says.
IT Investments Made a Difference in Sammamish
Schommer was instrumental in getting Sammamish back on track. At a meeting with city officials and other volunteers, he drew up a plan for first building an IT perimeter and developing a secure environment for work. His approach was accepted, and the city recruited him to come out of retirement and lead their recovery for the next six months.
The first step was to get a modern firewall deployed and allow only web traffic. Schommer had a good relationship with solutions vendor FireEye from his time with the utility, which was a long-term FireEye customer. His contacts there overnighted him one of their network appliances that inspects network traffic to give him visibility into what was going on.
“You can’t fight what you can’t see,” Schommer says. “That allowed me to discover the type of ransomware, the active callbacks to the command-and-control server and the type of encryption they had used. Then we started to rebuild the city systems from the ground up.” The city now deploys FireEye’s network, email and endpoint security tools.
Schommer says the increasing number of news reports about ransomware is making it obvious that many municipalities and state agencies large and small are in the same boat Sammamish was in before the attack. “Cybersecurity equipment, endpoint protection and advanced email protection are the three components that I think every agency needs to have, because that is how the attackers get in,” he says.
READ MORE: What are the benefits of cybersecurit automation in government?
A Fresh Start with New IT Leadership
As he left Sammamish last summer, Schommer and the city finance director estimated the six weeks or so of downtime for 120 employees cost the city $1.1 million in lost labor costs — the soft costs. The city felt the impact in other ways as well.
“Vendors were not getting paid, builders were not getting permits — there are a lot of downstream impacts.”
Fast-forward a year, and Sammamish is in a much better place, according to current IT Director Jim Hominiuk. “We have continued to expand our layered approach to cybersecurity. We leverage multiple vendors like Microsoft and Cisco to help us,” he says.
He adds that the city has redoubled its efforts on training and employee awareness of cyberthreats. “City staff members are much more aware of phishing attempts, and we have increased cyber awareness through training and communications.”
Hominiuk notes that city leaders also have a better grasp of the importance of investing in IT resources. “City council has provided an increased level of funding to the IT department to enable us to increase our infrastructure’s security posture,” he says.
His advice for other municipalities? “Partner with your executive leadership group and educate them about cyber-attack prevention.”
MORE FROM STATETECH: Find out how states can best quantify cybersecurity risks.
Oklahoma Centralized Security Personnel and Resources
From 2012 to 2016, the state of Oklahoma went through an IT consolidation effort, combining 111 separate IT units into one.
In 2019, Matt Singleton, who was COO for the information services division during the consolidation, became CISO. He describes how centralized cybersecurity leadership and solutions accompanied the consolidation effort.
Before 2012, some state agencies had IT security solutions in place, and others didn’t. “When they started standing up a central security office for the state, endpoint security was one of the first things they tackled, and because it was the first time the state had taken an enterprise approach to it, they were able to negotiate a pretty fantastic deal to go to a standardized platform,” Singleton says.
16%
The percentage of reported breaches involving public sector entities
Source: Verizon, “2019 Data Breach Investigations Report,” May 2019
Oklahoma deployed the Symantec IT Management Suite, including Symantec’s Endpoint Protection, DeepSight Security Intelligence and Business Critical Support solutions.
“That resulted in a successful rollout and cost savings for the state,” Singleton says. “Although we will still take a centralized approach going forward, we are now in the process of re-evaluating providers. It is time to go through and modernize a lot of our toolsets, specifically those things that directly impact our end users.”
DOWNLOAD: Read this white paper to find out how your agency can protect data in a changing security landscape.
How to Enhance Cybersecurity Collaboration
Singleton also has developed a roadmap for the additional investments Oklahoma plans to make. One goal is to establish a cybersecurity supply chain practice in the state. “With some nations buying up portions of companies,” he says, “we really want a good understanding of all the players in our supply chain, what their motivations might be, and what protections we have in place.”
The state also is focused on increasing collaboration. Singleton recently hired a director of security engagement and outreach.
“One of her primary deliverables is standing up an Oklahoma-based information sharing and analysis center that will complement what is happening at the federal level through the Multi-State ISAC as well as focus on building partnerships with organizations in Oklahoma. “It is important that we do this together,” he stresses. “No single organization will have all the intelligence, capabilities and resources to be successful.”
Singleton’s emphasis is on resilience. He wants the organization to not just survive and bounce back from attacks, but also learn from them. “Using machine learning capabilities, we want to get to the point where we can ignore most attacks because they have very little effect on our operations,” he says. “The idea behind that is to let our analysts and engineers focus on the hard stuff — the advanced persistent threats. We want to come back stronger after attacks.
