Cities and Counties Gear Up to Defend IT Against Catastrophe
Earthquakes are part of life in Southern California, and a major temblor is always a threat to shake up the region. When it does, the Beverly Hills IT department is ready. The city’s leaders have made disaster preparedness a top priority since 2005, when Hurricane Katrina slammed into the Gulf Coast and local, state and federal agencies made missteps in the response and rescue efforts.
In recent years, Beverly Hills has built two backup data centers, with redundant network connections to ensure that critical applications, such as police and fire communications, continue to operate during an emergency. To further improve resiliency, the city is also virtualizing its servers, investing in multiple backup solutions and considering a third backup data center out of state.
“Every year, we have made incremental investments into disaster recovery and business continuity,” says Beverly Hills CIO David Schirmer. “The thought is that any of our data center locations could take a catastrophic hit, and we could still carry on.”
SIGN UP: Get more news from the StateTech newsletter in your inbox every two weeks!
To Protect IT, Expect the Unexpected
Like Beverly Hills, other cities and counties are stepping up their technology investments to bolster DR and emergency management operations. They not only have to worry about natural disasters, such as hurricanes, tornadoes and flooding, but they also must prepare for other emergencies, such as terrorist attacks, mass shootings or even IT-related issues, like malware infections, say industry analysts and government IT leaders. Solutions ranging from cloud services and data backup and recovery software to storage and networking equipment are key elements to being ready for the unexpected.
“Natural disasters get the headlines, but in many cases, people will use disaster recovery capabilities to recover from mundane kinds of situations, like a pipe bursting in a data center,” says Phil Goodwin, IDC’s research director for data protection, availability and recovery.
A good business continuity and DR plan requires a combination of people, processes and technology, Goodwin says. IT organizations need to create a game plan and cross-train IT personnel to manage a variety of tasks to ensure applications and services remain available in emergencies — or to bring them back online.
“IT has to establish a relationship and communicate with departments to have a comprehensive organizational plan,” Goodwin says.
Orlando Urges Cities to Plan Ahead
Orlando’s IT team has done just that and played a major role in the Florida city’s successful response to Hurricane Irma in September 2017 and the Pulse nightclub shooting in 2016, which killed 49 people and injured 53 others.
Orlando CIO Rosa Akhtarkhavari is diligent about planning, so four days before Irma arrived, she reviewed her to-do list to make sure systems and equipment were ready at the city’s emergency operations center (EOC).
Photography by Cy Cyr.
She and her IT staff checked that the Wi-Fi and wired networks were operational and that the city’s transportation and public safety cameras were accessible. They used an IP phone system to set up a phone bank so citizens could call for information and report damage. The IT team also set up computers for staff who didn’t bring their own devices.
The city’s primary data center is designed to fail over to another center in the region. But the day before Irma’s arrival, Akhtarkhavari used a package delivery service to mail encrypted tapes of critical data to Seattle’s IT department for safekeeping and uploaded less sensitive data to a cloud provider.
Real-Time Maps Simplify Recovery
As the storm approached Orlando, Akhtarkhavari and five IT staffers reported to the EOC. The storm brought rain and 79 mph winds that knocked down trees and caused power outages. “We were there to monitor connectivity to all critical infrastructure,” she says.
A video wall in the EOC displayed a real-time geographic information system (GIS) map that showed the latest power outage and damage information throughout the city. The map was updated live as citizens called in with information and as first responders and public works teams assessed the damage.
A year earlier, when the nightclub shooting occurred, Akhtarkhavari had no time to prepare. She and other department heads reported to the EOC immediately to manage the response.
Orlando’s IT staff deployed a disaster hotline with 48 IP-based phones, and provided computer and internet access for two centers: one for family reunification and another for family support.
Akhtarkhavari called key tech vendors for help. Microsoft supplied loaner devices and Cisco Systems contributed switches. When Orlando’s IT team developed a donation website, Microsoft hosted the page on its Azure cloud.
The IT staff was at the EOC throughout the aftermath of the nightclub shooting and during the hurricane to make sure IT equipment remained operational.
“When a disaster occurs, we work like crazy and make things happen,” Akhtarkhavari says.
Beverley Hills IT Prioritizes Redundancy
In California, Beverly Hills’ IT department plans for disasters by trying to eliminate single points of failure in its technology infrastructure every year, says Assistant CIO Mark Hobson.
The affluent community is home to about 35,000 residents, Hollywood celebrities and the luxury boutique stores and upscale restaurants of Rodeo Drive. But the Hollywood and Santa Monica faults run underneath the city, and the Newport-Inglewood and San Andreas faults are nearby.
To prepare for earthquakes and other disasters, the city built a backup data center at a colocation site near Los Angeles International Airport and a second back-up center in downtown Los Angeles.
“Our data centers are located on different fault areas, so the thought is that two of the data centers would not be impacted at the same time,” Schirmer says.
Photography by Aaron Smith.
To improve redundancy, the city built multiple connections to each remote data center — a 40-gigabit-per-second fiber network that’s backed up by leased lines.
The IT team uses VMware vSphere to virtualize its servers to further improve continuity of operations. When the project is complete, the city will consolidate its 300 Dell, Lenovo and Hewlett Packard Enterprise servers down to 25 servers at its main data center, Hobson says.
Furthermore, the IT department is investing in multiple data backup technologies. Beverly Hills uses Barracuda Networks appliances for site-to-site replication between data centers. Recently, the IT staff purchased Tintri VMstore Flash storage appliances, whose snapshots of virtual machines and data allow for faster data recovery, Hobson says.
The city purchased Actifio storage appliances as a third backup solution.
Denton, Texas, Tests and Updates to Stay Ahead of Disasters
While planning for contingencies is essential, so is making sure procedures will work. Updating and testing a business continuity and DR plan every year allows an IT department to ensure effectiveness, says Melissa Kraft, CTO of the city of Denton, Texas.
“We don’t want to think about it during a disaster. We want to be prepared,” she says.
Denton, a city of 133,000 residents in the Dallas-Fort Worth metroplex, is at high risk of tornadoes.
Kraft and her IT staff update the city’s continuity of operations plan every year with city leaders and departments. The process includes performing a risk assessment and business impact analysis to prioritize applications and services, discussing expected service levels, and determining the recovery point objective (or RPO, the amount of data that can be lost) and recovery time objective (or RTO, the length of time applications can be down).
The city’s two data centers are active-active, meaning if the main data center goes down, the other data center keeps IT operations running.
The city has standardized on Cisco UCS servers, EMC storage and Cisco networking. Its EMC VPLEX Flash storage systems replicate data between sites, and Veeam replication software backs up data with real-time snapshots.
To keep costs down, the IT department prioritizes applications and services by splitting them into different categories. Mission-critical applications, such as enterprise resource planning software, email and the IP-based phone system, are considered Tier 1 applications and are kept active-active in the data centers, Kraft says.
Tier 2 applications, which include work order systems, have a four-hour RTO; less important Tier 3 applications have a 24-hour RTO.
Similarly, in Beverly Hills, the IT department has analyzed its applications and created a workbook documenting which systems need to come up first and their RTO and RPO, Hobson says.
If the main data center fails, the backup data centers will keep critical applications running, such as the 911 dispatch system, Schirmer says.
“The high-priority applications will continue seamlessly. The less critical ones will take some intervention to work,” he says.
Denton’s IT department runs monthly exercises with different departments, Kraft says; for instance, taking live applications and restoring them from backup.
“We can analyze how we did,” she says, “and improve it for the future.”
