Cloud

Disaster Recovery via Hybrid Cloud Appeals to Local Governments

Agencies look to cloud-based backup to maintain critical citizen services.

Chris Hayhurst is a contributor to the CDW family of technology magazines.

It was spring 2019 in Grey County, Ontario, and Evan Davis was losing sleep.

As the technology and infrastructure manager for the county, which includes nine municipalities in the Canadian province’s southwest, Davis recently had overseen a “significant” investment in the government’s IT holdings. “We’d bought all kinds of things we needed to run our systems — new servers, new firewalls, new switching equipment,” he says. “What we hadn’t yet done is come up with a good plan for what we’d do should anything go wrong.”

As it surveyed its new infrastructure, the IT department realized it could be looking at days or weeks of downtime in the event of a serious system failure. Certain disasters — like a ransomware attack — could potentially lead to loss of reams of valuable data. “That just wasn’t acceptable,” Davis says. “The way the world is now, nothing can go down. Everything has to be up and available all the time.”

Grey County ultimately moved backup and disaster recovery to a hybrid cloud for reasons familiar to analysts who watch the issue. “I believe it’s becoming best practice to explore cloud as a DR solution, and I expect adoption to increase in the coming years,” says Mike Andres, managing director with Deloitte Consulting.

Most agencies that do embrace cloud-based DR platforms find the migration to be relatively straightforward, Andres says. While few organizations ever actually need to “execute their DR solutions,” those that do often benefit from the much lower recovery time and recovery point objectives that cloud services offer their systems.

As the threat landscape continues to grow, and as agencies gain confidence in cloud solutions in general, many will re-evaluate their allegiances to traditional DR vendors “and begin moving to the cloud rapidly as their contracts come up for renewal,” Andres predicts.

Using the Cloud to Return to Business Quickly

Like many regional governments, Grey County is responsible for a wide range of programs that are critical to its more than 96,000 citizens, from emergency, medical and welfare services to plowing and general road maintenance. To maintain continuity, Davis’s team previously relied on what he describes as a “standard, old-fashioned backup solution” managed on-premises at IT headquarters. Each night, at a designated time, more than 100 computer systems across the county were backed up to this single server.

After the upgrades, Davis and his team began working with a consultant to develop a comprehensive disaster recovery assessment. That process, which took about a year, helped to clarify what was at stake — and it led them to a backup and recovery solution with the Zerto continuous data platform and Microsoft Azure as the recovery target.

“We’d thought about adding a second data center, but that was going to be too expensive,” Davis says. Relying on Zerto in Azure instead allows his team to scale up as necessary when new systems are added to its IT matrix. Zerto connects seamlessly with Azure, which the county now also uses in a hybrid cloud computing platform.

77%

The percentage of government backup that will be cloud-based by 2022

Source: veeam.com, “Modern Data Backup for State and Local Government,” May 19, 2021

“Doing it this way is cheaper and a whole lot easier than housing a backup solution onsite,” Davis says. “The great thing about it being in the cloud: It’s not something we have to manage or take care of ourselves.”

Grey County hasn’t faced a disaster scenario in the year or so it has relied on Zerto. In the event that one does occur, however, Davis is certain the technology will save the day: “Knock on wood, of course, but even with a full-scale disaster, we should be running again a lot sooner than we would have previously.”

Davis knows of other Ontario agencies that lost data and took months to recover following attacks by hackers. “If they had this, they likely would have been in a much better position,” he says.

FREE RESOURCES: Get the tools you need to modernize your IT infrastructure.

Minimizing Data Loss in the Event of Ransomware

It was a ransomware attack that drove the city of Lodi in California to walk away from its traditional backup system. Right around the time that Evan Davis was worrying about Grey County’s data vulnerabilities, Lodi IT Manager Benjamin Buecher went to work to find that many of his agency’s files and systems had been encrypted.

“Luckily, our backup server wasn’t touched, but that whole experience made us rethink our approach to data protection and security,” Buecher says.

His team researched its options, Buecher says, and in the end decided to switch to a backup and recovery service from Rubrik. “The way Rubrik’s internal system operates on our network makes it far less vulnerable to an attack than our old system. Being a small shop, my guys wear multiple hats, so having something that is intuitive and flexible was important to us as well.”

A plug-and-play Rubrik Cloud Data Management appliance now sits at Lodi’s IT headquarters, Buecher says. The system allows the team to set policies for backup frequency and retention and automates the work accordingly.

If a disaster should occur — something Lodi hasn’t experienced since the city’s mishap two years ago — data loss is minimized with what Rubrik claims is a “near zero” recovery time.

Restoring Virtual Machines and Data at Scale

The 2019 attack temporarily crippled Lodi’s ability to access the data and systems it needed to provide municipal services, Buecher says. Describing the recovery process as “incredibly tedious,” he says it took his team almost a full month to manually restore all of the agency’s virtual machines.

The process was especially drawn out because the old DR solution required file-level backups. Instead of hitting a button to rebuild the agency’s systems, the team was forced to put them back together from scratch.

The new DR system does away with such challenges with tools that allow the Lodi IT team to restore its virtual machines at scale. Buecher, however, has no intention of ever putting the Rubrik solution to the test. Instead, he says, he’s focused on prevention — on avoiding situations where disaster recovery would come into play.

“The biggest takeaway for us from this experience wasn’t just that we needed better DR, it was that we had to train our users to help protect our environment,” Buecher says.

The city of Lodi will happily turn to its cloud-based system for disaster recovery when necessary, “but for now, we’re just trying to improve our weakest link: the person who decides to click on that rogue email.”

Illustration by Jacey