Data Center

Offsite Data Storage Helps Local Agencies with Disaster Recovery

Local governments sidestep disasters at their primary sites by backing up data in remote locations.

Karen J. Bannan is a freelance writer and editor who has written for a variety of publications including The New York Times, The Wall Street Journal, Time and CIO.

Ottawa County, Mich., has about 150 virtual servers that hold a variety of data from the local government’s 34 different departments. The servers contain everything from court and law enforcement files to federal income tax documents. But three years ago, security and business continuity were never assured for the county because its backups were all held in the same location.

The 2003 “Ottawa County Hazards Analysis Report” warned that the local government must be prepared for service disruptions from natural disasters such as “floods, high winds, winter storms and other similar events.”

Today, the county stores its data backups remotely to avoid loss in catastrophes like those weather events, and it’s not alone. Local government agencies establish offsite backup centers as a means to preserve their data and quickly restore operations in the event that a natural disaster hits their primary data center. Modern solutions provide local governments with complete visibility and access to their data in remote locations, along with capabilities to reconstitute their information systems in the event of a localized disaster.

“The ultimate goal is to evolve an automated data governance model that can better manage data growth, provide visibility into dark data and deliver policy-based management regardless of the physical location of the data — in whatever tier, on-premises or off-premises. Data should be maintained based on its actual business value rather than by when it was created or how long it’s been since it was last accessed,” says Steven Hill, senior analyst for applied infrastructure and storage technologies at research firm 451 Research.

Ottawa County, Mich., No Longer Worries About Backups

Mike Morrow, Ottawa County’s manager of technical infrastructure, formally started the process of moving data backups offsite in 2017. The county installed a pair of Cisco UCS S3260 Storage Server devices — one in each of its local data centers — and Veeam software for recovery and replication. The project was complete in three days.

“We loved that Veeam’s focus is backing up virtual machines, and 98 percent of what we’re trying to protect is virtual machines,” Morrow says. Right away, business continuity ceased being an issue since the county’s servers were fully replicated to a separate data center office.

Network Administrator Aaron Becker once spent the bulk of his time backing up and managing county data.

If someone asks for a restore, I can tell them, ‘Give me just a minute and I’ll have it ready for you.’”

Aaron Becker Network Administrator, Ottawa County, Mich.

“I used to stay up at night worrying about our backups: Would we have something recoverable if there was an issue? Plus, aside from just day-to-day work and the monitoring — making sure everything was working — I was spending at least one full day every three weeks just coordinating the movement of that data from disk to tape. It was a huge burden,” says Becker, who tracked four sets of tapes, ensuring the backup and security of all county data.

Now, handling user requests is simple, Becker says. “If someone asks for a restore, I can tell them, ‘Give me just a minute and I’ll have it ready for you.’”

Lewis County Public Utility District Upgrades to All Flash

Power outages are bad enough for consumers, but imagine how utilities deal with service disruptions when they also must keep the lights on.

In 2009, data at the Lewis County Public Utility District, based in Morton, Wash., was relatively structured and easy to store. But now, each of the utility’s 32,000 customers has an electric meter that sends data back to the organization every five minutes. This constant data stream helps Lewis County allocate and schedule enough power to meet demands, which means downtime and data loss aren’t options.

“If we don’t meet our customers’ utility needs, we could get penalties from our administrator, which could cost hundreds of thousands of dollars,” says Jeff Baine, Lewis County Public Utility District’s IS manager.

The growth in data demands coincided with the utility’s 10-terabyte storage server facing its end of life, Baine says. At the time, restoring a single VM could take up to four hours, which was unacceptable. So, the utility replicated its servers in the data center to a separate location about 45 miles away from its primary location. Lewis County PUD purchased a Pure Storage 17TB all-flash server. And it replicates all utility data between two Pure Storage M50 flash arrays, one offsite and one in the office.

“It’s completely amazing,” Baine says. “We have total visibility into our storage. I can look into my system as easily as opening up the web page to it, and I can see my total data reduction too.”

Hutchison, Kan., Learns to Reuse and Recycle Assets for Storage

There are a few reasons why agencies may have skipped offsite replication in the past, cost and logistics chief among them. Installing and maintaining enough hardware to handle both primary and secondary data copies can be expensive.

Jeff Roberson, network administrator for the city of Hutchinson, Kan., acknowledges those difficulties. The city, which has about 45,000 residents, was constrained by not only costs but bandwidth. Using tape, the city performed offsite backups infrequently.

Hutchinson faces the threat of natural disasters like tornados and flooding, which could knock local systems offline. But the city modernized its IT infrastructure, moving to a virtualized environment, adding Veeam software and upgrading its connectivity. Still, there was no money set aside for dedicated storage. But then Roberson had an idea.

“The biggest aha! moment was that I realized I had all this old equipment lying around and a backup software that had replication built in,” Roberson says. He put the old equipment to work.

Today, the primary file and email servers are backed up nightly. The financial files are replicated every 15 minutes, and there’s no undue stress placed on the county’s three-person IT staff.

“You don’t necessarily have to spend a lot of money to do offsite replication, but it’s something that everyone should think about,” Roberson says.

Why Testing Backups Is Critical

Roberson says he always reminds colleagues of the cardinal rule of disaster recovery: A backup is only as good as the quality of replication. “If you don’t test it, you won’t know if what you’re doing is good or not,” he says.

Experts suggest drawing up a plan that assesses the backup objectives and asks these questions:

Are there specific retention goals or requirements?
What are the recovery time objectives? Do they differ depending on the department or user?
Are policies in place to protect the most valuable data and make sure it can be duplicated within those time objectives?
What about the applications? How are they protected?

“Today’s business environment expects recovery time objectives and recovery point objectives of minutes and hours instead of the traditional days and even weeks it could take to reconstruct a complex IT environment from backups,” explains Hill of 451 Research. “The new goal is to protect business-critical and mission-critical applications as well as their data.”

Photography by Logan Zillmer