Oregon Data Center Services handles daily backups for roughly 4,000 state servers, a task made more manageable by a platform that allows its fellow agencies to do some of the work themselves, whether they’re operating applications on the DCS infrastructure or, eventually, in the cloud.
“It’s bread-and-butter IT,” says DCS Director James Foster.
“Agencies are beginning their journey to the cloud,” Foster says, “so we’re looking at our overall data protection strategy to account for that. But we’ll be in a hybrid state for quite some time.”
In state IT, there are few things less sexy — yet totally critical — than data backup, restoration and archiving, and disaster recovery.
“When you think about backup, for example, it’s one of those things people love to hate,” says Phil Goodwin, research vice president for infrastructure software platforms at IDC. “It’s absolutely necessary, but nothing that really adds to the value of the organization. Governments don't offer better services to the community because they have awesome backup.”
Still, ensuring the availability of those services is key, and high availability comes down to ensuring data is accessible, accurate and up to date. And, at a time when state data is created both on-premises and in the cloud, by people and by machines, delivering continuous data protection can be an ongoing challenge.
“States have so many environments: legacy systems, cloud, physical Windows, virtual infrastructure, paid Linux, unpaid Linux. They have traditional structured databases like Oracle and SQL Server, and different file systems. Now they’re getting into containerized applications,” Goodwin says. “It’s this fragmentation of data and the way applications are used that dictate how agencies do data protection, including as a service.”
How Oregon Agencies Rely on Backup as a Service
Oregon DCS, part of the state's Enterprise Information Services, provides IT infrastructure for 69 state agencies, which manage their applications on the DCS infrastructure and, increasingly, in the cloud. Among DCS’s offerings are Backup and Recovery as a Service.
Several years ago, as Oregon’s data was growing rapidly, DCS adopted Commvault Backup and Recovery as the basis of its “as a service” data protection strategy — a unified platform that could increase backup performance while making it easier for agencies to protect and manage their data on the DCS infrastructure. In addition to better handling more than 20 petabytes of deduplicated backup data, the Commvault solution provides a portal-based, self-service capability so individual agencies can run their own backups.
“Each agency can now log in and see what files have been backed up, what servers have been backed up, or if there have been any backup failures,” says Gary Kreiger, deputy director of Oregon DCS. “The portal also allows them to do backups on the fly; for instance, if they’re doing an application upgrade on a web server and they need to initiate a backup on demand, they can now do it without contacting us and opening a ticket.”
Click the banner below to learn how Backup as a Service can enhance data protection.
Since Oregon DCS implemented Commvault, the data it helps protect and manage has continued to increase. To handle the growth and ensure services remain available, the state adopted Commvault’s IntelliSnap virtual server snapshot capability. Because the DCS data center is almost completely virtualized, rather than backing up data on a file-by-file basis, the Commvault IntelliSnap technology streamlines backup and restore by effectively taking point-in-time pictures of the data.
“I don’t think we would be able to back up 4,000 servers nightly without having that technology available,” Kreiger says.
The Commvault platform allows DCS to manage all of that data as one large repository, a capability Foster envisions extending to agencies through the as-a-service model.
“We’re interested in a single pane of glass because we're going to be in a hybrid mode for quite some time,” he says. “We want a seamless, end-to-end experience for our agency partners so they can back up data on-premises or in the cloud, whichever is best.”
DISCOVER: How agencies are using cloud-based analytics for smart solutions.
Utah Is Using Virtual Machines in the Cloud
Like Oregon DCS, the Utah Division of Technology Services has evolved its data protection strategy to be more efficient while also embracing the flexibility offered by cloud technology. In 2016, DTS took the opportunity to improve its data protection by adopting Rubrik security software and appliances.
“We needed to reconsider our backup environment, which was expensive,” explains Dan Harmuth, chief operations officer for Utah DTS. “And we were having trouble doing restores using the product we had at the time.”
Utah runs roughly 1,000 virtual servers in its data center and another 100-plus in a VMware environment in Google Cloud. Therefore, in addition to its on-premises Rubrik data protection platform, Utah DTS manages virtual Rubrik appliances in the cloud for direct backups of its cloud-based VMware servers.
“The cloud presence has two purposes: to back up the servers that are running in the Google VMware environment and to allow us to move stale data to the cloud,” Harmuth says. Utah DTS backs up about 1 petabyte of on-premises data to a cloud storage bucket. In the event of a disaster, this data can be restored to the Google Cloud environment via the Rubrik platform.
“The state saved millions of dollars by reducing the maintenance costs and hardware footprint of the system,” Harmuth says. Utah DTS even received the Governor’s Award for Excellence for its hybrid cloud data protection solution.
EXPLORE: How storage virtualization eases the management of modern workloads.
“It’s improved our disaster recovery restoration services,” Harmuth says. “We regularly have users from agencies calling up and wanting us to recover a file or restore previous versions of application code. We're able to do that more easily and quickly than we could with our old system.”
Snapshots of Utah’s VMware environment are stored on Rubrik hardware on-premises for a short period, while backups are archived to the cloud and kept for longer periods to save expensive local storage space.
“Our admins go into the Rubrik environment, and from the snapshot they can stand up the virtual machine, look at it, determine it’s the right one and then vMotion the server straight from Rubrik into VMware. All the while, users in the production environment keep working and don’t know they’re using a virtual server in Rubrik, even as it's being evaluated,” Harmuth says.
149%
The expected compound annual growth rate of Cyber Recovery as a Service between 2023 and 2026
Source: IDC, “Worldwide Data Protection as a Service Forecast, 2022–2026: Continued High Growth and New Opportunities,” November 2022
Why the Right Storage Methods Matter Across Mississippi Agencies
The Mississippi Department of Revenue had made a studious practice of backing up the SQL databases that underpinned DOR’s vital tax collection and motor vehicle registration systems. When those backups became cumbersome and unreliable, it began the process of overhauling its data protection.
“One day, our tag and title system had a data corruption event and we had to start pulling from backups in the legacy system,” says Mike Dehaan, systems architect for the DOR’s Office of Information Technology. “That was the longest Monday of my career.”
The agency moved to a Pure Storage FlashBlade file and object storage infrastructure, and backup time dropped from days to hours. Restores were also streamlined.
READ MORE: How ransomware threats against state and local governments are changing.
“Part of our business process is restoring production data into staging environments for our developers to work on,” Dehaan says. “Before, we’d start in the wee hours of Sunday morning and it could still be running, still restoring data into staging, come Monday morning. Our new system has revolutionized our workflow.”
Due to the public-facing nature of DOR’s principal systems, backup is important, but replication and restoration are critical. It’s the difference between data resilience and business continuity, Dehaan says. On top of its new, high-speed FlashBlade storage, the state runs Zerto continuous data protection and disaster recovery software to ensure high availability.
“The first time we did a full DR test of our tax system, we gracefully shut it down before starting, failed it over and then brought it back up in under 10 minutes,” Dehaan says. “We can fail over our environment with much tighter service-level agreements than we could if we were restoring from backups.”
