“We are a 24-hour operation. Downtime is not an option,” says Ivan Phillips, Information Systems Director, Muskegon County, Mich.

Apr 14 2020

Why Active-Active Data Centers Fit the Bill for Local Governments

Some cities and counties eliminate the worry of outages by investing in two running data centers.

In Michigan’s Muskegon County, some departments need 24/7 access to computers, applications and the IP-based phone system, including the Muskegon County Sheriff's Office, the Muskegon County Juvenile Transition Center and the Muskegon County Wastewater Management System.

To improve uptime, the county recently upgraded its IT infrastructure and turned its primary and secondary data centers into an active-active environment. Now, if one data center goes down, the second site automatically continues IT operations. “We are a 24-hour operation. Downtime is not an option,” says Ivan Phillips, information systems director for the county.

Local governments are increasingly embracing active-active data center ­configurations to improve resiliency and bolster continuity, says Greg Schulz, founder and senior analyst at StorageIO, an IT consulting firm in Stillwater, Minn. Government agencies rely on ­critical applications such as 911 systems, and residents increasingly want anytime access to services, such as the ability to pay property taxes during off hours.

“More and more are doing active-active. It used to be for the large, high-profile organizations,” Schulz says. “There was a perception and belief that smaller organizations didn’t need it. But the reality today is that smaller organizations — whether they are small state agencies, counties or cities — are realizing that information access is time- sensitive, and that the risk and cost of not doing active-active outweighs the expense of doing it.”

Minimize Downtime with Active-Active Data Centers 

Bad winter storms have the potential to knock out power in Muskegon County. And while the county uses uninterruptible power supplies and generators, its information systems department doesn’t want to take chances. It’s moving from an active-passive to an active-active data center model this spring.

With approval from the county board of commissioners, the IT staff replaced aging servers and storage equipment that were reaching capacity with new Cisco Unified Communications System servers, flash storage from Pure Storage, the latest VMware vSphere server virtualization software and VMware Horizon desktop virtualization software.

Each data center is now powered by 10 Cisco UCS B200 M5 blade servers and a Pure Storage X50R2 150-terabyte flash array, which provides enough capacity for growth for the next five years, Phillips says.

It’s converged infrastructure that’s prevalidated and configured to seamlessly work together, making it easier to deploy and manage, says Mark Hansen, the county’s information systems manager.

The county hosts about 150 virtual servers and about 600 virtual desktops for its employees. Nearly every county department relies on the IT infrastructure, including the county airport and a court system that is going electronic and paperless, Phillips says.

 Ivan Phillips, Information Systems Director, Muskegon County, Mich.
If servers go down, our users will never know it. There will be no disruption in service, which is our goal.”

Ivan Phillips Information Systems Director, Muskegon County, Mich.

The county spent more on hardware to deploy the active-active configuration, but it’s worth it, Phillips says. In the past, a prolonged power outage would have forced the IT staff to manually bring applications up in the secondary data center.

Now, with Pure Storage software, data replicates from one data center to another while the county operates the virtual servers and desktops as a single stretched cluster, meaning they are load balanced, with half the virtual machines running in the main data center and the other half in the second data center, Hansen says.

If one data center goes down, VMware software will automatically migrate those VMs to the second data center. “If servers go down, our users will never know it. There will be no disruption in service, which is our goal,” Hansen says.

The active-active configuration also allows the IT staff to update software and firmware without causing downtime. “We can upgrade and do maintenance and keep operations running,” says Dave Majeski, the county’s system administrator. 

READ MORE: Discover how software-defined everything is changing state government. 

Active-Active Data Centers Help Support Critical Apps

When Vanetta Pledger joined the city of Alexandria, Va., as CIO, one of her first projects was to relocate the IT infrastructure to a new data center space. The new data center featured electronic access control systems, generators and a new power and cooling system.

While relocating, Pledger and her team discussed disaster recovery and how to reduce the risk of downtime. “We noticed the data center had become a single point of failure, so we asked ourselves, ‘How do we make sure there’s resiliency and that we always have reliability for our mission-critical applications?’” she says.

Over the past decade, the city had already adopted some active-active measures, such as subscribing to redundant internet services and virtualizing servers. So, she and her team decided to go all-in and implement an active-active data center design for the city’s most critical applications, such as the 911 emergency communications system and financial applications.

Pledger developed a business case and pitched it to city leaders, who approved the investment. She standardized on new Dell EMC servers and storage arrays for both a primary and secondary data center and completed the project in fall 2018.

To ensure success, her team received training on managing an active-active configuration. To lower some software licensing costs, not every application fails over automatically. If the primary data center goes down, for example, the IT staff will manually press a button to bring Microsoft SQL Server databases back up at the secondary site, she says. 

MORE FROM STATETECH: See how hyperconvergence provides fast storage for government apps.

Deduplicating Hardware Cuts Costs for Local Governments

The city of Denton, Texas, has deployed active-active data centers for the past 11 years for mission-critical Tier 1 applications, such as enterprise resource planning, email and its IP-based phone system. But during the past year, the city went further, modernizing its IT infrastructure and moving all the second- and third-tier applications to an active-active model.

The city was able to afford it because it recently saved more than $1 million by consolidating its electric utility’s IT infrastructure with the city’s general IT infrastructure and because of deduplication in its new storage hardware, says Melissa Kraft, Denton’s CTO.

The city upgraded its infrastructure with converged infrastructure that includes Cisco UCS blade servers, Cisco switches, Dell EMC Unity flash storage hardware and VMware virtualization software. Dell EMC’s flash storage deduplicates data, eliminating redundant copies of data, which reduces the storage space required by the city. Deduplication reduced the city’s storage requirements by 20 percent, which made an active-active configuration more affordable, says Paul Desjardins, the city’s enterprise infrastructure manager.

Kraft’s department purchased the new hardware last March and completed the project in January.

“Through deduplication, we were able to easily afford an active-active scenario, whereas before, the storage itself would have been too cost-prohibitive,” Kraft says.

With the upgrade, the city’s applications went from 20 percent active-active to 100 percent, Desjardins says.

Both data centers are essentially operating a mirror image of each other. “People would consider our secondary site our disaster recovery location, but there is no longer a primary or secondary data center. It’s just active-active,” says Desjardins, who adds that the new hardware will meet the city’s growth needs for the next five years.

When exploring whether to deploy active-active data centers, government IT leaders must assess what their risk tolerance is for potential downtime, Kraft says.

If budget is a concern, and it always is in government, IT staff can start with the most critical Tier 1 apps first — just like Denton did, Kraft says. That allows government leaders to see the benefits of active-active. Over time, as budget allows, the IT department can move more apps to active-active mode, she says.

“Active-active brings value as more services move toward 24 hours,” Desjardins says. “You don’t want to be down overnight.”

Photography By Logan Zillmer