Cities and Counties Cut Downtime with IT Infrastructure Upgrades

Frustrated by frequent network outages and data center downtime? When it comes to technology woes, Otsego County, N.Y., proves that, sometimes, agencies just need to toss out the old and start over.

After 14 years of service, the ­county’s network equipment had reached the end of life, while some servers and storage were more than a decade old and approaching capacity. During nightly backups, storage hardware frequently ran out of disk drive space and took applications down.

“It was unstable,” IT Director Brian Pokorny says. “On a weekly basis, it seemed like one system or another went down, and it seemed like we were spending the majority of our time fixing network issues. So we finally upgraded.”

No city or county ever wants to suffer downtime. To ensure high availability, some governments like Otsego County modernize antiquated equipment. State-of-the-art technology, such as converged or hyperconverged infrastructure, takes advantage of virtualization and offers high redundancy. Local governments also find success through a broad set of strategies and technology to bolster uptime.

San Mateo County, Calif., for ­example, designed a network and data center with high availability and uses software to monitor uptime, while the city of Cumberland, Md., deployed a good data backup and disaster recovery plan.

“Be prepared. Plan for things to fail and break, and design around that. It’s the classic ‘strong offense is the best defense,’” says Greg Schulz, founder of the Server and StorageIO Group, an IT consulting firm.

SIGN UP: Get more news from the StateTech newsletter in your inbox every two weeks

Ostego County Charts a Road Map to IT Redundancy

In Otsego County, tight budgets prevented the IT department from replacing outdated infrastructure. But last fall, county officials finally found funding to address the downtime issues and invest in new ­equipment to build a robust network and two fully redundant data centers, ensuring that critical ­services like computer-aided dispatch at the 911 ­center stay up and running.

After consulting with CDW, Pokorny opted for a converged ExpressPod solution featuring ­­Cisco Systems servers and network gear, NetApp storage area networks (SANs) and VMware virtualization software.

ExpressPod offers a preconfigured and prevalidated solution that integrates different components — servers, storage and network — into one unit, making it easier to deploy and manage.

CDW engineers began the installation last fall and took about four months to deploy the equipment at the county’s main and backup data centers. Both locations house two Cisco UCS C220 rack servers, a NetApp FAS2554 SAN and a Cisco Catalyst 4500-X Series switch at the core.

For added redundancy, the NetApp SANs feature two dual controllers, and each rack server has two 10-Gigabit Ethernet network connections, says Yasir Jahangiri, a CDW senior consulting engineer.

Today, the two data centers replicate to each other in real time, providing the county with redundancy that it didn’t have before, says CDW FlexPod Architect Joel Nordsving.

With virtualized servers, the IT staff now has the flexibility to move virtual machines (VMs) from one server to another to perform upgrades and install patches without taking systems offline. In the past, the county had to wait until lunch or ­after-hours to take down systems for maintenance.

“The new system gives us a more reliable environment and gives my technicians maneuverability to work on systems without affecting users,” Pokorny says.

For networking, the county’s new 4500-X core switches increased speeds tenfold, from 1 gigabit per second to 10Gbps. The IT staff also upgraded the edge switches with Cisco Catalyst 2960s.

The county has a strong network, but the IT staff plans to improve it even more in the future, Pokorny says.

As budget allows, the county will upgrade the core with Nexus switches and improve redundancy.

“It’s refreshing to start off with a good foundation and be able to build on that,” Pokorny says.

San Mateo Embraces Holistic IT with Hyperconvergence

San Mateo County, which supports more than 200 applications, has consistently enjoyed 99.94 percent uptime since Jon Walton joined as CIO four years ago. To provide employees and citizens highly available IT services, Walton and his IT team took a comprehensive approach by deploying software to ­monitor uptime, upgrading to a hyperconverged infrastructure and simplifying the network architecture.

Walton’s first project was to install SolarWinds’ software tool to monitor uptime and get a baseline measurement of how the county’s core infrastructure performed so he could look for areas to improve. Today, if downtime occurs, IT staffers rely on the software to analyze and pinpoint the cause.

“If individuals have trouble accessing an application, we need to determine if the server really is down or if there is latency on the network,” he says. “That gives us a holistic view.”

Age, complexity and a lack of standardization of IT infrastructure can also cause downtime or extended outages, Walton says. In 2013, the county’s data center had a hodgepodge of servers and storage equipment from multiple vendors. “If you have a lean staff with only one or two people who understand the AS/400, and one or two people who understand EMC SANs, not everyone is available 24/7 to fix problems,” he says.

To resolve that issue, the team standardized on a Nutanix hyperconverged solution, which combines servers and storage in one appliance for easier management, as well as power and space savings.

Today, the county’s main data center replicates to a backup data center. If a disk or storage node on the Nutanix equipment fails, others in the cluster take over to ensure VMs continue to run and data is accessible.

“It’s one of the reasons we rarely have downtime,” Walton says.

Cumberland County Prioritizes Backup and Disaster Recovery

The city of Cumberland, Md., found itself in the same boat as Otsego County four years ago, with aging, unstable servers. To boost availability, the city purchased new HP servers and Nimble Storage arrays for its main and backup data centers, and virtualized its applications.

Investing in data backup technology also bolsters disaster recovery and ­minimizes downtime, says Johnna Byers, director of management information systems for the city of Cumberland.

Cumberland used to rely on tape backups, but three years ago it purchased two ExaGrid disk-based backup appliances. The city chose Veeam’s data backup software and the two ExaGrid systems to provide two backups of ­virtual servers and data. The new ­technology speeds up nightly backups and gives Byers peace of mind.

“My old boss once said, ‘You’re only as safe as your worst backup,’ and for years we were dealing with tapes, and sometimes they didn’t work,” she says. “Because we deal with public safety, it’s important for us to back up data ­reliably.”