Automation Spurs Efficient State and Local Data Center Management

Two years ago, Durham County, N.C., faced a ­classic problem: As technology demands from its 300,000 constituents increased exponentially, static budgets made replacing the aging tech ­infrastructure difficult. For CIO Greg Marrow and his team at the ­county’s Information Services & Technology department, the solution was software-defined data center architecture and automation.

Back then, the Cisco Nexus 2000 and 5000 Series network switches inside the county data center were nearing the end of their support lifecycle, Senior Network Engineer Seth Price says. The team decided to move from traditional data center architecture to Cisco’s Application Centric Infrastructure, a software-defined networking solution that allows for greater flexibility, scalability, ­visibility and ease of management, he says.

“We needed to simplify our data center, to do more with less,” Price says. “We don’t have enough staff to cover all the work that can come our way. So, the driving force behind going to ACI and SDN was improving our efficiency and being able to automate processes.”

By setting up the right network policies, Durham County now automates repetitive tasks such as load balancing and network provisioning, which otherwise require human intervention. For example, setting up a physical network can take 40 minutes or longer, Price says, but doing so with ACI takes less than three.

SDNs also allow IT staff to microsegment policies and isolate each machine on the network, limiting who they can communicate with while greatly enhancing network security.

“I can put a little firewall around every person attached to our public wireless network that only allows them access to the internet,” Price says. “If that person is infected with malware, they don’t have the ability to spread it to anyone else on the network.”

Redmond, Wash., network ­systems engineers Simrat Sekhon (left) and Keith Laycock say hyperconverged infrastructure has led to hands-off management. Photo: Rick Dahms.

Before moving to ACI, Durham’s IT team spent 80 percent of its time just keeping the data center up and running, leaving only 20 percent for improving services to citizens. Now those percentages are flipped, Network Services Manager Joel Bonestell says.

It’s difficult to put a solid number on the return on investment offered by automation, in part because the time required to perform each task can vary widely, says John Myatt, assistant director of information services and technology. But he sees other, tangible benefits in efficiency and employee retention.

“The bottom line is, I’m saving enough that I don’t need an extra body to do this work,” he says. “I’ve got someone who’s working less overtime and is more likely to stay here, so I don’t have to hire a new employee.”

Louisiana Makes Moves Toward Touchless Computing

The surge in size and complexity of IT environments, as well as growing interdependency between networks, make automation a necessity, says Eric Hanselman, chief analyst at 451 Research. It reduces human error and saves valuable time, he adds.

“One of the greatest benefits is reliability,” Hanselman says. “The whole point of software definition is that you don’t have to wait for physical changes to take place. You don’t have to touch the equipment; you don’t have to rack and stack. You just make those changes in software.”

The state of Louisiana began its move toward a software-defined data center in 2014 when the state legislature consolidated some 20 disparate state IT agencies into the Office of Technology Services (OTS).

The office’s first big project was working with the federal Centers for Medicare and Medicaid Services to refresh the Louisiana Department of Health’s aging medical eligibility system, says Michael Allison, state CTO. That quickly grew into a multiyear effort to overhaul the state’s two data centers in Baton Rouge, which is on track to be completed this summer.

“We could have gone with a traditional three-tier data center architecture, but our state CIO Dickie Howze and Deputy CIO Neal Underwood challenged us to get ahead of the technology curve,” Allison says.

Photo: Charles Harris.

To meet that challenge, OTS adopted a Dell and Nutanix hyperconverged infrastructure as a first step toward ­virtualizing the entire stack using VMware NSX. That allowed OTS to route traffic more efficiently between the state’s data centers, Allison says, while also leveraging automation to make the network more secure. Instead of setting up access control lists for every device on the network, for example, each one was tagged with specific rules dictating how it would communicate.

“Database servers can only talk to application servers. Application servers can only talk to our web servers. And our security partners, like Palo Alto Networks and McAfee, can honor the tags we defined with NSX and extend them to the network edges and gateways,” Allison says.

The trend toward containerized applications running on Docker, open source software, and serverless architecture has made automation a necessary step, Allison adds. And that, in turn, has allowed OTS to begin offering Infrastructure as a Service to its developers.

“They can begin to blur the lines between what a traditional developer used to do and what an infrastructure manager does,” he says. “They can write a Docker script, hit play and spin up their own database application and firewalls without having to leverage any infrastructure.”

Redmond, Wash., Enters Hyperconverged Hyperspace

In the summer of 2016, IT managers in the city of Redmond, Wash., faced a ­similar decision. The HPE and NetApp infrastructure in Redmond’s­ on-­premises data center had reached the end of its support cycle. The city could either upgrade to a newer appliance or make the switch to Nutanix hyperconverged infrastructure solutions.

For Network Systems Engineer Keith Laycock, the solution was a no-brainer. Having a single machine that contained networking, compute and storage made the Nutanix appliance much easier to manage and support. The device’s small footprint meant it consumed only about a third of the electricity of a traditional setup, saving the city a significant amount of money. And it was easy to scale: To increase network capacity, just add another network node.

But the key was simplicity, Network Systems Engineer Simrat Sekhon says: “Nutanix offers a single pane of glass into the network. I don’t need to be an expert on a range of technologies. We just have one management console, and when it’s configured, all we have to do is create new virtual machines as needed.”

Redmond deployed two clusters of Nutanix Acropolis and NX-8000 — a six-node cluster in its production environment, and a four-node cluster in a second data center used for disaster recovery — running roughly 120 virtual servers. Since the Nutanix machines were deployed, Laycock says his team has barely had to touch them.

That’s a far cry from the old data ­center ­infrastructure, which sometimes required eight hours a week just to manage storage. Upgrades happen almost automatically, Laycock adds.

“It’s a one-click upgrade,” he says. “Nutanix orchestrates the entire Nutanix AOS and VMware ESXi upgrade process. You can do it live with no VM guest server downtime.”

Automation Lets State and Local IT Do More with Less

As with most major technology shifts, the greatest challenge in automation often comes when trying to persuade people to approach their work in a new way.

“It’s a big cultural change for any organization, let alone a state organization,” Louisiana’s Allison says. “You run into a lot of, ‘If it ain’t broke, don’t fix it,’ and the tried-and-true ways of setting up services and servers. All of that had to change. We have to be able to use automation as a workforce multiplier and an accelerator.”

State and county employees who are under increasing pressure to do more with less will find that data center automation is one of the ways they can achieve that. “We are government; for us, that means we run a lean team but also need to be able to keep pace and leverage new technologies,” Price says. “This solution allows us to be more efficient, effective and, ever important today, more secure.”