How Next-Generation Technology Is Solving Agencies’ Capacity Problems

Most people starting a new job get a honeymoon phase to ease into the position and acclimate to new coworkers and responsibilities. But on his first day as IT director for the Central Ohio Transit Authority (COTA), Shawn Prince went into full crisis mode.

"I hit 'start' and 'my computer,' and it showed that all the shared drives were red," indicating they were nearly full, Prince recalls. "So I went to the technical staff and said, 'We clearly have storage issues. I need to know how bad.' "

The storage challenges were significant. The regional transit authority in Columbus, Ohio, relied mostly on server storage and a few small, workgroup-class storage area networks (SANs), which were nearly full and not scalable or interconnected. Prince asked COTA's leaders to commit resources for a new, enterprise-class server and storage solution.

Today, almost two years later, the data center is running smoothly and server and storage resources are plentiful, thanks to a converged FlexPod solution featuring Cisco System servers and network gear, a NetApp SAN and VMware virtualization software.

As governments modernize their data centers, they've begun to shift away from a traditional IT infrastructure that has separate systems for servers, storage and the network. Instead, they're deploying converged systems (which engineer separate components to work as an integrated unit), or hyperconverged systems (which combine server and storage into single appliances).

These integrated systems are easier to deploy and manage, support server and storage virtualization and consolidation to save money and space, and allow agencies to operate more effectively and efficiently.

"For some, this convergence is new and revolutionary. For others, it's déjà vu as to how you used to get your systems back when they were turnkey and bundled together," says Greg Schulz, founder of the StorageIO Group, an industry consultancy. The systems offer the latest storage features, including solid-state drives (SSDs), which provide faster data access; deduplication and compression, which eliminates redundancies and stores data using less disk space; and thin provisioning, which better utilizes available storage.

Roadmap to Better Storage

COTA provides 18.7 million rides a year in Franklin County and parts of four neighboring Ohio counties. Its two main applications that run the bus system are critical — one tracks bus location and provides radio communications between dispatchers and bus drivers, while the other is an asset management system used 24/7 to log bus maintenance and warranty repairs.

If storage actually reached full capacity, the data center would crash. "Everything would come to a screeching halt, and we'd suffer data loss," says Prince, whose IT staff avoided a full meltdown by keeping a close watch on storage limits and deleting unnecessary files.

When Prince joined COTA in late 2012, the transit authority had virtualized 40 percent of its servers, but couldn't virtualize any more because of the limitations of its aging servers and hodgepodge of storage devices. "The existing virtual farm was full, and the SANs were not scalable," he says.

Prince received funding for FlexPod when COTA's leadership bought into his vision of a comprehensive enterprise solution to centralize storage, which would pave the way for the IT department to virtualize 95 percent of its servers and strengthen the organization's disaster recovery strategy.

The IT staff began the installation in November 2013 and spent about five months deploying the FlexPod system at its main data center and a backup data center.

Both locations house six Cisco UCS B200 blade servers, a clustered NetApp FAS3200 Series SAN with two controllers for redundancy, VMware vSphere software and dual Nexus 5000 Series switches.

Each hybrid NetApp SAN controller offers 512 gigabytes of solid-state storage, providing fast access to important or frequently accessed data. The remaining data is housed in hard disk storage.

Improved storage performance enables COTA's workforce to be more productive.

"Our business units have told us that life is so much easier now that they can access files so much faster," Prince says.

Meanwhile, Prince continues to make storage improvements. The SAN currently houses 50 terabytes of usable storage, but he says that number will increase when the IT staff deploys data deduplication.

Hybrid Cloud Approach

In Washington state, King County historically took a decentralized approach to IT, allowing individual departments to manage their own IT infrastructure. But to reduce redundancies and cut costs, in 2011 the county consolidated IT services and staff into one service-based IT organization. And in 2013, the county built a private cloud infrastructure.

"The more decentralized you are, the more expensive things are. For the cloud, we wanted to standardize and make it more of an enterprise offering," says Bill Kehoe, CIO for King County.

Kehoe and his staff deployed a converged FlexPod architecture of Cisco UCS B200 M3 blade servers, Cisco Nexus 5596 series switches, a NetApp FAS8040 SAN and Microsoft Hyper-V virtualization software. In doing so, they consolidated 28 data centers down to two, saving $700,000 a year.

The NetApp SAN supports multitenant environments, allowing individual county departments to have an isolated portion of the SAN that they have full control over. Recently, one department needed more storage and would have previously had to spend $150,000 for new hardware. Instead, the county simply provisioned extra pooled storage to that department, Kehoe says.

To maximize storage utilization, IT administrators take advantage of deduplication and thin provisioning, which allows the IT staff to allocate a large amount of virtual storage for applications or users, but only use the amount of storage that's required at the moment.

Photo: Rick Dahms

King County, Wash., built its own private cloud and is exploring using a cloud service for disaster recovery, say Bill Kehoe (right) and Bob Micielli.

King County isn't done maximizing its resources. The NetApp SAN features native Server Message Block/Common Internet File System support, which means the SAN can be used to provide shared access to files. The county plans to eliminate most of its 37 Windows file servers, which can save $110,000 a year, says Bob Micielli, King County's senior manager of enterprise technology services.

Moving forward, the county wants to further reduce storage costs and is considering using cloud providers for data backup, archiving, disaster recovery and other services. "We're doing proof-of-concepts to examine the cost model and see how they function," Micielli says.

For example, the county can store a copy of its virtual machines and data backups in the cloud and keep services running if the main data center goes down. "In the past, it would cost millions of dollars to create a whole new backup data center and staff it," Kehoe says. "The virtual servers would be dormant in the cloud. We are not paying for compute power, just for storage. If something happens, we can enable the virtual machines and turn them on in the cloud.

Embracing Hybrid Storage

Two cities — Del Mar, Calif., and Conroe, Texas — standardized on Nimble Storage, which provides a hybrid storage array that features SSDs and hard disk drives.

In 2011, Del Mar had eight physical servers with direct-attached storage that was starting to run out of space, and a disaster recovery strategy that relied on tape. That year, Emily Bernardo, the city's computer operations manager, began a virtualization and storage consolidation project. She purchased an 8TB Nimble CS210 storage device, paired it with two new servers and virtualized applications using Citrix XenServer software.

The CS210 features two mirrored controllers so if one controller fails, the other keeps the storage system running. "We have complete hardware redundancy within the box," Bernardo says. "Performance is great. It's rock solid."

Elsewhere, Damien Tijerina, network services supervisor for Conroe, was in the midst of virtualizing servers when he needed faster storage I/O performance.

In the first phase of virtualization, he deployed VMware, four blade servers and a midrange SAN. But last year when he was ready to virtualize mission-critical applications, including SQL Server databases and Exchange email, the existing SAN's I/O performance wasn't fast enough.

Nimble Storage's hybrid storage arrays met Tijerina's speed requirements. He bought two Nimble CS220s and one Nimble CS240G for City Hall, the police department and a disaster recovery site. With the new storage in place, the city has now virtualized 95 to 97 percent of its servers.

"With Nimble, we get the best of both worlds," Tijerina says. "We could use solid state for the important workloads and the spinning disks for workloads that don't require fast speeds."

City employees rave about the speed of applications. The finance department can now process payroll in 20 to 40 minutes, whereas it used to take three hours.