Conquering Storage Woes

When the village of Skokie, Ill., purchased a storage area network this summer, MIS Director Bryan Gilley took care of the village's data storage, disaster recovery and green computing needs in one fell swoop.

Before the SAN, Skokie's IT department faced the same maddening problems with direct attached storage that other government IT leaders have encountered. Some server hard drives were underutilized, wasting storage space. Other servers were crammed to near capacity, forcing Gilley to buy new boxes with bigger hard drives. Or to save money, he and his staff would work after hours to add additional drives and restripe the disk arrays on existing servers -- a tedious, time-consuming task that was prone to errors.

Now, with data consolidated and centralized into a SAN, Gilley can easily plug in more disks when needed. Because the SAN takes snapshots and replicates data across multiple data centers, continuity of operations is improved. And through improved disk utilization, the SAN uses less power and cooling, saving money.

"The features and functionality that the SAN provides us are remarkable," Gilley says. "When our applications need additional storage, we can easily allocate additional disk space. In a disaster recovery situation, it cuts down our restore window from days to hours, if not minutes. And with our village's green initiative, getting greener is a big bonus."

To combat skyrocketing data growth, many public-sector organizations like Skokie have turned to networked storage -- such as block-level SANs and file-level network-attached storage (NAS) -- to simplify data management and improve disaster recovery. IT administrators are tapping technologies such as thin provisioning and data deduplication to improve disk utilization and maximize performance, and they're using best practices such as tiered storage to keep costs down.

"Organizations may have started with disks dedicated on servers, but as they grow larger, they discover that's ineffective," says Gene Ruth, a Burton Group storage analyst. "It's not a big step to realize that if they put storage in a shared environment, it's more effective and a natural evolution."

Ready and Redundant

When a handful of Skokie's servers reached retirement age in 2009, Gilley used it as an opportunity to overhaul his data center architecture by virtualizing servers with VMware and centralizing storage with an HP LeftHand P4300 iSCSI SAN. In doing so, he improved continuity of operations.

When Skokie was hit by a massive storm that flooded part of the village, it made Gilley realize that he needed to significantly improve disaster recovery readiness in case a storm knocked out power or flooded the data center. At the time, he had about 20 standalone servers with no redundancy. The data on the servers were backed up to a disk array and then to tape. But if a server went down, he'd have to buy a new server, reinstall the application and restore the data from tape backup, a multihour process. If the entire data center were damaged, it would take days to get the environment running again.

To improve reliability, Skokie's IT department deployed three new HP ProLiant DL380 servers and three HP LeftHand SAN nodes that feature 2 terabytes of usable storage. Using virtualization, the IT department consolidated the standalone servers and moved the applications onto two DL380 servers in the primary data center. So far, they have moved 1.1TB of the village's data to the SAN, and the data replicates between the two nodes. To further boost redundancy, the staff installed the third server and SAN node at a secondary data center, which stores snapshot copies of the virtual servers and data. If the primary site goes down, the staff just fires up the equipment at the secondary site.

"Centralized storage is fantastic for disaster recovery," Gilley says. "With this technology, we can be back up in minutes, and at most, a couple of hours. It gives us the redundancy that we need."

Skokie's IT department recently purchased a fourth server and SAN node. Last fall, staff planned to install the additional equipment in the secondary data center to provide two-way data replication between sites. If one site goes down, the other will automatically keep IT operations running.

Selecting a SAN

Gilley briefly considered a Fibre Channel SAN, but went with an affordable iSCSI SAN because 1-gigabyte speeds are fast enough for his needs, and he could use the existing Ethernet network, rather than have to build a new, expensive Fibre Channel network.

Larger cities, counties and states may acquire multiple network storage solutions for different needs. Albuquerque, for example, standardized on an 80TB IBM System Storage DS4800 Fibre Channel SAN two years ago for e-mail and other critical applications, says systems programmer Larry Vasey.

The city's other SANs include a LeftHand iSCSI SAN for disaster recovery and virtualization testing, and EMC Clariion storage for various databases, he says. The city recently purchased an IBM System Storage DS3200 NAS device for a police application that allows for secure wireless network communications.

Albuquerque purchased each network storage solution based on the city's needs and budget at the time and whether it was needed for departmental or citywide tasks, explains systems programming manager Dan T. Jones. Vasey chose a full IBM solution that included a Fibre Channel SAN, blade servers and Tivoli Storage Manager for tape backup because it provided the necessary speed and reliability, along with the assurance that they work seamlessly together, he says.

"We wanted a single point of contact for any problems that might arise," Vasey explains.

While iSCSI SANs offer competitive speeds today, Fibre Channel at the time was much faster. Putting the SAN on its own Fibre Channel network isolated the data from security risks, such as denial-of-service attacks on the Ethernet network, Jones adds. "It's in a closed architecture, and that protects the infrastructure," he says.

Simplified Storage Management

Regardless of the specific solution, IT administrators rave over networked storage because it eases management and improves storage utilization.

With direct attached storage, having to add storage is a hassle. So IT departments historically have purchased servers with what they believed to be enough drive space for the usable life of the servers. But that wasted resources. "You might buy storage and initially use 15 to 20 percent of it, and save 80 to 85 percent for future growth," says Ken Phillips, systems administrator for the city of Atascadero, Calif., who switched to a 3TB SAN two years ago. "But that means you're powering those drives, and paying for those drives up front."

If the drives never fill up, the storage is wasted, says Skokie Network Engineer Nooruddin Tharwani. Worse yet, if a server needs more storage, IT would have to bring down the server to add disks, resulting in downtime. In contrast, with SANs, IT staffers can provision more storage to servers immediately with a few mouse clicks. "With resources centralized, we can assign more disk space on the fly," Tharwani says.

Through SAN technology called thin provisioning, IT administrators can better utilize available storage by allocating the exact amount of storage space a server needs at any given time. The result is better storage utilization and reduced energy consumption because a large amount of storage space is not sitting unused, Phillips says.

SANs also make it easier and faster to recover files. Periodically, Skokie Network Administrator Iqbal Kalota receives calls from users who've accidentally deleted important files. "It used to take a couple of hours looking for it on a backup tape, longer if it was stored offsite. Now, using a snapshot, I can get it in minutes," Kalota says.

Measuring ROI

In Santa Clara County, Calif., CIO Joyce Wing has reaped many benefits since she installed an EMC Clariion CX3 SAN two years ago. Before, 10 systems administrators were needed to manage the direct attached storage. Today, because SANs are simpler to manage, the county needs less than one full-time storage administrator.

The SAN has also improved storage utilization. With direct attached storage, more than 40 percent of drive space was unused. With a SAN, it's between 10 to 20 percent. Continuity of operations capability has also improved: Wing's Recovery Point Objective has reduced from 24 hours to minutes, and her Recovery Time Objective has been cut from days or weeks to hours.

Elsewhere, North Carolina's Catawba County in 2008 purchased HP blade servers and an EMC Centera SAN that has since grown from 4.5TB to 30TB. CIO Terry Bledsoe estimates that his switch to virtualization and SAN storage will save the county $300,000 over the next three years from hardware costs alone. Through the hardware consolidation, he's even turned 300 square feet of the data center into office space.

The benefits of the technology are clear-cut, says Skokie's Gilley.

"From an efficiency, redundancy and disaster recovery standpoint -- along with the fact that the data center becomes greener -- it's just the way to go," he says.