Infrastructure Overhaul

Almost two years later, things have changed. "You walk in there now, you know you're in a data center," says Cross. He and his team built a highly available, redundant and power-saving storage infrastructure with help from electricians, consultants and vendors.

"We started with the infrastructure and did it all at once," says Cross. "I bought two of everything because I wanted redundancy."

He began by purchasing APC Smart-UPS VT 30 kVA battery boxes and APC InRow ACRD data center air coolers to replace the old, inefficient ones. "They ran full-blast constantly, whether it was 70 degrees or 100 degrees," says Cross. The air unit was 18 years old and pulled 23.6 amps continuously, he says. "The new units modulate from around 8 amps to a maximum of 16 amps, depending on temperature."

APC helped revamp the data center, drawing up a diagram that Cross and his team tweaked slightly before implementing. APC's power systems laid the groundwork for virtualized servers and an EMC CLARiiON CX4 120 Fibre Channel storage area network.

Wayne County previously used direct-attached storage. "If a server failed, all the applications on that would be down until we fixed it," Cross says. The new SAN provides up to 22 terabytes of data, he estimates. He and two senior engineers did a lot of reading and Internet research, visited other county IT departments, attended conferences and called in several vendors when researching which type of SAN to purchase. "We decided to go with Fibre Channel for performance, though it's more difficult to configure," he says.

Such a move is common, says Roger Cox, research vice president at Gartner. "It's a natural evolvement to move to Fibre Channel in the SAN world," Cox says, noting he sees a mix of SAN infrastructures based on Fibre Channel and iSCSI. iSCSI can be a natural evolvement too, he says, if an organization has fewer host systems. Fibre Channel SANs entail more complexity and a steeper learning curve, but the performance is often faster because the data isn't competing with other Ethernet traffic.

For backup, Wayne County deployed STORserver Spectra T120 tape/storage backup appliances in the data center. The appliances store all data to disk like a SAN, says Cross. "That's done every night, and if somebody loses a file we can restore it almost instantly." One backup appliance protects their main data center, while the other is a mile away at a county office building, serving as a disaster recovery site for disk backups.

Ensuring continuity of operations is typically a top priority for IT expenditures, notes Mark Peters, senior analyst at Enterprise Strategy Group. "For most people it comes down to what they can afford. For a state government, they're going to need to protect their data" because they are accountable to taxpayers. He adds that buying double the backup capacity and splitting it across sites is a good approach to disaster recovery.

In keeping with Cross's goal of redundancy, data also goes to HP 400Gb C7972A backup tapes each night, which the county stores in a vault in another facility. While tape is not the preferred backup method, Cross says it provides another level of protection. "You can back up a tape and take it anywhere."

Moving tape offsite for long-term storage is still common, says Gartner's Cox, though tape backups continue to decline as tape becomes more of a long-term archival solution.

Putting the Plan into Action

Cross drew on financial knowledge and IT experience gained from time in the Air Force to show senior leadership that the county's systems needed a serious investment. "I was a financial analyst, and I knew how to justify requirements," he says. "My boss, the county manager, knew the infrastructure needed to be updated." Virtualized servers helped close the deal when Cross showed how much the county would save on maintenance and replacement costs. "If we don't put the money up to do it now, it's going to cost more later," says Cross. "That's what I had to show."

"Senior management was impressed with what they saw, especially regarding the ROI," he continues. "Now when my boss comes in, he sees a completely different data center. It's structured, it's got the latest technology, and it's very efficient and redundant."

With backup power and virtualization ensuring high availability, the county has not experienced any downtime since implementing the new infrastructure. "Even when we've had a power outage, our system kicked in like it was supposed to," Cross says.

Future Steps

Next up for Wayne County is desktop virtualization and possibly data deduplication, Cross predicts. "We may do it in the future, but we're not really hurting on disk space because of our SAN," he says. Dedupe would give the county about 30 percent more storage than it currently has, for about the same cost as 7TB to 10TB of disk space.

The Wayne County IT team is also working on a SAN project that will store virtual server backups in another area for continuity of operations. Cross says that protecting taxpayer data is crucial. "All of that data, tax information, financial information, all of it goes to our systems. It's critical data," he says. "If you don't have an infrastructure to support that, it could be catastrophic."