Stephen Cross, IT director for Wayne County, N.C., knew he had work to do when he first saw the county's data center. "It looked like a storage closet," he says of the filing cabinets and equipment cluttering the space, along with badly placed wires and plugs. "Our infrastructure was extremely inefficient, to put it mildly."
Almost two years later, things have changed. "You walk in there now, you know you're in a data center," says Cross. He and his team built a highly available, redundant and power-saving storage infrastructure with help from electricians, consultants and vendors.
"We started with the infrastructure and did it all at once," says Cross. "I bought two of everything because I wanted redundancy."
He began by purchasing APC Smart-UPS VT 30 kVA battery boxes and APC InRow ACRD data center air coolers to replace the old, inefficient ones. "They ran full-blast constantly, whether it was 70 degrees or 100 degrees," says Cross. The air unit was 18 years old and pulled 23.6 amps continuously, he says. "The new units modulate from around 8 amps to a maximum of 16 amps, depending on temperature."
APC helped revamp the data center, drawing up a diagram that Cross and his team tweaked slightly before implementing. APC's power systems laid the groundwork for virtualized servers and an EMC CLARiiON CX4 120 Fibre Channel storage area network.
Wayne County previously used direct-attached storage. "If a server failed, all the applications on that would be down until we fixed it," Cross says. The new SAN provides up to 22 terabytes of data, he estimates. He and two senior engineers did a lot of reading and Internet research, visited other county IT departments, attended conferences and called in several vendors when researching which type of SAN to purchase. "We decided to go with Fibre Channel for performance, though it's more difficult to configure," he says.
Such a move is common, says Roger Cox, research vice president at Gartner. "It's a natural evolvement to move to Fibre Channel in the SAN world," Cox says, noting he sees a mix of SAN infrastructures based on Fibre Channel and iSCSI. iSCSI can be a natural evolvement too, he says, if an organization has fewer host systems. Fibre Channel SANs entail more complexity and a steeper learning curve, but the performance is often faster because the data isn't competing with other Ethernet traffic.
For backup, Wayne County deployed STORserver Spectra T120 tape/storage backup appliances in the data center. The appliances store all data to disk like a SAN, says Cross. "That's done every night, and if somebody loses a file we can restore it almost instantly." One backup appliance protects their main data center, while the other is a mile away at a county office building, serving as a disaster recovery site for disk backups.
Total number of users in the 25 departments Wayne County IT supports
Ensuring continuity of operations is typically a top priority for IT expenditures, notes Mark Peters, senior analyst at Enterprise Strategy Group. "For most people it comes down to what they can afford. For a state government, they're going to need to protect their data" because they are accountable to taxpayers. He adds that buying double the backup capacity and splitting it across sites is a good approach to disaster recovery.
In keeping with Cross's goal of redundancy, data also goes to HP 400Gb C7972A backup tapes each night, which the county stores in a vault in another facility. While tape is not the preferred backup method, Cross says it provides another level of protection. "You can back up a tape and take it anywhere."
Moving tape offsite for long-term storage is still common, says Gartner's Cox, though tape backups continue to decline as tape becomes more of a long-term archival solution.
Putting the Plan into Action
Cross drew on financial knowledge and IT experience gained from time in the Air Force to show senior leadership that the county's systems needed a serious investment. "I was a financial analyst, and I knew how to justify requirements," he says. "My boss, the county manager, knew the infrastructure needed to be updated." Virtualized servers helped close the deal when Cross showed how much the county would save on maintenance and replacement costs. "If we don't put the money up to do it now, it's going to cost more later," says Cross. "That's what I had to show."
"Senior management was impressed with what they saw, especially regarding the ROI," he continues. "Now when my boss comes in, he sees a completely different data center. It's structured, it's got the latest technology, and it's very efficient and redundant."
With backup power and virtualization ensuring high availability, the county has not experienced any downtime since implementing the new infrastructure. "Even when we've had a power outage, our system kicked in like it was supposed to," Cross says.
Next up for Wayne County is desktop virtualization and possibly data deduplication, Cross predicts. "We may do it in the future, but we're not really hurting on disk space because of our SAN," he says. Dedupe would give the county about 30 percent more storage than it currently has, for about the same cost as 7TB to 10TB of disk space.
The Wayne County IT team is also working on a SAN project that will store virtual server backups in another area for continuity of operations. Cross says that protecting taxpayer data is crucial. "All of that data, tax information, financial information, all of it goes to our systems. It's critical data," he says. "If you don't have an infrastructure to support that, it could be catastrophic."
Powering the Data Center
Before Wayne County could upgrade its storage, it had to tackle the nuts and bolts of the electrical and cooling systems. The old air-cooling unit ran at full blast constantly, says IT Director Stephen Cross, and the backup generator lacked the power that IT needed for the overhaul.
Cross liked the APC Smart-UPS VT 30 kVA's battery backup, but learned it would exceed the capacity of the county's existing generator. Rather than upgrading to a new generator, he consulted with an electrical engineer. He upgraded the server room to 480-volt power, allowing him to deploy the 480-volt APC boxes, which pull far fewer amps of electricity from the generator. "By upgrading the electrical system and upgrading to the 480-volt, we were able to get everything on the backup generator without having to upgrade it," he says.
Wayne County also rebuilt its old air cooler and added two new units during the upgrade.
"It was $19,000 to upgrade the electrical system," says Cross, while upgrading to the 480-volt APC units was an extra $10,000. "A new generator would've cost us at least $100,000 or $120,000, if not more."