Cities Finding iSCSI Connectivity Invaluable

In a sense, it takes a village to make a city’s information technology. A village replete with small “shops,” such as tax collection, emergency services, building inspection and sanitation — all having their own applications, data types and servers.

Responsibility for managing different applications often rests with the individual departments. But not so the data associated with those applications. When it comes to storage capacity planning, and backup and recovery, many departments, although working in a distributed computing architecture, turn to the centralized IT department. The problems associated with that paradox have caused more than one IT professional to keep bottles of aspirin handy.

“All the different data types, different servers, and the growing size of our data storage had made our systems very hard to administer,” says Jay Brummett, chief technology officer for Ogden City, Utah.

Like most small- and medium-sized organizations, Ogden City went with storage systems directly connected to individual servers — direct-attached storage (DAS). But managing that type of network can be costly and inefficient. “When I first arrived at Ogden City [about three years ago] we were several million dollars in the red. The balance sheet was a disaster. We had to find a way to do data consolidation so we could reduce the amount of staff we needed for systems administration,” says Brummett.

Ogden City shifted to a storage area network (SAN), a central storage system that maintains data for all servers, no matter where they are located or what kind of data they store.

The ISCSI Advantage

Until about three years ago, virtually all SANs relied on Fibre Channel, a mature high-performance transport architecture, but one which requires new, often expensive, network devices and specialized skills.

Fortunately, for those seeking a way out of the complexities of administering distributed storage, the new Internet Small Computer System Interface (iSCSI) has emerged as an inexpensive and more familiar transport architecture. Using existing Internet Protocol Ethernet network devices and generally eliminating the learning curve associated with a new network protocol, iSCSI can reduce both cost and time required to implement a SAN.

In 2003, about the time Brummett was becoming aware that he needed the centralized benefits of a SAN, the final iSCSI industry draft was just being ratified.

Being on the leading edge of a new technology is often rewarding, but also risky. So Brummett didn’t accept iSCSI without a lot of careful considering. “We did our own analysis on the technology and an evaluation of the vendor [LeftHand Networks]. We decided the technology was sufficiently stable and any potential downsides — and we didn’t really expect any — would be countered by the overall cost savings.”

Brummett’s initial SAN implementation of 1 terabyte of data also moved quickly. “It took us about four hours to complete the installation of the SAN. And that includes cutting the tape to open the three cartons, racking the units and setting up and activating the first volume out from a server,” Brummett says.

The cost of administering data storage at Ogden City plummeted from $1.67 per megabyte to less than 25 cents per megabyte. Now approaching the three-year anniversary of the SAN implementation at Ogden City, the system is smoothly handling 12 terabytes of data located in one data center and a disaster-recovery center.

Differences Narrowing

When the iSCSI standard was first ratified, many people looked at it as a scaled-down cousin to Fibre Channel, suitable only for smaller operations. But over the last three years, iSCSI’s disadvantages relative to Fibre Channel — both actual and perceived — have been narrowing, according to Mark Bowker, an analyst with the tech analyst firm Enterprise Strategy Group (ESG).

Fibre Channel’s seniority allowed it to enjoy more vendor options for plug and play SANs, says Bowker. But increasingly, vendors have been moving to support iSCSI as well. An important indicator of growing acceptance of iSCSI is the operator system support it has garnered. Shortly after the iSCSI standard was ratified, iSCSI initiator software either came with or was offered as a free download for AIX, HP-UX, NetWare, and Windows. About a year ago, Linux and Solaris joined the bandwagon. And although Apple is still a laggard in this regard, recently a few third-party companies, such as ATTO Technology, provided it for Mac OS X.

Fibre Channel does have a theoretical, and probably temporary, performance advantage over iSCSI. Fibre Channel specifies throughput speeds of 2 or 4 gigabits per second compared to iSCSI’s 1Gbps rate. But as many people say about age, throughput is just a number and not always a very meaningful one. Says Bowker, “Everyone gets caught up in benchmarks because it’s an easy-to-understand and clear-cut way to compare architectures. But in the real world, very few people are even using 1Gbps of throughput.”

But Bowker does admit that users of some very interactive applications may notice a performance difference between 1 and 2Gbps. But he expects that within a few years iSCSI network interface cards will reach and possibly leapfrog over Fibre Channel speeds. And in any case, organizations can boost their iSCSI throughput now by trunking multiple NICs together.

As a result of this, Bowker says, “Unless you already have a heavy investment in Fibre Channel and in-house expertise to administer it, there’s usually no reason to move in that direction.”

Even organizations that have an installed Fibre Channel SAN often complement it with iSCSI for some applications, often those which involve moving data across long distances. Says Karl Chen, vice president of business development at LeftHand Networks, “iSCSI is native IP, so there are no distance limits with IP. Fibre Channel is limited typically by the physical media properties to 10 kilometers without specialized switches and converters.”

Although easier implementation, lower cost and a reduced learning curve may make iSCSI more attractive than Fibre Channel, the existence of the new architecture in itself is not a good enough reason to install a SAN.

“No one should take on a SAN project — whether iSCSI or Fibre Channel — unless they have sufficient pain points to justify it,” says Bowker.

In fact, most cities and states first determine if they actually need a SAN. Only after making that decision do they turn their attention to whether Fibre Channel or iSCSI is best for them.

Easier Capacity Planning

For example, Utah’s Park City Muncipal Corp. was having a severe problem with planning capacity for its DAS storage systems. “We had some storage systems using only 15 or 20 percent of their capacity, and in other servers, we were constantly adding storage,” says Jeff Drury, Park City’s systems administrator. Park City’s IT sometimes shuffled applications among the servers; other times they added storage.

The server-by-server administration not only added workload to IT, it precluded Park City from instituting a failover system.

Drury understood that a SAN-based architecture could solve the storage problems. At first, he considered using Fibre Channel because it was a more mature technology. But when even Fibre Channel vendors admitted to him that the technology entails a steep learning curve, he began to worry. The fact was that Drury didn’t have much time for detailed training sessions: He was given only about 45 days to implement both the SAN and move his organization from Novell to Microsoft Exchange.

“I was skeptical that we would be able to meet that goal at all. Add the complexity of using a technology that we have no experience with and we’d definitely be in trouble with the deadline,” he says.

Smooth Rollout

As it turned out, because Park City was able to use its existing IP infrastructure, implementation of the SAN had very little impact on the timetable for the messaging conversion project, which was completed in the specified month and a half. The SAN implementation took around six hours, and the only time-consuming job, file transfer of data to the SAN, was accomplished over a few nights. “There was no downtime. Most users didn’t even know we had moved to a SAN,” Drury says.

Using the existing IP Ethernet infrastructure provided another benefit to Park City. Data could easily be sent to any building without adding wiring. Drury took advantage of that by installing four SANs, with four disks each, in two buildings. Because of the way data is spread over the primary and backup centers, Park City can lose any two of its SANs without interrupting availability of data to users.

Like Drury, many IT professionals are unwilling to install a new technology like iSCSI until they see for themselves that it will handle their workflow. This is especially true if the organization already has a successful Fibre Channel SAN in place. Accordingly, ESG’s Bowker recommends running iSCSI in a test environment for a month or two to work out any kinks in the implementation and to prove the concept to any in-house iSCSI skeptics. The technology is still new enough, and Fibre Channel supporters still have sufficient political clout to justify careful testing before implementation, Bowker says.

Although iSCSI may overtake Fibre Channel as a transport architecture for SANs, Ogden City’s Brummett believes Fibre Channel technology has at least one thing to teach people configuring iSCSI SANs: Keep the SAN data separate from other data. Brummett deployed his iSCSI SAN on a separate Ethernet segment, segregating SAN traffic from other data on Ogden City’s common IP network. A major advantage of this process is that implementation of the SAN did not require any network downtime.

As a result, the process of augmenting storage capacity at Ogden City has no impact on users. “We add storage without any network or server disruption. We simply rack a new box and add it to one of our existing SAN clusters,” Brummett says.

Brummett adds that he has had both drives and whole array units fail. But he never experienced the kind of failure penalty such as latency that he had to suffer through with traditional redundant array of independent disks failures. “And when the component is replaced, the process also does not adversely affect a user’s perception of performance,” Brummett says.