IT's Post-Katrina Scramble

WHEN HURRICANE KATRINA barreled through the Gulf Coast last August, it exacted a toll that nobody could have predicted. Yet, behind the headlines of lives lost and property destroyed, another enormous problem emerged: Government agencies and citizens needed access to information in order to put their lives back together.

“Getting information back online and available was a top priority,” says Aaron Davis, the IT director for the Louisiana Fourth Circuit Court of Appeals in New Orleans.

But with offices flooded, computers destroyed and roads nearly impassable, getting systems back up and running was a daunting task. “Even with a disaster recovery strategy in place and a business continuity plan in order, there was no way to fully prepare for the devastation and resulting problems,” Davis says.

A Taxing Situation

Disaster recovery. These days, the words send a chill down the spine of every IT director. And with budgets tight and resources scarce, planning for a hurricane, earthquake, flood or other disaster can tax a state or local agency to the limit.

“Events are highly unpredictable, but it is possible to put systems in place to deal with situations as they arise,” says Deb Mielke, the managing director of Treillage Network Strategies, a McKinney, Texas, consulting firm.

Mielke says it’s nearly impossible to plan for 100 percent of the possible outcomes. “Financially, you can protect yourself only so much,” she says. “So, it’s wise to prepare for 75 percent of all possible scenarios, but know what to do if something falls outside the known boundaries.”

Shawn McCarthy, program manager for government and education at IDC, a Framingham, Mass.-based market research firm, adds, “The biggest problems result when there’s no detailed emergency plan to allow for quick relocation of any government function.”

Winds of Change

In today’s technology-centric world, the fallout from a computer disruption can be far-reaching. Government agencies that aren’t online can’t pay employees, provide essential services to citizens — including the distribution of food stamps and welfare payments — and aren’t able to ensure public safety.

What’s more, the potential loss of records and data could prove devastating and could leave tax collection agencies, motor vehicle departments and courts in shambles.

This is a scenario that Davis knows about firsthand. When Hurricane Katrina took aim at Louisiana, he began disaster recovery operations before the storm hit. Davis and his IT staff ran a full backup on Aug. 26 — about three days before the eye of the storm made landfall in Louisiana.

On Aug. 28, Davis tried to go to the court offices to shut down 16 servers and about 100 workstations, but Katrina was already near full force, so he wound up shutting down the servers remotely.

When the hurricane struck, all communication stopped for nearly a week, and it wasn’t possible to get into New Orleans.

Finally, on Sept. 9, Davis joined an armed convoy that left from Baton Rouge. He borrowed a truck from his relatives and, along with the director of human resources, hauled servers, computers and other gear out of the third-floor offices at the court’s building in downtown New Orleans.

“We loaded the truck with the most important servers and headed to Hammond,” recalls Davis, who found space at an old, unused police station on the campus of Southeastern Louisiana University in Hammond, La., about 60 miles from New Orleans. “We set up a small temporary office for human resources, so we could continue to pay bills and issue checks. We had to keep basic operations going.”

During the next few weeks, Davis and a network administrator put desks, chairs, cables, phones and a T1 line in place. Using Citrix NFuse remote access software — which was already in place before the hurricane — employees could access files as well as 10 years’ worth of court records.

Meanwhile, Davis and his IT staff members began getting systems in New Orleans back up to speed, including malfunctioning T1 lines, discharged uninterruptible power supply systems and failed hard drives.

On Oct. 31, they were informed that the courthouse in New Orleans would reopen on a limited basis on Nov. 7. Half of the staff could move back in, but the building would remain closed to the public, so the temporary office in Hammond remained open for filings.

On Nov. 3 and 4, the IT department began the arduous task of moving all of the main servers and half of the computers back to the New Orleans location. They spent those two days testing the servers to ensure that remote access was working and would allow the Hammond office to remain functional. Throughout the ordeal, “the court didn’t lose any documents or data, and we had things running without any unnecessary delays,” Davis says.

The courthouse officially reopened the first week in December, more than three months after the hurricane struck.

The Calm After the Storm

The Fourth Circuit Court of Appeals wasn’t the only agency to take quick action during Hurricane Katrina. In Mississippi, the Department of Information Technology Services had the onerous task of restoring telecommunications for approximately 350 of the state’s 2,800 offices and schools that were without communication services the morning after the storm.

Mississippi’s mission-critical systems and databases are centralized, and, aside from extended power outages, systems in Jackson, Miss., were never threatened by the storm. However, some coastal offices in the southern counties of Hancock, Harrison and Jackson were destroyed, and equipment was permanently damaged or lost. In addition, 17 schools were destroyed, and 34 were severely damaged.

“Buildings were no longer there, and equipment was washed away,” recalls David Litchliter, executive director for the Department of Information Technology Services. “Making sure we could distribute food stamps and other human services was essential.”

Despite developing a disaster plan beforehand, Litchliter says that there was “no way we could anticipate all the problems and challenges.” Besides setting up temporary offices in the field with local area networks and computers, the state established call centers for service-providing agencies to handle the high volume of calls.

At the same time, state government officials used geographic information system (GIS) technology to obtain coordinates by geo-coding street addresses. “Since entire streets and addresses no longer existed, we needed a way to find people stuck in trees and on roofs,” Litchliter explains.

“We had family members calling in and people using cell phones to report lost loved ones. The GIS made it possible for the U.S. Coast Guard to dispatch helicopters and find them.”

In fact, the coordinates that were provided by the GIS technology helped the U.S. Coast Guard lift more than 1,700 people to safety.

Dealing With the Unexpected

Treillage’s Mielke says that effective disaster planning involves far more than simply backing up systems and data. It’s essential to keep critical government operations, including 911 systems, functioning and to understand the basic processes and relationships that underpin effective government.

“If you lose all your equipment, you must have a way to buy or lease new equipment quickly,” Mielke points out. “It doesn’t do any good to have data that you can’t reach.

“It’s essential to have partners and agreements in place and to have ways to deal with the unexpected.”