A Decade After Katrina, IT Leaders Share Lessons Learned
When the next hurricane strikes, the Louisiana Supreme Court is ready.
CIO Mike Evanson was already impressed with the court’s disaster recovery plan when he joined the court three years ago, and it has since grown stronger. The 15-page document outlines the plan at a high level, spells out what’s most important and has a timeline for when things are expected to happen.
“It’s short and to the point,” says Evanson. “It doesn’t have volumes of detail like others that are never really looked at.”
But Evanson’s confidence waivers in one area. “Like most people in Louisiana, we always think about hurricanes, but they’re not the biggest risk,” he explains, noting that hurricanes typically provide some warning.
“I’m not sure that we’re adequately prepared if we have no warning. What if everything was suddenly gone? Then what?”
As the 10th anniversary of Katrina approaches, many are reflecting on the disaster recovery best practices derived from one of the costliest and deadliest hurricanes to hit the nation. The coastal cities and towns in Alabama, Louisiana and Mississippi have learned that continuity of operations planning isn’t a static project with a beginning and an end, but rather a process that evolves alongside changing technologies, new threats and fresh insights.
“Ten years ago was a very different world in IT,” says Evanson. He and his staff update their disaster recovery plan annually before hurricane season and test it quarterly. They’ve also recently begun to walk through other scenarios, such as a fire, to determine what would be different.
For instance, the court has a backup data center, but failover isn’t automatic. So if employees have to evacuate the office and leave everything behind, they need backup equipment to initiate the failover.
“We have to think through who’s going to do what and how they will do it,” Evanson says. “If they don’t have access to what’s in the office, what do they need to meet their responsibilities?”
Who, What, Where?
Much has changed in the decade since Hurricane Katrina’s gusts and waves toppled levees, flooded buildings and crippled utilities. Back then, the Louisiana Supreme Court didn’t have a remote backup site. It had been in the process of developing a disaster recovery plan that included one, but the storm hit before the plan got off the ground.
The Supreme Court building in New Orleans’ French Quarter kept its systems going after the storm with a backup generator, but once that ran out of fuel, the data center went dark, according to Frank DiFulco, court management information systems director.
Court security staff remained in the building to ensure the courthouse’s security. At the direction of the Court, key staff members were contacted so critical data systems could be relocated from New Orleans to Baton Rouge. Staff members available at that time moved equipment to the new temporary space.
“We set up a small-scale office rather quickly,” DiFulco recalls. But it was a challenge: Not only did they have to find office space, furniture and equipment in record time, but they were working with a bare-bones crew. With cellphones and the Internet down, it was difficult to tell employees where to report. Many couldn’t make it in anyway because of the devastation in their personal lives. The Louisiana Supreme Court now has a comprehensive disaster recovery plan complete with a backup site in Shreveport. Evanson recently moved it from a remote northern part of the state because he wanted it in a metropolitan area with a solid infrastructure.
“Two things you really need to run a data center are reliable power and communications, and I wasn’t real comfortable with either of those,” Evanson explains. “Now we’re in very good shape. We’re on a very reliable power grid, served by multiple utilities, and we have very strong, very adequate data communications provided by multiple vendors.”
In addition to relocating the backup facility last year, the court also upgraded its network infrastructure. Evanson’s team installed new servers, a storage area network and communications equipment, and it increased the data center’s capacity. The court uses Cisco Catalyst series switches and Microsoft .Net, SQL servers, Hyper-V virtual servers and Data Protection Manager for replication.
The Shreveport facility is a hot site, which means it’s replicated from the main data center in real time. “If I have to fail over, I can, and I know I can because we test it quarterly,” Evanson says. “We’re in a constant state of flux, like any IT organization. We’re always upgrading equipment, so it’s important to test frequently to make sure that all the changes you made are incorporated.”
Don’t Put Off Until Tomorrow…
One of the biggest lessons the state of Mississippi learned from Katrina is that disaster recovery planning can’t sit on a to-do list until the time and funds are available.
In fact, during the summer of 2005, the state had just begun looking into building a network exclusively for first responders, but the storm gave a new urgency to the project.
“It really showed how challenging communications can be, because the devastation impacted not only the coast but the bottom third of the state, leaving a tremendous swath of first responders unable to communicate with each other,” says Craig Orgeron, CIO and executive director of the Mississippi Department of IT Services.
The state created a wireless commission that became ground zero for the development of the Mississippi Wireless Network, a 700 megahertz, push-to-talk land-mobile radio network that cost more than $300 million to build. The network, which has 144 towers, is now available to more than 20,000 first responders.
“Now the state has this progressive, border-to-border network,” says Orgeron. “I think you could argue convincingly that without the storm and the galvanizing of that will, it would not have happened.”
Katrina delivered another major blow to Mississippi in the form of destroyed data. “There was a lot of destruction, and a lot of medical records were lost,” says Orgeron. After the storm, Gov. Haley Barbour created a Commission on Recovery, Rebuilding and Renewal, and one of the initiatives it explored was creating electronic medical records.
That morphed into a statewide health information exchange called the Mississippi Health Information Network. While the exchange aims to make entire medical histories available at the point of care, it has the added benefit of digitizing records in case a disaster wipes out physical documents.
Protecting data in general should be at the core of any disaster recovery plan, Orgeron adds.
“For years, agencies could survive without backing up their data, because it was a cost they could not afford,” he says. But that’s changed with the growth of the cloud. “We’ve seen more entities backing up the data offsite and acknowledging that if something truly cataclysmic happens, the devastation is complete and total, making the backups invaluable.”
Not only does the cloud ease backups, but it makes the process more affordable. “It’s like long distance,” Orgeron says. “It’s so commoditized and bundled, you don’t even think about the cost.”
1800: The number of casualties attributed to Hurricane Katrina
90,000: The number of square miles affected
$108 billion: The total damage estimate in the United States
Sources: Federal Emergency Management Agency, “By the Numbers — One Year Later,” August 2006; NOAA National Climatic Data Center, “Hurricane Katrina: A Climatological Perspective,” October 2005 (updated August 2006); National Hurricane Center, “Tropical Cyclone Report, Hurricane Katrina,” December 2005 (updated September 2011)
The New Normal
When Katrina hit, New Orleans had a 1,500-square-foot emergency operations center with 13 seats in a U-shape facing a computer screen on the wall. Ten years later, the city’s EOC has 90 seats instead of 13 in a 10,000-square-foot space.
The chief lesson the city learned from Katrina is that communication and coordination are essential, so it rebuilt its emergency operations center to accommodate the many partners — federal, state and local agencies; private businesses; nonprofits; and faith-based organizations — that must work together during a disaster as well as on a day-to-day basis.
The goal is to “create a second nature to preparedness,” says Aaron Miller, deputy director of the New Orleans Office of Homeland Security and Emergency Preparedness. “We meet regularly to deal with planned and unplanned incidents.” Coordination suffered during Katrina because first responders couldn’t communicate, so the state and city have invested heavily in the Louisiana Wireless Information Network, an interoperable safety network for federal, state and local public safety.
“All of our officers on the street have 99.9 percent coverage outdoors with a portable radio,” Miller says. Backed up on microwave, the system also handles day-to-day incidents and planned major sporting events or Mardi Gras. Social media has proven to be another valuable disaster recovery tool. NOLA Ready, the city’s multichannel public awareness campaign, helps build a year-round culture of preparedness.
“We can provide crucial information about events as they happen to keep residents informed,” Miller says. The city’s public information officers monitor social media to gather information from the public about emergencies. The spirit of collaboration has helped the city as its biggest disaster evolves into one of its biggest events. In honor of the 10th anniversary of Hurricane Katrina, the city and its various partners created a website, katrina10.org, where they pool resources and information into one place.
“What we’ve figured out is bringing people together at a holistic level creates a better planning process,” Miller says.
Trading Up
Technology developments such as mobility and cloud computing have improved disaster planning in many ways. A focus on accessing data from devices regardless of location makes it easier to maintain operations even if buildings become inaccessible, power grids falter or networks crash.
The Louisiana Supreme Court supplies employees with mobile devices and is looking into virtual desktops. The court also makes e-filing available. In addition to strengthening disaster recovery, newer technologies can also reduce costs.
For example, the court used to ship cartons of case files to justices every week, but the justices can now access digital case files through Apple iPad devices. The savings in shipping charges quickly made up for the cost of the devices. “It was an easy return,” says CIO Mike Evanson.
That said, Evanson doesn’t consider return on investment in disaster planning. “We don’t really have a choice,” he explains. “It’s not a matter of risk and reward here. There are certain things that we just must do. Government has to function.”
