Proper Planning Kept IT Agencies Afloat During Hurricane Sandy

Despite escalating warnings about October's perfect storm and a Water Street address, the engineers working in New York City Department of Transportation headquarters were so confident that the building could withstand Superstorm Sandy that they parked in the underground garage when they went in to prepare for the storm.

"We really didn't expect that we would be put out," says DOT Chief Technology Officer Cordell Schachter. "Then King Neptune had his way, and water came up like no one predicted." The lobby and three lower levels were flooded. Electricity, phones and emergency power generators went out. And the engineers who parked below lost their cars, Schachter says.

Yet a week later, mission-critical DOT employees had temporary office space outfitted with new all-in-one desktop and notebook computers. What normally would have taken months to set up took only a few days. But DOT is just one agency among many that could have been crippled by Sandy, had it not been for some technical insight that kept them running when residents needed them the most.

Although many workers had mobile devices, DOT still needed about 400 computers. Because Director of Technology Services Yongli (Tony) Duan had kept abreast of products in the market, he knew what to buy to get employees back to work. "That allowed us to move quickly," he says about receiving the HP all-in-one PCs. "You plug in the power cord, you connect to the wireless network, and you're in business."

DOT is now integrating lessons learned from the storm into its mission statement: Have a stronger cloud presence, better remote tools and more automated failover, though Schachter adds those things would have helped only so much. "We can't stockpile desktop equipment. If another storm hits, we still need partners who can react very quickly."

You Can Hear Me Now

Due South, Sandy battered the New Jersey shore, and Ortley Beach in the Township of Toms River was among the hardest hit. "Homes were just removed from their foundations," says Eliot Perez, IT manager for Toms River. "All our boardwalks are gone."

At the height of the storm, Toms River emergency vehicles lost wireless access, leaving police, fire trucks and emergency medical vehicles with no way to connect to the state database. That meant they couldn't check licenses, write reports or obtain critical information such as addresses for responding to emergencies.

"This was their communications back to the station," explains Perez. "We had a full force out there because of the storm, and they couldn't access anything on the system."

Credit: Matthew Furman

First responders in Toms River, N.J., maintained communications using NetMotion mobile VPNs, says Eliot Perez.

But as luck would have it, Perez and his team had begun a pilot of NetMotion Wireless Mobility XE a week earlier, because Toms River's existing vendor had plans to discontinue the township's long-time wireless solution — and when that solution failed, the NetMotion virtual private networks worked.

It was about 1 a.m. when the cars lost wireless access. "The minute we sent out an email [to get additional NetMotion licenses], somebody got back to us, and we were up and running probably within an hour," recalls Perez. He and his co-manager of IT, Joe Jennings, were piloting about 50 licenses, and NetMotion gave them an additional 25.

"Basically, anything we needed, they provided," Perez says. "It really got us through."

After the storm, Toms River embarked on a full-scale NetMotion rollout. While the old system worked fine before the storm, Perez says the NetMotion solution has several advantages: tools to analyze data and block traffic on different parts of the network, a faster connection, and more robust security. He adds, "The end users don't even notice the difference. They just log in, and they're at work."

Radio Days

From the 9/11 attacks to the 2003 blackout, the New York City Department of Information Technology and Telecommunications (DoITT) has learned not to rely on any one mode of communication. With each crisis, explains Steve Harte, associate commissioner of wireless technologies, "You put a plan in place to attack any vulnerabilities you have, and you get stronger and more successful for the next one."

During Sandy, for instance, when wireless networks failed, DoITT distributed radios to agencies throughout the city — from the parks department to the mayor's office — so they could coordinate with one another during the storm. "Everyone depends on commercial technologies," says Harte. "It almost becomes like water — you think it's going to be there all the time, and unfortunately, it's not."

Before the storm, DoITT worked with the city's Office of Emergency Management to form the Healthcare Evacuation Center, a unit established to coordinate the safe movement of patients from hospitals, nursing homes and other health facilities.

"I was monitoring over the radio, and it was moving to know that communications you've just deployed were saving lives," Harte says. "You'd hear, 'We're on the ninth floor. We need ambulances right away.' This was their 911, essentially."

DoITT's radios also came in handy for a Department of Health–led mission to send teams to affected areas to see if people who had stayed in their homes needed medical attention. "They were out there at night in unsafe conditions with no power," Harte says.

Meanwhile, DoITT occupied its war room in Brooklyn's MetroTech Center. "We were taking various power hits," says Harte, "but we built our networks to be hardened and resilient so that we could manage them through events like Sandy."