Baker County Relies on Virtualization in Its Recovery from a Devastating Flood
When workers at the Baker County courthouse saw the damage that had been done after a pipe burst over Thanksgiving weekend in 2010, they were shocked.
"It was like it was raining inside the building, there was so much water," says Bill Lee, technology director for the eastern Oregon county with a population of about 16,000.
The first and second floors were flooded, along with the basement.
The water wrecked office equipment as indiscriminately as it damaged the floors, ceilings, furniture and books of records more than 100 years old. In all, the cost of recovery came to roughly $800,000, says Fred Warner, chairman of the Baker County Board of Commissioners.
"It was like it was raining inside the building, there was so much water."
Bill Lee, Baker County
Mark Bennett, the county's director of emergency management, credits the Baker County IT staff with helping the local government bounce back quickly from the disaster. Led by Lee, the IT staff moved operations to an alternative site and had the county government back on its feet quickly.
"We were back in operation with 13 county departments and three state departments within three days," Bennett says.
The county followed up the episode with a virtualization project that has cut the required number of physical servers, reducing costs for the county while improving its ability to recover quickly from future disasters.
Alternative Plans
In the wake of the flooding, county officials quickly assessed the damage and came up with a plan for getting the government back on its feet. The finance and IT departments remained undamaged by the flooding, so they remained in an annex to the main government building. Most of the other departments moved to alternative locations, most of them to an unused elementary school building.
The IT staff ran nearly 15,000 feet of cable to repair the courthouse network and extend it to the alternative facility, taking the opportunity to upgrade the old CAT 5 cabling that had been installed in the main building to CAT 6. Ultimately, 60 workers were moved from the courthouse to new locations.
After working offsite for months, county employees returned to the courthouse in May 2011. Their new IT environment featured some significant upgrades. In rewiring the building, the IT staff color-coded and mapped the entire network in detail, providing a transparency that previously wasn't possible.
"Now if someone is having an issue, we know exactly what cable they're on," says Jill Finney, an IT technician for Baker County. "We're able to troubleshoot a lot better."
Virtual Healing
In answer to the disaster, the county also implemented server virtualization, enabling IT staff to back up servers easily and reallocate new machines quickly. Data is mirrored on a backup server and can be stored on an external device as well.
"Server virtualization allows us to instantly relocate," Bennett says. "Even if we lost everything onsite, the IT department could move, re-establish operations, and we'd be back up."
Freeing the software from the server hardware has provided a host of other benefits. The IT staff implemented VMware's vSphere software, running 16 virtual machines on two IBM Series X servers, with two Promise VessRAID 1830i storage area networks, allowing Lee to reduce the number of physical machines from 10 to four.
The reduction in the number of servers also solved a cooling problem the county experienced when the server room air conditioner couldn't keep up with the heat produced by the servers. Fewer servers also let the IT staff make better use of the server room's physical space, a boon for a county with limited resources and capacity to expand its IT operations.
"In the future, our entire environment will be virtualized," Lee says. "We'll be able to run all our services on one server and have one for backup."
The savings provided by the virtualization has given Baker County the ability to pursue new technology, even though its budget has been flat for years. Lee says the county recently implemented a Barracuda Networks web filtering system and plans to add mobile device management from AirWatch.
"We had to become more efficient, and we had to cut costs. The way we do that is to utilize technology," Warner says.l
Need for Speed
Among the many benefits Baker County enjoys from implementing server virtualization is the ability to build server templates for later use.
IT technician Jill Finney says that allows the county to provision new servers quickly and easily. For example, when Technology Director Bill Lee told Finney he needed a new File Transfer Protocol server, she immediately provided him with a previously created template. He made a few minor changes and had the new server up and running in a matter of an hour or two, whereas installing a physical server would have taken more than a day.
"It shaves a lot of time off, having those ready to go," Finney says.
Lessons Learned
One of the most difficult challenges Baker County faced in the aftermath of its courthouse flooding in 2010 was quickly re-establishing Internet connectivity. The county's Internet service provider told officials the service could be restored to an alternative location in two weeks, a timeframe county officials considered unacceptable, both for the citizens who needed services and the employees who needed to be paid on their regular schedule.
County officials worked with the provider to restore services more quickly, but not before a period of great stress and difficulty. Preparing for such issues before a disaster would greatly smooth recovery, says Mark Bennett, the county's director of emergency management.
Fred Warner, chairman of the Baker County Board of Commissioners, offers these tips to help bounce back from disaster:
- The involvement of rank-and-file workers is essential. "I think you have to have good people," says Warner, adding that non-IT workers helped with some of the technology-related problems the county experienced, such as re-wiring the network.
- Communication is vital. County officials met quickly to settle on a plan to bounce back from the flooding, involving input from a variety of sources.
- Know what assistance technology can provide. Funding will always be short, Warner says, but knowing how IT can fill in the gaps can offer a head start on recovery.
