How 4 IT Departments Orchestrated Successful Data Center Moves

For an IT team, nothing may seem more stressful or daunting than the prospect of physically relocating the data center. But a move can also bring big benefits, including increased capacity and greater efficiency, whether moving thousands of servers or just a few.

“There are so many interdependencies inside a data center, and just about everyone in the organization counts on the data center to do their jobs effectively,” says John Heinlein, IT manager for Livingston County Community Mental Health Authority in Michigan, which completed a relocation in May. “It’s not something you can just tackle in an afternoon. It takes a lot of planning, patience and cooperation across the whole organization.”

The good news for both small and large government entities contemplating such a move is that Heinlein’s agency and an increasing number of other IT departments can provide advice as well as inspiration for a successful transition. In the state of Oklahoma, for example, IT staff worked closely with stakeholders to ensure good results. In Canfield, Ohio, planners ensured a smooth relocation through tactics such as color coding and creating backup plans.

“There is no reason to blow through this,” says Dominic Rozzo, network and systems administrator for Canfield. “Plan well and execute the plan, and it will work out.”

SIGN UP: Get more news from the StateTech newsletter in your inbox every two weeks

1. Livingston County Upgrades Operations with a New Data Center

For officials at the Livingston County Community Mental Health Authority, the decision to move to a larger, refurbished facility located just a mile away was a pragmatic one. The old data center didn’t have a backup generator. Instead, a few UPS devices kept equipment running during temporary blackouts. Too often, power outages outlasted the backup power, and Heinlein needed to come in, stand up the servers and spend extra hours testing them and making sure everything was running properly.

“The real push for the move was the realization that being in a facility with a generator and more reliable power would be better for the equipment, better for the end users and better for those we serve,” Heinlein says. Heinlein and organization officials decided to consolidate most of the agency’s virtualized but outdated servers and a separate storage area network into a single, hyperconverged solution from Nutanix.

“Instead of having two servers and a storage array, which take up about six units of rack space, we went down to having everything in two units of rack space,” Heinlein says, explaining that most of his 175 end users rely on Citrix thin clients to access files and applications.

Recognizing the difficulty of relocating, Heinlein started planning for the change a year prior to the move. CDW engineers assessed infrastructure needs in the new facility, which included updating the core data center network with newer Cisco Systems components, installing a new APC locking rack and a UPS at the new facility, and updating the existing VMware vSphere software. Finally, a Nutanix engineer began consolidating the legacy equipment into the new hyperconverged solution, and a CDW engineer helped set up the server migration.

Heinlein also worked with every end-user device to institute the necessary networking changes and coordinated with AT&T to time the cutover of the network. All of this preparation paid off. The actual move took just 24 hours to complete, and everything has run effectively and efficiently since then.

“The more you can do ahead of time, the more you can front-load, the better you can anticipate and adjust for challenges, the more smoothly everything goes,” he says. “It’s the old ‘measure twice, cut once’ philosophy. It applies to IT just like it does to framing a house.”

2. Organization Makes Canfield's Data Center Move Successful

In Canfield, Rozzo faced the daunting challenge of moving an existing data center from inside the local police department to a newly constructed wing in the same building. But given the center’s cramped conditions, he enthusiastically embraced the change.

The original center, which consisted of two racks of servers running the city’s entire infrastructure, was “essentially shoved into an electrical closet with a mess of other wiring,” he says. “We could barely move around in there ourselves, let alone add additional equipment. It was crazy.” Rozzo and his team worked directly with the architect and builders of the new wing to make their specific technology needs known.

“The room has a raised floor with a drain, and all of the walls and ceiling are reinforced with steel rebar to withstand a building collapse. And we have the room we need to work comfortably and grow over time.”

The team created backup plans and left nothing to chance. They designed the networking plan to minimize downtime, determined cooling needs, ensured ongoing connection to the old center, mirrored the switch configurations between the old site and the new one, and coordinated timing with users and other services.

Color coding and labeling wires sped the process, Rozzo says. Blue indicated a data line, white signaled a phone line and cyan specified fiber.

“With one look, you know exactly what each wire is, and that minimizes the troubleshooting,” he says. “When you have so much going on and when you wear so many hats like we do, it makes a huge difference in both efficiency and confidence.”

Rozzo preconfigured a stack of Cisco 3850 switches and made sure all networking and backup plans were in place, which helped him and a non-IT professional to complete the transition in just 24 hours. Despite the careful preparation, a phone server for the police dispatch system inexplicably powered down. Relying on a backup plan, Rozzo quickly switched the line over and minimized downtime.

“It just shows how critical it is to take your time to do everything carefully and correctly,” he says.

3. Oklahoma Offers Better Services Through Upgrades

Data center moves have become a routine operation for Oklahoma’s Office of Management and Enterprise Services (OMES), which in 2011 started a long process to consolidate individual agency-run centers into one state-run Tier III location in Oklahoma City.

The most complicated task: moving the largely antiquated data center of the state’s Department of Human Services (DHS), which comprised 8,222 square feet and operated 55 highly critical systems.

For months before the scheduled move over Memorial Day 2016, state CIO James “Bo” Reese and his team worked in tandem with the DHS IT team and business users to effectively prepare the environment. They retired older servers, virtualized some physical servers using VMware and further virtualized a large number of critical systems by moving them to an HP 3PAR StoreServ 7400 thin-provisioning storage/BLc7000 server blade enclosure, a system collectively known as Zeus. They then coordinated the transition right down to the minute. Their efforts also included preparing test plans to ensure the systems operated properly when everyone returned to work.

“When we started firing things back up that weekend, we had DHS users working with us who could log in to their systems and immediately start working through their test plans and identify and log any issues,” Reese says.

That level of partnership and patience ultimately paid off. The teams completed the move on time with no outages. What’s more, the DHS systems now require just seven racks and 100 square feet in the OMES data center, a reduction of 810 percent.

“I was trying to be a cheerleader every step of the way, but it was intense, nerve-wracking really, because these were systems that truly impact citizens and their kids every single day,” Reese says. “So we were absolutely thrilled when everything came together so successfully.”

4. Florida's FAST Introduces Holistic Upgrades

In late 2015, officials at Florida’s Agency for State Technology (AST) planned how to best combine, slowly and prudently, two data centers into one by 2019. When environmental issues turned up at the older of the two facilities, officials received an order to vacate in 120 days. The plan needed to come together much sooner.

“I think words may have failed me at that point,” AST executive director and state CIO Eric Larson says. “We were a bit dumbfounded at first as to how we were going to do a move in that time frame.”

The challenge was immense. The contaminated data center housed more than 2,000 servers for 12 state agencies, including the Department of Children and Families and the Department of Environmental Protection. The team had to disassemble, decontaminate and re-assemble the servers and other equipment.

But the shortened timeline ultimately proved to be a benefit when it came to getting the cooperation of the affected end users. Agency executives ordered cooperation throughout each of their departments, which ensured a holistic, high-priority approach.

“Everybody was aware that this was an extraordinary set of circumstances,” says Larson, who notes that agencies were given the choice to have their servers virtualized upfront and migrated to the new data center or physically moved by truck.

Virtualization played the most crucial role in speeding up the move. Because of their existing plans to merge the two operations, the AST team had already put in place a data interconnect between the two data centers that provided significant bandwidth.

“That allowed us to move not only virtual workloads between the two sites, but also move an application server or a database to the target facility. That way, any dependencies that remained behind would still function across that interconnect,” Larson says.

Still, AST ended up playing a kind of shell game, migrating workloads to the new site and then moving the now freed-up equipment from the old site before more migrations could happen.

“We would move that during the week, which added more capacity to the new data center in time for the next migration event,” Larson says, adding that the process allowed his team and other IT personnel to inventory and identify more than 300 servers that needed to be retired. “It was iterative that way.”

The plan and process worked better than anyone could have anticipated, and the AST team managed to decontaminate and move the data center in 100 days —nearly three weeks ahead of schedule.