From Crisis to Convergence

Helping people is easier with VoIP as Michigan's Department of Human Services takes its first migration step.

Human crises present enough challenges without an antiquated phone system that is unreliable and virtually unmanageable. A basic Voice over Internet Protocol (VoIP) implementation eliminated this problem for Michigan’s Department of Human Services while laying the foundation for next-generation converged applications.

With the old Time Division Multiplexing infrastructure largely beyond repair, DHS turned to VoIP, zipped through a 30-day comparison of three leading solutions, and mapped out a five-month rollout of Cisco’s IP telephony across 77 sites. Quickly learning that smaller steps make for a smoother migration, DHS abandoned this site-per-day schedule and made sure users at each location were seated securely on the VoIP bandwagon before moving on to the next site.

This patience paid off in fewer help desk calls and greater user satisfaction as it helped streamline communication with colleagues and service recipients. Next-generation applications will have to wait for migration to DHS’s remaining sites; meanwhile, users are learning new capabilities every day.

Beyond Repair

Like many geographically dispersed organizations in the pre-convergence era, Michigan’s Department of Human Services had collected an incompatible hodgepodge of voice technologies, including private branch exchange platforms, small key systems and Centrex service. Maintaining the aging equipment was increasingly inefficient and expensive; some of it was no longer supported by manufacturers.

With such antiquated voice technology, redundant configurations were either impossible or too expensive to contemplate, so critical services such as 24-hour crisis hotlines had no fall-back options in the event of system failures. The old infrastructure was also hampering the evolution of a mobile and location-independent workforce.

In early 2004, the Michigan IT Executive Council started looking at VoIP as a likely solution. Voice communication is particularly crucial in human services, and in October 2005, DHS — the state’s second-largest agency — was selected as the initial migration target. The total budget for the first stage, which would migrate 77 of the agency’s 148 sites, was $4.3 million.

“Short term, DHS wanted to replace the antiquated systems, centralize and standardize voice services, and save some money,” says Patrick Hale, Michigan’s director for infrastructure services. “Long term, the goal was to create a foundation for next-generation telephony services and applications.”

Picking a Platform

To choose the best VoIP solution, the convergence team decided on a bake-off among three leading IP telephony vendors whose products already figured heavily in Michigan’s legacy voice and data infrastructure. They piloted Avaya, Cisco and Nortel systems separately at three different sites.

One disadvantage of these brief 30-day pilots was their standalone nature: Because each environment was self-contained, the IT team was unable to test wide area network capabilities. Instead, the staff had to evaluate such capabilities on paper or accept vendor- supplied information.

IT polled users and managers at the end of the pilot period. The results showed all three solutions worked well with the underlying IP data network and there was good installation support from all three vendors. However, the tipping point was how the end users experienced them up close and personal.

The Michigan Department of Human Services’ VoIP rollout required lots of customization, says Ruth Goldman, telecommunications manager.

“We paid a lot of attention to the employees participating in the pilots,” says Ruth Goldman, the telecommunications manager for DHS. Cisco got the highest ease-of-use ratings, and users were particularly enthusiastic about the design and tactile characteristics of the physical IP phone sets.

DHS purchased 6,539 of the IP phones, along with four Unified Communications Manager servers, one Contact Center server, five Unity voice-mail servers, and 78 Integrated Service routers.

Timetable Stretches to Nine Months

In an effort to contain project cost overruns, DHS initially planned an aggressive five-month rollout. The schedule called for one site migration per day once the installation of core services was finished.

“Migrating 7,000-plus desktops in five months was way too aggressive,” recalls Hale. “You only get one chance to make a good first impression, and we didn’t want to have people thinking this technology is difficult or arduous to use.”

Adjusting all the local processes to fit one standard was a bigger challenge than the IT team had anticipated. One obstacle that needed to be overcome was incorporating DHS’s unique voice standards into the Cisco platform.

“We couldn’t use a cookie-cutter approach because each office had its own nuances, and all these customizations created a lot of extra work,” Goldman says. The rollout was moved to a nine-month timeframe and morphed quickly from a standard data technology implementation project to a uniquely designed voice management project. Some business change was necessary at most of the sites, and the IT team had to do a lot more hand-holding than they originally anticipated.

For the first phase, project planning started in March 2006 with the central CallManagers installed between May and June 2006. The system rollout began on July 1, 2006, and was completed in March 2007. Migration of DHS’s remaining sites awaits additional budget allocations in 2008 and 2009.

Rollout Results

One unpleasant surprise of the DHS rollout was the need to replace some 20-amp electrical outlets with 30-amp single-phase twist lock outlets. What’s more, two sites had to be rewired because they still had Category 3 wiring.

“We weren’t expecting that, and we had to postpone the installation at that site and rewire,” says Jack Harris, director of telecommunication for Michigan’s IT department. “Doing a physical inventory can get you 90 percent of the way there, but you are still going to miss some things. You need a flexible approach to your project that lets you move things around as you go.”

At every stage of the project, IT worked to build consensus by involving the users. “Executive support is all well and good, but you need the support of people who are responsible for things at the local offices,” Goldman says. “Otherwise you can get stuck on menu choices and other basic decisions forever.”

The Michigan team also took user training seriously and convened formal sessions conducted by professional trainers. The technical people stayed onsite for the first few days to field questions from users and resolve problems. Hale estimates that this strategy reduced the number of help desk calls resulting from the rollout by 60 percent to 70 percent. The team further minimized the potential help desk burden by using a dedicated VoIP help desk with its own phone number.

“Customer service is very important because you don’t want your users to have an initial VoIP experience that is negative,” Hale says. “Your people need to drop everything and fix user problems as soon as possible.”

Similarly, all the new reporting capabilities are a big plus. IT staff can monitor usage statistics on its 24-hour hotline to count the number of incoming calls, the number of abandoned calls, the average queue time, agent status, and other metrics. DHS is using such information from automatic call distributors at three locations to develop formulas for staffing requirements and target the best times to have staff available for handling calls.

Return-on-investment numbers for the initial phase of the VoIP migration are not going to be available for at least a year. However, the team is confident that the Cisco VoIP solution is at worst no more expensive than installing updated Time Division Multiplexing equipment would have been. It is probably generating some savings. More important, it provides a foundation for next-generation converged applications that Time Division Multiplexing platforms do not, no matter how new they are.

“On paper, the installation of [primary rate interface] trunking at most of the DHS sites showed significant savings to the state over the costly Centrex systems or the cost for incoming/outgoing trunks,” says Goldman. “We also reduced our telecommunications repairs budget by more than 50 percent.”

DHS at a Glance

The Department of Human Services is Michigan’s second-largest state agency with nearly 10,000 employees. Organized under the tenets of the state Social Welfare Act, it administers a $4 billion-plus annual budget.

In addition to the cash assistance programs, the department administers child and adult protective services, foster care, adoptions, juvenile justice, domestic violence programs and child support programs. It licenses adult foster care, child day care and child welfare facilities. Its staff serves 1.5 million medical assistance cases and 1.2 million cash and food assistance cases statewide.

Lessons Learned

  • Start with a thorough network infrastructure and facilities assessment. Expect to miss a few things and be prepared to deal with them.
  • Make sure there is enough port capacity on each data switch to accommodate the voice traffic being added, and retire outdated models.
  • Decide on a statewide dialing plan. Extension-only dialing may not be the best way to accommodate future requirements.
  • Choose one location for the core call managers and a second site for a redundant system to ensure 100 percent availability.
  • Some software problems are inevitable, so establish a special help desk hotline from the outset.
  • Give users formal training and keep technical people onsite for the first few days.
  • Use a technical SWAT team to triage problems and resolve issues quickly.

Streamlining Applications and Services

Children’s Protective Services of Wayne County experienced the most tangible, immediate benefit of the VoIP migration at the Michigan Department of Human Services. The inherently distributed technology afforded a level of redundancy and disaster recovery capabilities that was not possible with the old Time Division Multiplexing infrastructure.

Thanks to VoIP, the department can rest assured that its hotline really is available 24 hours a day. Elsewhere, voice mail has been centralized and integrated with the phone system, and personnel across the agency are enjoying enhanced services and streamlined workflow.

“I appreciate the new system,” says Jan Baszler, DHS director for Clinton County and Gratiot County, who calls the VoIP system “very customer friendly.” She divides her time between offices in the two counties, and the VoIP system automatically locates and routes incoming calls to her. Callers don’t have to figure out which office to call, and intercounty calls avoid toll charges by traveling across the wide area network.

Employees in one county can reach colleagues in the other by simply dialing their five-digit extensions. They can also make outgoing calls on a virtual line for the remote office, effectively inheriting the opposite county’s local dialing radius.

“As a dual-county director, I deal with matters that need attention the day they occur,” Baszler says. The VoIP system allows her to conduct such business “quickly and with no additional expense. I appreciate that, as does my budget.”

Personnel across the agency are enjoying streamlined and improved voice-mail services now that voice mail has been centralized and integrated with the phone system. Going forward, videophones seem feasible, and the agency is considering their deployment for foster care reviews when the budget allows.

While the initial focus is on rolling out basic IP telephony, the main intent was to create a foundation Michigan could build on in the future. DHS is transitioning to a philosophy of putting staff where the clients are — such as on tribal land in the state — and wants an infrastructure that supports this vision of a mobile, location-independent workforce. Future plans will include consolidation of other various departmental platforms onto the VoIP platform.

Sep 26 2007