Staying in Touch
Staying in Touch
Well-crafted end-user surveys can unearth problems and offer actionable insights. Here’s how.
In Denton, Texas, CIO Alex Pettit says he almost dumped an e-mail app until a survey found that city employees liked it and used it widely.
Every five years, the city of Denton, Texas, surveys 1,000 city employees to gauge the quality, satisfaction and performance of its software.
The survey is one of Chief Technology Officer Alex Pettit’s favorite evaluation tools because it serves as a temperature check on how well the city’s systems are doing their jobs. “In a lot of cases, the user departments review software applications and choose the one that they would like to have implemented. But sometimes things look more enchanting in the store window than they do in real life,” Pettit says.
Denton’s most recent survey was a real eye-opener. It revealed that city employees widely used and liked an e-mail application that trade magazines had blasted for poor performance compared with competing messaging apps. Based on what he had been reading, Pettit says he “was going to fry that application. Now, I’m not going to do that.”
On the flip side, Pettit had thought that a four-year-old public-safety operations management application, used by the city’s Police Department, was meeting workers’ needs. Not so: 150 users gave it their lowest rating, saying that despite vendor upgrades and improvements, it was still too difficult to use.
Armed with the survey results, Pettit took action. The city is adding a new freeware front-end to the public-safety app to ease its use by officers when writing tickets on their handheld devices.
When it comes to gauging the climate of the tech-user community, surveys can be a CIO’s most valuable tool, if they are done correctly. State and local leaders say such surveys yield actionable insight when done anonymously, by professionals, with a lot of pre-survey marketing and careful analysis of the results.
To demonstrate the finer points of effective survey-taking and assessment, consider best practices from survey work done by the state of New York and the city of Denton.
59% The response rate for New York’s survey to assess IT workers’ skills.
Keep it short: Both Denton, Texas, and the state of New York used no more than 60 questions on surveys so that respondents wouldn’t find them daunting.
Get an Outside Assist
An outside survey administrator can offer anonymity to employees and yield honest responses — not to mention the experience it can bring to crafting the surveys and extracting actionable results.
New York State CIO Michael Mittleman wanted to assess the training needs of the state’s 5,000 information technology employees and identify future IT skills needs. With the help of the CIO’s Human Resources Committee, he enlisted the Center for Technology in Government at the University at Albany, State University of New York, to conduct an IT workforce skills assessment survey. “Its reputation as an independent research committee encouraged workers to participate,” says David Gardam, co-chairman of the HR Committee.
Denton has enlisted a research team from the University of North Texas to perform several surveys over the past five years. “It’s worth having somebody else do the evaluation,” such as a university, community college or consulting firm, Pettit says. Otherwise, “people think it’s disingenuous. There is a tendency to be defensive about technology by technologists.”
Divvy Up the Demographics
Once the list of survey-takers is identified, “it’s important to have a good set of demographics that can be cut in different ways,” says Glenn Cudiamat, vice president of research services at Strategic Directions International, a market research company in Los Angeles.
For instance, if supervisors will be evaluating software that they chose, the results analysis should separate their responses from end users’. Likewise, consider whether to parse out responses from different departments if they use the same technology for separate tasks and may not agree on its usefulness.
New York state crafted two surveys. It directed the first to IT workers and the second to CIOs in state agencies.
It’s also helpful to survey a smaller group first sometimes and then a full range of users, Cudiamat says. “You get an idea of how people are answering initially. Then you can alter questions or add a follow-up question.”
Denton first surveyed a few “champions” who have significant input and influence on applications. For the second phase, it solicited responses from all IT employees.
Fine-tune the Questions
Surveys must be easy and understandable, and should zero in on specific technologies. New York evaluated the need for training in 126 specific skills. Denton focused questions on its top 20 software applications.
At the University of North Texas, researcher Dr. Victor Prybutok and his team used a series of survey questions to measure constructs that gauge a respondent’s overall satisfaction with the system, its usefulness, ease of use and social influence.
New York customized a survey the federal government designed to gauge its IT workforce. The federal government created its survey in response to the Clinger-Cohen Act of 1996 that requires agencies to look at IT performance metrics. New York restructured the survey to include more information on demographics, and added a CIO-only section and specific training questions for other departments.
New York and Denton each kept its surveys short — 55 to 60 questions. “There’s a limit on how much time somebody will spend filling out a survey,” Prybutok says. Also, to encourage participation, New York put all its surveys online.
Refine the Results
After compiling the survey data, the Center for Technology in Government held three analysis meetings with the CIO’s office. First, it presented raw demographics. During the second meeting, the center team did a first take on detailing and packaging the information, but also asked for opinions of the results and what additional queries the state CIO staff wanted to make regarding the data. At the third meeting, the center presented the results.
The center also crafted 57 agency-specific reports so that each agencies’ training personnel and CIOs could use this information to make their own training adjustments.
For Mittleman, the survey findings solidified his concern about retiring workers. “We were able to pinpoint areas where the critical mass of a particular resource could be threatened by retirements.”
What’s more, many departments were planning to migrate to more Web-enabled technologies but lacked the skills to do so. “It was something to see how desperate we really are in having an adequate critical mass of people with those skill sets,” Mittleman says. Based on that gap analysis, the center identified 15 high-impact skills that IT workers will need in the next three years.
With New York’s IT future riding on the recommendations, Mittleman says he’s more confident that training funds will be well spent because the data will support requests.
“We’re paying thousands of dollars for employees to acquire new skills,” he says. “It’s incumbent upon us to make sure we have identified those training opportunities that maximize value for the state.”
StateTech Quick Tips on Conducting User Surveys
- Get buy-in from all interested parties. Make sure you have an executive sponsor, and run a public outreach campaign to generate interest and participation.
- When crafting survey questions, talk with a few people first, then to the full range of users. Get an idea of how people are answering initially so you have the opportunity to alter questions and add follow-up questions.
- Don’t do the survey yourself. It’s worth having somebody else do the evaluation to garner trust with your survey respondents and to capitalize on expertise.
- Periodically reapply the survey to get a quantifiable measure of success. The first survey creates a baseline against which you can measure if and to what extent your changes are working.
Tip: Do Some Early Marketing
More than two months before it issued its survey on employee information technology skills, the state of New York started an extensive outreach campaign to publicize it.
“We involved everybody we could think of — the Civil Service Department, employee unions — to be part of the process,” New York State CIO Michael Mittleman says. “We were scratching an enterprise itch. Employees, CIOs and commissioners were well aware that we have limited resources for spending on training. So it was in everybody’s best interest to use this kind of a device to target where the training budget would bring maximum returns.”
To keep momentum high, the CIO’s office, the Center for Technology in Government (which helped the state develop and manage the survey program) and labor union leaders each sent letters to employees urging them to participate in the survey. Liaisons were assigned to each department to gain further support. Posters promoting the survey were hung in IT departments.
The efforts paid off. More than 59 percent of the state’s IT workforce responded. “That’s extraordinary compared to most surveys” that garner single-digit response rates, Mittleman says.
