Wake Up from Your IT Nightmares
When Matt Ceniceros decided to accept the newly created position of IT coordinator of Grand County, Utah, he expected the job to be a challenge. What he found, though, was a nightmare: Grand County boasted a homegrown, obsolete technology environment made up of ad hoc departmental networks, a jumble of unlabeled point-to-point connections, overtasked servers and outdated operating systems and applications. The situation had resulted after years of technology implementation without any centralized planning or management.
"When I was hired, I was supposed to come in and essentially administer the network, administer the servers and take care of the IT needs of all the departments and employees," Ceniceros recalls. "I knew that I would have to upgrade many of the systems, but instead I inherited a network that was basically on the verge of collapse."
Ceniceros didn't panic or quit. Instead, he did what good IT leaders do when they chance upon a nightmare scenario: Recognizing opportunity, he not only fixed the problem but crafted a solution that would better position Grand County for the future.
"I decided to start over," says Ceniceros, who by himself supports the computing needs of 200 employees. Since joining the county last November, he has developed a multiyear strategy to create a standardized IT environment based on Windows 7, a centralized phone system and an enterprisewide network that's robust, flexible and scalable enough to support continuity of operations and e-government services.
According to Ceniceros, the trick to addressing an IT nightmare isn't always about identifying the latest and greatest technology. "You've got to have the technical skills, but you've also got to be a diplomat who can gently nudge people toward a more effective solution while also letting them realize what's ultimately in it for them in terms of cost savings and productivity gains," he advises.
$42 billion:
Total cost of spam to U.S. organizations in 2009 in lost productivity, help-desk support and spam control software solutions, according to Ferris Research
The Grand County IT nightmare might be more shocking than most, but anyone who works in IT will at one time or another experience the specter of something gone terribly wrong, whether it's a security breach, a sudden loss of network performance or an insufficient data recovery plan. But a bad situation is also an opportunity to take notice, implement a solution, and enjoy a quantum leap in function and productivity. This can be seen in the following tales of woe that resulted in happy endings.
Dreams of Disaster
Many IT professionals lose sleep when they realize that their technology configurations are flirting with disaster. That was certainly the case for Chad Coble, IT director for Stanly County, N.C., who worried openly about the 911 operations center, which had no remote data replication and backup for its computer-aided dispatch system, records management server and related applications.
Because dispatchers work in the basement of the county courthouse, "a burst pipe, flood, fire, or tornado would have wiped us out," Coble says. "We were very fortunate that we never had a situation that did us in."
And now it's likely that they never will. Coble and his IT team deployed a disaster recovery solution that features VMware server virtualization coupled with an HP StorageWorks storage area network. The two products work together to enable synchronous data replication and automatic failover of all mission-critical data and applications to virtual servers at a secondary communications center a few miles away.
The new solution is straightforward to manage and makes it easier for the IT staff to perform routine server maintenance, such as updating applications, adding memory or replacing hard drives. "We can just migrate those virtual machines to another host and do whatever we need to do, and there's zero downtime," Coble says. "That's been very positive."
A viable backup plan for the 911 center has also enabled Coble to streamline the county's overall disaster recovery plan. The biggest reward, though, is peace of mind. "If there's a disaster of any sort, our frontline emergency system is going to be back up online and serving the citizens' 911 emergency needs within minutes, if not immediately," he says. "I do sleep a lot better at night now."
Monsters in the Dark
Some IT nightmares sneak up on you, as David Jordan, chief information security officer for Arlington County, Va., can attest. He knew his organization had a spam problem in 2007, but he couldn't be sure of the magnitude of the problem because he rarely received any complaints from users.
"Getting inundated with spam is kind of like having the shock absorbers go bad on your car -- you get used to the ride as it slowly degrades," Jordan explains. "With spam, people don't pick up the phone and call you, they just keep hitting the delete key."
At the time, spam was a nuisance, but in large quantities it's a major drag on network capacity and performance. In the recent past, it has become a prime mechanism for cybercriminals to introduce malware into an organization.
After Arlington County tested the Symantec Brightmail Gateway Antispam appliance, Jordan finally realized just how insidious the problem really was. The county was receiving more than 125,000 unwanted messages a day (now more than 300,000). He and his team conservatively estimated that the spam was costing more than $1 million per year in lost productivity.
Not surprisingly, Jordan decided to go ahead with the Symantec Brightmail purchase. As a result, "we almost immediately experienced huge productivity gains. The application paid for itself in a couple of weeks," he says. "Basically, spam is under control, and our engineers are now free to investigate the more insidious cyber threats to the network."
Before embarking on a network overhaul, Matt Ceniceros advises IT leaders to spend time building trust with customers to ensure buy-in.
Photo credit: Chad Hurst
Feeling Exposed
Like many small local governments, the city of Diamond Bar, Calif., has only three full-time IT staff and relies heavily on vendors to help it effectively maintain its collection of financial, customer relationship management, e-government and other applications. The IT department was using an IPsec virtual private network so that about 25 different software makers could remotely access the servers housing their applications and fix or upgrade them to keep them running smoothly around the clock.
But the VPN was creating more problems than it was solving, says Ken Desforges, Diamond Bar's IS director. For starters, the solution required the installation of client software at the vendor site, and that meant that there were a limited number of computers from which the technical staff could access the Diamond Bar system.
As a result, Desforges recalls, "We didn't always get timely support because the one person who might be able to solve the problem might not be in the right office or for some reason didn't have immediate access to a workstation or notebook that could access our system."
What's more, Diamond Bar officials had no way to ensure that the machines connecting to the network were clean and updated with the latest patches, unnecessarily exposing the city to a potential security breach.
Desforges and his team quickly discovered a better option in the F5 FirePass Secure Sockets Layer VPN. The appliance gives vendors convenient access to the Diamond Bar network via the web, and also requires third parties to install an ÂActive-X client on whatever device they're using. This guarantees that anyone logging in has appropriate antivirus software and other protections.
The new solution delivered immediate benefits. "Our support from the vendors has improved dramatically because they don't have access issues or limitations, and that has led to greater uptime for the applications for both our internal users and our external constituents," Desforges says.
In fact, the success of the SSL VPN prompted the IT staff to install F5's Big-IP Link Controller application, which provides redundancy and load balancing for the city's robust collection of online services.
"Now we get rave reviews about our online applications and their availability," Desforges says. "So it's all good now."
Eyes Wide Open
IT nightmares could increase in the near future as the public sector struggles to stay abreast of rapidly advancing technology requirements in an era of budget constraints. Here are some tips IT managers can use to foresee potential disasters and devise a solution before the problem becomes reality.
- Imagine "what-if" scenarios. Chad Coble, IT director for Stanly County, N.C., routinely has brainstorming sessions with his staff in which they think through potential technology issues to their most damaging conclusions. Then they take steps to try to ward off calamity. "The more specific you can be, the better," he says. "The devil is
in the details." - Build trust. Executive leadership will be more likely to take quick steps to avoid disaster if IT personnel have built a record of success and make it a habit not to cry wolf, says Ken Desforges, IS director for the city of Diamond Bar, Calif.
- Keep your warnings relevant. Don't just say you need more money for better security; say, for example, that a pending security nightmare will result in the loss of personally identifiable information. The more specific you can be about how an event might negatively affect your organization, the more likely you'll receive project approval, Desforges notes.
- Learn from your mistakes. IT nightmares happen -- it's the nature of a fast-paced, complex environment. When one occurs, says Coble, all you can do is see it as a learning opportunity and work hard to make sure it doesn't happen again.
