May 06 2020

Q&A: Virginia CIO Nelson Moe on Scaling Up IT in a Hurry

The commonwealth had to rapidly expand its VPN capacity to handle a surge in state workers teleworking from home.

Over the past few years, Virginia has undertaken a significant transformation in how it procures and deploys technology services.

Starting in 2018, the Virginia Information Technologies Agency (VITA) moved away from a massive legacy IT contract and embraced a multisourced infrastructure environment, with contracts for messaging, mainframe, managed security, network and server storage, and voice and video network services. All of that work is overseen by a multisourcing service integrator responsible for coordinating and monitoring activities of the other vendors.

So, even before the coronavirus pandemic, Virginia CIO Nelson Moe had a lot on his plate when it came to digital transformation. Now, with telework tools reshaping how the commonwealth does business, Moe has had to manage even more.

As part of StateTech’s coverage of the virtual NASCIO Midyear 2020 Conference, held online due to the pandemic, we are speaking with state CIOs about how they have managed the transition to telework and how they will evolve their approach to IT service delivery moving forward. 

Moe spoke about how the state has adapted to remote work and scaled up its VPN infrastructure and how the culture of the state workforce will change.

STATETECH:  How has VITA adapted to remote work given the governor’s stay-at-home order, and how have you helped other Virginia state agencies adapt to remote work?

MOE: In a matter of a few short weeks, we were able to take a remote work technology set that was arguably geared toward the small peaks to almost seven times that to adjust for a significant amount of the commonwealth workforce working from home or alternate locations, including VITA, my particular agency. 

And we’re working closely with the governor’s unified command structure to make sure that the agencies and suppliers are addressing needs in a timely manner. The response in a tough time has gone and reasonably well, and I’m very excited about it.

STATETECH:  Before the pandemic, roughly how many state workers were working from home, and where are you guys at right now? 

MOE: Remote work for a vast majority of the commonwealth was something that was about work-family balance, but not a strategic aspect. And specifically, we have a virtual private network infrastructure that we’ve upgraded. But before this, we baselined it. The weekend before everybody started working home, the peak usage was about 3,000 concurrent users at any one time, and then about 66,000 devices on the network. So, some people did work from home and needed a VPN connection, but it was not near the peak it is today. And the maximum we were able to support at any one time was 5,000 concurrent connections. We never really got to that. We were always able to support the demand with existing infrastructure. 

After the COVID-19 events and the executive orders started happening, the demand for VPN technologies and other ones we have generated since then was dramatically increased almost overnight. In fact, today we’re seeing about 11,000 concurrent connections at any one time. And so, from a 3,000 peak to an almost 11,000 peak is amazing. And separately, we’ve changed our capacity to support those connections from 5,000 to 35,000. This was just traditional VPN. And so, we’re very comfortable with that.

In addition to that, we’ve been able to respond with other technologies. The company is called Zscaler, and they have basically an application layer VPN that is born in the cloud. We went from our on-premises VPN, which is the traditional VPN, to adding an application layer VPN, which agencies like [the Virginia Department of Transportation] love. It’s a little easier to use and just as secure. 

We were getting requests by the Department of Social Services because they have to connect to entities that do not have Commonwealth of Virginia accounts. And so, we have to set up virtual desktops for them. That technology is not brand-new, but we just could never come up with a business case to do it. Only a couple of people wanted it. Now, we have a huge demand for it. So, we’re able to jump in and roll out virtual desktops, and we’re using Microsoft’s Azure virtual desktop platform to meet those needs. 

STATETECH: How has VITA shifted in terms of how it works with the other state agencies? 

MOE: Virginia has positioned itself uniquely to provide an extremely adaptable service set. The phrase you use is “CIO as a broker,” providing services across a large number of things. I don’t have to own it; I just have to provide the contract to it. We have a supplier model, an integrated supplier model, and a contract structure that allows for a significant amount of flexibility. We’ve been moving toward the cloud for a number of years and have the ability to add services within the platform and also outside of it. In fact, we were able to adapt much more quickly with our service model in rolling out technologies — the VPN in the cloud, the virtual desktop, the laptops and cloud services — in our model now than we would have ever been able to in our previous model. 

Virginia CIO Nelson Moe
For the initial week or so, we really had to roll out and expand the capacity as fast as we could. After about two weeks, that was solved.”

Nelson Moe Virginia CIO

It’s been two years now that we’ve been on this journey. We’re seeing the fruits of being able to have a contract and service structure like a service catalog that is very adaptable to adding services. 

READ MORE: Find out how state CIOs think their operations will evolve. 

STATETECH: What has been the most challenging aspect of this from your perspective? How have you been able to address potential challenges?

MOE: For the initial week or so, we really had to roll out and expand the capacity as fast as we could. After about two weeks, that was solved. Now, it’s being able to meet the demand on the number of people who want to get connections. Now that we’re through that, the call center times are getting much, much better. The VPN requests are trailing off. I think the initial flood was what happened in every state. We were able to support our unemployment insurance agency as far as processing claims. They went from 2,000 claims to 600,000 claims in a matter of a month. And they were able to scale because of what we were able to provide them. And that was amazing. So, we were able to work through those challenges. 

The other challenge that we’ve been able to work through: A couple of agencies, their buildings got closed because they had to clean them. So, we had to set up a few disaster recovery sites on the fly overnight. We did that for the Department of Medical Assistance Services — they had to move out of their building. But we’ve been able to work through it and we’ve been able to manage it in a very secure manner. One of the lessons learned, which other states have said, is to not compromise cybersecurity for this type of thing. And we’ve been able to do that. 

STATETECH: What do you anticipate are going to be the long-term consequences, either from a continuity of operations perspective or just a state workforce cultural perspective?

MOE: The new working arrangement — what that looks like going forward, I really don’t know. But the conventional thoughts I’ve seen kicked around are that there’d be some expectation of working from home because of a virus event. 

People realize they can get their work done, they can minimize their cost structure and their physical footprint from a rent perspective. And it helps with disaster recovery because you’re not all in one spot. 

There’s a significant number of human resources policies that have to be fine-tuned. Onboarding is difficult. Creating a culture is more difficult, at least from my perspective, but it can be done. And we’ve had an all-hands meeting for my staff. It went really well. You want to see more interaction, but in the absence of physical contact in one big room, I think it’s the new norm, or at least part of the new norm. 

As we start bringing people back in, we will do it slowly and measuredly. We would maintain the social distancing, maintain the best practices and make sure people are safe and felt safe. And we communicate all that, how that’s being done. And, of course, we follow the best practices put out by the administration.

Check out this page for more coverage from the NASCIO Midyear 2020 conference, and follow us on Twitter at @StateTech, or the official conference Twitter account, @NASCIO,  and join the conversation using the hashtag #NASCIO20.

PHOTOGRAPHY BY KHUE BUI