Large Vision Models: What Are They, and How Can Agencies Use them?

What Are Large Vision Models?

An LVM is an AI model that analyzes visual data, such as still images and videos. Trained on massive data sets, LVMs learn how to recognize objects and patterns. This is useful for such tasks as object detection and image classification.

Based on complex machine learning models, LVMs “analyze visual inputs and generate a likely set of outputs,” says Adam Rabin, senior product marketing manager for video security at Verkada.

“You can think of them as the visual counterpart to large language models, which process vast amounts of text-based data,” says Matthew Dietz, global AI leader at Cisco. Unlike LLMs, which focus on words, LVM are “trained specifically to process and understand visual data like images, videos and diagrams.”

This capability has significant potential for state and local organizations.

EXPLORE: Moving physical security to the cloud enhances surveillance.

Key Use Cases for State and Local Agencies

Public agencies can leverage LVMs across a range of use cases.

In public safety and emergency response, for example, “the LVMs can analyze disaster imagery to assist emergency responders, to support search-and-rescue operations, and provide actionable insights in real time,” Dietz says.

Law enforcement can benefit, as well. 

“Suppose there’s a suspicious vehicle on the loose,” Rabin says. “Someone caught a glimpse of what they would describe as a 1970s red muscle car with a white stripe on it. Those are a handful of very specific attributes.”

Given that description, “and perhaps even getting even more granular, say there’s a bumper sticker or a dent on the side,” a local police department could leverage an LVM-supported tool to search traffic camera videos and other sources, he says.

LVMs likewise can help with routine traffic management. Planners could use them to understand “what the best routes are that we should take as we design our roadways, and also to make sure that we’re reducing the amount of wear and tear along these roadways,” Dietz says. “We’ve all dealt with traffic congestion in our lives. LVMs could potentially impact the traffic flow.” 

DISCOVER: Balance speed and practicality at the network’s edge.

State and local agencies and their partners can also tap into LVMs for infrastructure and asset management. For example, “satellite imagery or drone imagery is helping to inspect the critical infrastructure, monitor road conditions and assess damage after things like natural disasters,” Dietz says.

“Imagine a utility company that is responsible for thousands of miles of power lines or pipelines or infrastructure spread across a vast area,” he says. “With LVMs, drones equipped with high-resolution cameras could fly over all of these assets, capture all of these images in minutes, and then analyze that visual data in real time to detect things like cracks in pipelines or corroded power lines.”

Technology Solutions for Leveraging LVMs

To harness the power of LVMs, state and local agencies likely will take advantage of commercial products that, in turn, depend on open-source models.

“It wouldn’t be practical for state and local agencies to build their own LVMs, because there are already so many great open-source ones that have been trained on billions of inputs, with billions of parameters,” Rabin says.

Commercial products use open-source LVMs to enable end users to search for and analyze video footage. Verkada, for example, combines LVMs with LLMs, enabling users to enter text that describes what they are seeking, whether that’s a suspect’s vehicle or a downed power line.

When looking to operationalize a commercial LVM-driven solution, it’s important to think about the technology infrastructure. “These LVMs demand immense computing power, low-latency networks and airtight security to ensure that they are being used responsibly,” Dietz says.

For state and local, latency is a key consideration: Whether it’s law enforcement or disaster response, speed is a factor in the usefulness of an LVM solution. Security also matters, especially for law enforcement and other government agencies handling potentially sensitive visual data.

At Cisco, “we provide the advanced compute, networking and storage infrastructure needed to power these large vision models alongside our industry-leading security to safeguard all of the AI operations,” Dietz says.

Considerations for Introducing LVMs in Public Sector Projects

LVMs offer tremendous potential, “but it’s also important to consider some very important factors for successful implementation,” Dietz says. First, there’s the technical infrastructure. “Organizations need the right compute, storage and network capacity to process the vast amounts of data that are being generated.”

To make use of video data, agencies need to have it readily available. That means “ensuring that you have the onboard retention for whatever deployment or series of deployments you’re looking at,” Rabin says.

SUBSCRIBE: Click the banner to get weekly updates.

It’s also important to evaluate “the total cost of ownership, from training to maintenance of these models, as well as scalability,” Dietz says. In addition, state and local agencies “need to ensure you have the necessary skills on staff to manage these LVMs effectively.”

Another big consideration is privacy and security. “Handling visual data raises many privacy concerns, so it’s important to make sure that you are in compliance with regulations,” he says. “Strong data protection measures are essential to leveraging these LVMs.”

Overall, agencies using LVMs need to think about “doing this in a way where you don’t come off as too surveillance-y or too Big Brother-y,” Rabin says.

As with any AI-driven application, state and local organizations need to be sensitive to the need to build and maintain public confidence.

RELATED: AI transforms the citizen experience. 

For example, “if there’s a high-profile situation — a suspect on the loose — you are explaining to the public that you have identified the suspect’s vehicle” using AI tools, he says. “There are ways of instilling public trust through transparency in the process of getting to what you wanted to get to.”

AI Isn’t Ready for Some Use Cases

LVMs, in general, aren’t ready to perform complex tasks — operations that demand a deep understanding of context. They can’t offer nuanced interpretations.

For example, an LVM might be able to detect a basic emotion — happy or sad — but it likely won’t pick up on sarcasm or other subtle expressions. And it may not grasp context: In an ambiguous situation, it’s up to humans to understand precisely what is happening.

As with any AI-driven solution, an LVM application requires human guardrails, especially when used by state and local organizations. For the public sector, “ethical considerations come into play,” Dietz says. “Biases in these LVM systems can impact equitable service delivery. Transparency about how this visual data is being used is critical for public trust.”

Despite the current limitations, LVMs can be a powerful ally for state and local agencies, as well as public utilities and other key stakeholders, as they look to drive mission-focused action in response to visual inputs.