Twitter Could Be the Next Predictive Analytics Platform

Chicago is finding all kinds of ways to use Twitter data for public safety. StateTech previously reported on an initiative that mined Twitter data for mentions of foodborne illness. That project, known as Foodborne Chicago, examines tweets for reports of sickness and then contacts users who claim to have gotten sick at a local restaurant. That information is passed along to the Department of Public Health, and the restaurant could be inspected as a result.

Researcher Matthew Gerber of University of Virginia's Predictive Technology Lab is upping the ante. In a recent experiment, Gerber collected 1.5 million tweets from Chicago neighborhoods and compared the content of the tweets with crime reports in the same areas.

The Atlantic City’s Eric Jaffe explains the experiment in greater detail:

Next Gerber created a computer algorithm that separated the tweets into 1 kilometer by 1 kilometer neighborhoods, then analyzed the content of the tweets in each neighborhood to find out what people were tweeting about. The content was then lumped into hundreds of "topics." For instance, the foremost topic in the neighborhood around Chicago O'Hare pertained to travel, with tweets including words like gate, plane, flight, and of course, delayed.

Things get a bit technical from here. In basic terms, Gerber's model compared topics in a neighborhood to the historical crime data from that same spot in the city for a given month. The model formed correlations between topics and crimes, then used those correlations to predict crime in the same neighborhood for a subsequent month. The method is similar to the way Google Flu Trends uses search terms to predict outbreaks.

The results are impressive. Of the 25 crimes committed in the measured areas during the experiment, Twitter “did a better job predicting them than did the historical crime data alone.”

This is one of many ways Twitter data — which is essentially open data when users choose to display content and location — can be leveraged by local governments. This experiment was not associated with the Chicago Police Department, but other law enforcement units are already fighting crime with help from the data center. Major initiatives in Memphis, Albuquerque and Miami are already producing results.

How is your city using Big Data? Let us know in the Comments.