FBI seeks social media data mining tool
The Associated Press
Posted: Feb 13, 2012 9:06 AM ET
Last Updated: Feb 13, 2012 10:37 AM ET
Related
Related Stories
External Links
(Note:CBC does not endorse and is not responsible for the content of external links.)
The FBI says the system is only meant to monitor publicly available information and would not focus on specific individuals or groups but on words related to criminal activity.
(Reuters)The U.S. government is seeking software that can mine social media to predict everything from future terrorist attacks to foreign uprisings, according to requests posted online by federal law enforcement and intelligence agencies.
Hundreds of intelligence analysts already sift overseas Twitter and Facebook posts to track events such as the Arab Spring. But in a formal “request for information’’ from potential contractors, the FBI recently outlined its desire for a digital tool to scan the entire universe of social media — more data than humans could ever crunch.
The Department of Defense and the Office of the Director of National Intelligence also have solicited the private sector for ways to automate the process of identifying emerging threats and upheavals using the billions of posts people around the world share every day.
“Social media has emerged to be the first instance of communication about a crisis, trumping traditional first responders that included police, firefighters, EMT, and journalists,’’ the FBI wrote in its request. “Social media is rivaling 911 services in crisis response and reporting.’’
The proposals already have raised privacy concerns among advocates who worry that such monitoring efforts could have a chilling effect on users. Ginger McCall, director of the open government project at the Washington, D.C.-based Electronic Privacy Information Center, said the FBI has no business monitoring legitimate free speech without a narrow, targeted law enforcement purpose.
“Any time that you have to worry about the federal government following you around peering over your shoulder listening to what you’re saying, it’s going to affect the way you speak and the way that you act,’’ McCall said.
Monitors publicly available info only
The FBI said in a statement to The Associated Press that their proposed system is only meant to monitor publicly available information and would not focus on specific individuals or groups but on words related to criminal activity.
Analyzing public information is nothing new in the world of intelligence. During the Cold War, for example, CIA operatives read Russian newspapers and intercepted television and radio broadcasts in hopes of inferring what Soviet leaders were thinking.
But the rise of social media over the past few years has dramatically changed both the kinds and amount of freely available information. For example, Twitter CEO Dick Costolo said at a recent conference that users of the micro-blogging service send out an average of one billion tweets every three days.
“It really ought to be the golden age of intelligence collection in that you’ve got people falling all over themselves trying to express who they are,’’ said Ross Stapleton-Gray, a former CIA analyst and now a technology consultant who advises companies on security, surveillance and privacy issues.
The system sought by the research arm of the national intelligence director’s office would fuse together everything from web searches to Wikipedia edits to traffic webcams to “beat the news’’ by predicting major events ranging from economic turmoil to disease outbreaks.
The Defense Department’s tool would track social media to identify the spread of information that could affect soldiers in the field and also give the military ways to conduct its own “influence operations’’ on social networks to counteract enemy campaigns.
The intelligence director’s office and the Defense Department said they could not meet the AP’s deadline to answer specific questions about the proposed projects.
The FBI is seeking a web app that would automatically scrape social networks for data that could alert the agency’s operations center to breaking crises as they happen and plot them on interfaces like Google Maps
For such systems to work well, their developers would have to overcome several technological challenges, the easiest of which is handling the massive amount of data involved.
Developments in so-called “cloud computing’’ have made processing big data sets easier than ever before by spreading the work broadly across networks of computers.
Major hurdle: Understanding human language
Instead, experts in the field say the major hurdle is in effect teaching computers how to read. To sift the valuable information from the mundane, the software must understand the subtleties of meaning in tweets and blog posts to tell the difference between, for example, a serious statement and a joke.
Solving such problems falls to researchers in fields such as natural language processing and computational linguistics — the same specialties that brought the world the iPhone’s Siri voice-activated assistant and IBM’s Watson, which trounced its human opponents at Jeopardy.
Authenticity also becomes an issue in analyzing social networks. Computer programs known as “bots’’ already plague services such as Twitter with junk posts similar to email spam. Researcher Tim Hwang has scripted his own bots to see how much influence they could wield over social networks and says the ability to create bots that closely mimic humans will only improve over time.
This matters in intelligence gathering because bots could fool analysts — and their software — into thinking they’re witnessing a genuine shift in social trends that in reality could be a government propaganda campaign driven by, for example, Twitter users that don’t really exist.
“We have all the data. How do we know what’s real and what’s not?’’ Hwang said.
William McCants, an analyst at the Center for Naval Analyses and a former State Department official, monitors al-Qaeda propaganda online. He said he worries that the systems the FBI and other agencies are seeking could create an overreliance on technology at the expense of carefully trained human analysts who are still better at zeroing in on the facts that matter most.
“The more data you use and the more complicated the software, the more likely it is you will confirm a well-known banality,’’ McCants said a friend likes to joke. “You didn’t need to be on Twitter to know that a revolution was happening in Egypt.’’
Share Tools
Top News Headlines
- Kids from levelled Oklahoma schools recount deadly tornado

- Children from two Oklahoma schools levelled Monday by a powerful tornado are recounting what it was like to survive the "loud" and "scary" twister, while rescuers near the end of their search for any other remaining survivors or bodies.

more »
- Deadly Oklahoma tornado confirmed as most powerful type

- Emergency workers neared the end of their search Tuesday afternoon for survivors in Moore, Okla., following a deadly tornado that weather officials said was now classified among the most powerful type of twister. more »
- Senate debates expense audits amid greater scrutiny
- The expenses scandal dominated the first Senate session since the audits on senators Mike Duffy, Mac Harb and Patrick Brazeau were released and it was revealed Duffy's questionable expenses were repaid by a personal cheque from the prime minister's chief of staff. more »
- Only 1 set of human remains found at Millard farm, police say
- Hamilton police have confirmed that they are dealing with only a single set of human remains at the Waterloo region farm of Dellen Millard. more »
- Rob Ford faces more calls to address crack allegations
- Toronto Mayor Rob Ford went back to work after a holiday weekend, but he kept his mouth shut about an alleged video that two published reports say shows him smoking what appears to be a crack pipe. more »
Must Watch
Latest Technology & Science News Headlines
- Designing smart clothes to go with that smartphone
- Dresses adorned with flowers that slowly open and close or coloured patterns that change spontaneously are some of the futuristic designs by a Montreal researcher who is trying to make clothes "smarter." more »
- Microsoft's Xbox revamp: Is the sun setting on game consoles?
- With the rise of mobile and social games, the revival of PC gaming and a general proliferation of options for both developers and players, some are wondering whether game consoles matter anymore, writes Peter Nowak. more »
- Vancouver link to Hadfield's space guitar
- A Vancouver company says it will re-start production of a guitar that was used by Chris Hadfield in space, prompting thousands of dollars in new orders. more »
- Netflix and the rise of binge TV watching
- Netflix has been giving viewers the opportunity to watch entire new seasons of TV shows in one sitting and — for better or for worse — many have been doing just that. more »
Bob McDonald's Blog
Chris Hadfield: The gravity of gravity May. 17, 2013 9:58 AM After five months of being Superman and a media superstar, Canadian astronaut Chris Hadfield is now beginning the challenging task of adapting his mortal body and brain to life back on Earth.
Quirks & Quarks
- May 18: Apps for Apes May. 21, 2013 1:43 PM Scientists at more than 2 dozen zoos around the world, including the Toronto Zoo, have been using computer tablets to stimulate our bright orange primate cousins, the orangutans. And the orangutans have been loving it.
Latest Features
- Deadly Oklahoma tornado confirmed as most powerful type
- Microsoft unveils Xbox One
- 'Very upset' Harper wants fast Senate spending reform
- Only 1 set of human remains found at Millard farm, police say
- Kids from levelled Oklahoma schools recount deadly tornado
- Rob Ford faces more calls to address crack allegations
- Mountie sues 13 ex-colleagues for sex assault, harassment
- Jodi Arias asks jury to spare her life
- Microsoft's Xbox revamp: Is the sun setting on game consoles?

