What's the connection between a lower crime rate, the price of shampoo at your local big box retailer, the stock market and next quarter profits?

The answer, surprisingly, is data mining, combing through statistics and details to establish a relationship between the numbers and other information.

Data mining isn't new, but a convergence of faster and cheaper hardware, robust and flexible software and deflating storage costs are making it possible to not only crunch more numbers at a time but to get the answers almost instantaneously.

The upshot is, managers can make mid-course adjustments in their strategies to either limit losses or maximize profits. Until recently, only global scale enterprises like Wal-Mart could afford the technology — methodically tracking which items on which shelves sold at which stores and at what time, leveraging the data to ensure product was constantly replenished at the right rate for local sales and thus controlling costs on everything from shampoo to baby formula.

What's good for Wal-Mart is good for the nation and that business intelligence strategy is now turning up all over the place.

"The issue for a lot of companies is how to get more value from the data they have," notes Anne Milley, director of analytics for SAS, a major player in supplying the systems behind the technology. "The answers are all in the numbers, and in doing so the ripples are spreading out to crime detection, anti-terrorism, air quality monitoring and surveillance."

From mapping the human genome, to bringing drugs to market faster, to cutting costs in servicing ATMs, to better predicting hurricanes, the need to capture more data and compare it to a wider range of variables is growing exponentially, says Milley. And public policy is seeing some benefits.

Policing is a prime example. Earlier this year a bedazzled Rodney Munroe, chief of the Richmond, Va. police service, accepted the 2007 Business Intelligence Award for Excellence from Gartner, a leading information technology research and advisory company, at a black tie Chicago event.

The award usually goes to innovative business applications, but judges were impressed with how Munroe turned analytics into a formidable crime-fighting tool.

When Munroe took over as chief two years ago, his department was drowning in crime and data. Police had a mass of data from 911 calls and crime reports; what they didn’t have was a way to connect the dots and see a pattern of behaviour.

Using some sophisticated software and hardware they started overlaying crime reports with other data, such as weather, traffic, sports events and paydays for large employers. The data was analyzed three times a day and something interesting emerged: Robberies spiked on paydays near cheque cashing storefronts in specific neighbourhoods. Other clusters also became apparent, and pretty soon police were deploying resources in advance and predicting where crime was most likely to occur.

Coupled with some other technological advancement, such as surveillance videos wirelessly transmitted to patrol cars, major crime rates dropped 21 per cent from 2005 to 2006. In 2007, major crime is down another 19 per cent.

It's being able to connect those dots which was behind the thinking in an IBM project in Cape Breton last year, which looked at how technology could start to displace the traditional paper and pen logistics of policing.

Working with Cape Breton police — who cover a small but geographically challenging jurisdiction — and Cape Breton University, the project explored how the vastly diverse variables involved in crime investigation could be efficiently put into a database and then mined.

Crime by nature is chaotic. Minutia may seem trivial one day but could be pivotal the next. The challenge for technology was not only to create uniform reporting, but to find ways to search the interviews, reports and evidence gathered, and to also include esoteric materials such as bank camera surveillance and voice recordings of phone calls or wiretaps, which are much more challenging to search.

And that's where the cutting edge of the research is going, into developing ways of transcribing voice to text, and to tagging videos to allow them to be searchable within a database, along with all the other evidence.

Since that project wrapped, IBM has donated $5 million to Simon Fraser University's Institute for Canadian Urban Research Studies to create a crime prevention and analysis lab.

RCMP research chairs, and husband-and-wife team, Drs. Patricia and Paul Brantingham, will investigate how factors as diverse as a city design, the layout of road networks, and shopping mall hours affect the location, frequency and severity of urban crime.

"This is what Richmond is doing, but on steroids," said Paul McCullough, IBM's project manager. "We want to be able to look at years of crime information from a variety of police databases and find the connections and prove it out in a predictive sense." The irony of course, is that in business, and in public safety and policy, we may have known the answers to some of the biggest dilemmas and challenges facing us all along. We just couldn't always see them clearly because the mass of data obscured and intimidated us.

Technology may just be the solution to recognize the difference between the forest and the trees before the opportunity to change things has passed us by and we get lost.