Reality Check

Polling, accuracy and the contest for bragging rights

Tue, 26 Apr 2011 18:04:30 -0500

Everyone expects this race to be close, perhaps decided by just a fraction of a percentage point. There are several strong candidates. But none belongs to a political party.


This is a race between pollsters, not politicians.


For Canada's highly competitive, billion-dollar market research industry, election night is the equivalent of Grey Cup Sunday.


When the votes are all counted, which polling firm will have come closest to getting the numbers right, and how close will that be?


Public opinion polling has been around since the 1930s and has been shown to yield remarkably accurate results, 19 times out of 20.


But for many people, polling still carries the whiff of snake oil.


How can 1,000 people, chosen at random, represent the views of 33 million? How can a pollster know what I'm thinking when none of them have ever asked me?


On the line


That's why the polling industry's credibility is on the line every election campaign.


Many people wouldn't mind seeing pollsters get knocked down a few notches. They can be irritating and arrogant on TV, pontificating about what "Canadians" are thinking.


If nothing else, we want to show that we're not as predictable as we might appear.


Indeed, that almost happened in 2004.


That was not a good election for Canada's pollsters. All the big firms seriously over-stated the Conservative and NDP vote, while under-estimating the Liberals by amounts that lay outside their margins of error.


The NDP won 15.7 per cent of the vote in 2004, far less that the 19 per cent predicted in the final Ekos poll, and the 20 in the SES (now Nanos) one.


The Liberals ended up with 36.7 per cent of the vote, not the 32 per cent predicted in the final Ipsos poll.


Snapshots in time


Despite these results, the polling industry was unapologetic.


Polls, they reminded us, are "snapshots in time." They are designed to tell you what people are thinking when they are asked a question, and are supposed to have no predictive value.


This can be quite handy when doing political polling. Pollsters stop polling a day or two before the voting starts. So if the actual results don't match the latest polling data, don't blame the pollster.


Maybe they're right. Maybe large numbers of people did change their minds at the last minute. We don't know. But it must be nice to be in a business where you can never be proven wrong.


Better results lately


If you are keeping score, the past two campaigns have been better for the pollsters.


2006 was a particularly good year. Nanos won the crown for most accurate polling. Its last poll, taken the day before the vote, came within 0.01 per cent of the final totals for all four major parties.


You can't get much closer than that. Other major polling firms were not far behind.


2008 was not as good. Bragging rights that time went to Angus Reid. Its final, online poll proved to be the most accurate. But it was still about a percentage point off the actual vote total for each of the parties.


Most of the other companies floundered when it came to the Conservative vote, under-estimating it by around three percentage points.


In the spotlight


This year's election campaign comes at a time when the industry is under scrutiny as never before.

A few months ago, two of its leading lights, Allan Gregg of Harris Decima and Frank Graves of Ekos, complained to the Canadian Press that the industry needed to address serious methodological challenges if it hoped to remain credible.


In response, the industry's trade association, the Market Research and Intelligence Association, took out a full-page ad in Ottawa's Hill Times newspaper declaring its "confidence in the results of our polling and in the value that we provide to Canadians."


But the reality is that the polling business is changing dramatically.


Randomized telephone polling, which has been the backbone of the industry for decades, is rapidly giving way to online polling. See my earlier post.


This may be the last campaign where the telephone is the primary instrument of choice for pollsters. Everyone in the industry is scrambling to figure out where things are heading next.


Different approaches


For proof, you just have to look at the wide range of approaches that three of Canada's leading pollsters are employing.


Nanos continues to use randomized telephone surveys.


Ekos also uses the telephone, but it has replaced humans with machines. People who receive an Ekos robo-call are asked to punch in their answers to machine-generated questions on the keypads of their telephones.


Angus Reid uses an online survey. It randomly selects 2,000 people who have been non-randomly selected to participate in the company's marketing surveys, and asks them to answer questions about their political preferences.


All of the companies ask some variation of the "If a federal election were held today who would you vote for" question, but again, there are important differences.


Nanos asks people to name their favourite party. Unlike the others, it doesn't prompt them by providing the names.


All of them follow up the voter preference question with other questions designed to test, among other things, how strong the respondent's commitment is to their chosen party, and how likely they are to vote.


After all, with voter turnout now hovering below 60 per cent, roughly four out of every 10 people surveyed by pollsters probably won't wind up voting at all.


Figuring out who those people are, and properly factoring them into the equation, is critical to arriving at an accurate number.


All of these questions and methodologies reflect the latest in social science research and all are focused on one objective: to come closest to predicting how Canadians will vote on May 2.


Right now, there are still some fairly wide gaps among the polls, up to five percentage points difference in assessments of the NDP as well as the Conservative vote.


Over the next few days, you can expect to see those gaps narrow, as voters firm up their decisions, and pollsters increase the number of people they survey in order to reduce their margin of error.


It's a high-stakes, high-profile game that no one wants to lose.


Which is why politicians will not be the only people watching with sweaty palms as the votes are counted on Monday night.

« Previous Post | Main | Next Post »