Reality Check

More polling: methodology, online surveys, Nick Clegg

Mon, 25 Apr 2011 16:56:15 -0500

So how do polling companies come up with their numbers? It turns out, they all do it differently, which is one big reason why there is so much variation.


Despite the concerns about unrepresentative samples caused by low response rates and the exclusion of cellphone users, many companies still poll the old fashioned way: they call a randomized sample of people on the telephone.


For example, Nanos Research, which polls for the Globe and Mail and CTV, does 400 new phone interviews every night. Given the broad industry refusal rate of 85 per cent, that means Nanos might be dialling close to 2,600 numbers a day just to reach the 400 respondents it needs.


The Nanos poll is called a "tracking poll," which means the company takes a three-day running sample. Every day, the first 400 respondents are dropped off and a new batch is added, which allows the three-day tally to be based on a randomized sample of 1,200 people.


This yields a respectable margin of error of 3.1 per cent either way, 19 times out of 20.


Like most pollsters, Nanos addresses the concerns about over-representation of older Canadians and under-representation of younger ones, which arise with telephone polling, by "weighting" the result according to age, based on the 2006 census.


With a sample size of 1,200 people, Nanos can provide reliable results on a national level. But beware of drawing any conclusions about provincial results - the sample sizes are extremely small.


The NDP 'surge' in Quebec


Let's look at the NDP "surge" in Quebec, which Nanos and others have been reporting for the past few days.


Those Quebec results have been based on very small samples with very high margins of error.


For example, Nanos reported that its survey of April 17-19 showed the NDP had moved solidly into second place in Quebec, with 25 per cent of the vote, versus 21 per cent for the Liberals and 17 per cent for the Conservatives.


That result was widely reported in the press. What was less widely reported was that the result was based on a sample of only 164 people, with a margin of error of 7.8 per cent.


So depending on how the margin of error was distributed, the NDP was either solidly in second place, or still languishing in fourth.


Over time, these provincial findings may reflect genuine trends. But to create a headline story out of any single result, is to read into polls more than they are capable of delivering.


Online polls


Increasingly, over the past few years, many polling firms have been abandoning the telephone altogether in favour of the internet.


Until recently, online polling was viewed as a poor cousin to randomized phone surveys, too fraught with potential trouble spots to be considered very reliable.


But as the methodological concerns about telephone polling increase, and the worries about the accuracy of internet polling subsides, internet polls are now clearly the wave of the future.


In 2008, it was an internet pollster, Angus Reid, that came closest to calling the outcome of the federal election.


One of Canada's biggest internet polling firms is Leger Marketing, which is currently polling for Sun Media.


Leger draws on a database of more than 350,000 Canadians who were recruited by telephone surveys to be part of the company's market research operation.


It is a very large sample, but it is not randomly selected. For their political poll, conducted between April 15 and 17, Leger surveyed more than 3,500 people and reported a margin of error of just 1.7 per cent.


Its Quebec sample of nearly a thousand voters was by far the largest of any polling company, and had a margin of error of 3.1 per cent.


It continued to show the Bloc clearly ahead of the NDP, 34 per cent to 24 per cent, with the Conservatives and Liberals tied at 20 per cent. It is the only Quebec poll where the top three positions lay outside the margin of error.


But online polling has its critics. One of them, Paul Adams of Carleton University, told the Canadian Press that online surveys tend to tap disproportionately into young, urban, internet savvy voters, who tend to favour the NDP, but whose support tends to be softer and whose commitment to showing up at the polls on election day is perhaps not as strong as other groups.


The industry's regulatory body, the Market Research and Intelligence Association, also has some reservations.


It asks its members to refrain from reporting margins of error on polls where probability samples are not used, although many internet pollsters, including Leger, continue to do so.


The MRIA has also strongly defended its members, too, noting that opinion surveying is a billion dollar industry in Canada with a very successful track record.


Word to the wise


So what to make of all this?


The NDP "surge" may indeed be real, but it is probably too early to tell for sure. Be wary of jumping to conclusions based on a small handful of polls, particularly in any one province, where the samples are small and the margins of error are large.


If you want to be a savvy consumer of polls, rather than a passive recipient of poll spin coming from journalists, pollsters or politicians, keep the following points in mind:


Don't trust any poll that doesn't tell you how the poll was conducted, the sample size, and the margin of error.

Be aware of the strengths and weaknesses of online and telephone polling.

You are better off putting your faith in the aggregated results of several polls conducted over time, rather than the findings of any one single poll or pollster.

Stay away from seat projection polls, or polls that purport to show how any single riding might vote. There is no scientific validity to those polls.


And finally....


Two words that should keep NDP supporters awake at night: Nick Clegg.


Nick Clegg is the leader of the Liberal Democrats in the U.K. and, during last spring's British election, he became the darling of British pollsters.


After a strong performance in the leaders' debates, the perennial third-party finisher shot past the ruling Labour Party into second place in the polls.


Some polls even had him challenging the Conservatives for first place. After receiving 22 per cent of the vote in the previous election, Clegg was now polling at over 30 per cent.


The last polls taken before election day predicted the Liberal Democrats would capture at least 27 per cent of the vote and challenge Labour for second place.


But when the votes were counted, the Liberal Democrats received only 23 per cent, six percentage points behind Labour; the party actually lost five seats from its previous total.


Either voters had swung away from the Liberal Democrats at the last moment, or the pollsters had been wrong about the party's strength all along. It's impossible to know.


As for Nick Clegg, rather than leading a strong contingent of Liberal Democrats on the opposition benches, and positioning himself as Britain's prime minister-in-waiting, he and his 57 MPs are junior partners in an increasingly unpopular and fractious coalition government with the ruling Conservative Party.


Nick Clegg. Jack Layton's worst nightmare?

« Previous Post | Main | Next Post »