Pollsters grapple with methodology, impact of election polls
Polls seen to have had a 'major impact on the outcome' of last federal election
The Canadian Press
Posted: Sep 26, 2011 8:57 AM ET
Last Updated: Sep 26, 2011 11:35 AM ET
Pollster Darrell Bricker, the president and CEO of Ipsos Reid, shown here in a file photo, co-wrote an open letter to journalists covering the Ontario election earlier this month, sparking a fierce debate about polling methods and reporting standards for public opinion research. (Darren Calabrese/Canadian Press)
What if there had been no public opinion polls published during last spring's federal election campaign?
Would the NDP's orange wave have swept across the country if no one realized there was a wave to catch? Would support for the Liberals have collapsed so utterly? Would the Conservatives have captured their long-sought majority?
Such questions are not simply idle speculation.
Given the controversy swirling around the accuracy and methodological adequacy of polls, the professionalism of pollsters and the polling literacy of journalists who slavishly report them, questions about the influence of polls on election results are arguably key to the health of Canada's democracy.
The issue arose at a conference last week, where eight pollsters from the country's most prominent public opinion research firms discussed the lessons they've gleaned from last May's election.
"I do think this was an interesting election in that it's hard to deny that polls themselves had a major impact on the outcome of the election," said Derek Leebosh of Environics.
He said a CROP poll published shortly after the televised leaders' debate and showing the NDP had leapt into a commanding lead in Quebec was like "a depth charge" exploding in the campaign.
"Imagine if there had been no polling in the election campaign at all, nobody would have known that this phenomenon in Quebec was happening and the orange crush would never, may not, have spread into the rest of the country and even within Quebec people might not have thought the NDP was a realistic option."
Ekos Research's Frank Graves countered there's "strong evidence" the outcome wouldn't have changed much had there been no polls. Indeed, he argued there was no post-debate NDP surge, that his surveys showed NDP support "proceeding on a pretty placid, straight (upward) line" throughout the campaign.
Regardless, Graves maintained voters are "not that dumb;" even without polls they would've noticed the explosion of NDP lawn signs and anecdotal evidence of New Democrat popularity.
In any event, he said post-election polling found the vast majority of Canadians maintained they weren't influenced by the polls and, among those who were influenced, there was no clear pattern favouring one party over another.
Environics' Kevin Neuman was doubtful.
"People may say that (polls) don't influence, but it would influence the media and how the media cover the story and frame the story," he said, adding that the CROP poll "may have completely changed the media coverage."
While they disagreed about the impact of polls, there was consensus among pollsters at the conference that media coverage of them is often sorely wanting. Journalists, they agreed, are riveted on the bald horse race numbers, disregard issue-based surveys, misinterpret margins of error and make no distinction between proven and unproven polling methods.
The discussion was in many ways a polite echo of the rocket launched a couple weeks ago by Darrell Bricker and John Wright, top honchos at Canada's largest polling company, Ipsos Reid.
The duo penned an "open letter" to journalists covering the Ontario provincial election campaign, warning them that "some marginal pollsters" are counting on media ignorance and competitiveness to peddle "inferior" polls. They accused some unidentified pollsters of becoming "hucksters selling methodological snake oil" and some media outlets of publishing questionable polls simply because they support their editorial position.
"All of this MUST stop," the duo wrote. "We are distorting our democracy, confusing voters and destroying what should be a source of truth in election campaigns -- the unbiased, truly scientific public opinion poll."
Graves pointed out that pollsters used to have stable relationships with specific media outlets that paid well for quality surveys. Today, he said, it's become a "sort of auction to the bottom," with pollsters giving their research to the media for free as a publicity tool. As a result, the quality of polls and the level of "methodological fluency" among journalists reporting on them have plunged.
It's not that pollsters are knowingly peddling "crappy" polls, he insisted. It's just that they can no longer afford to do surveys with the "level of depth and rigour that we'd really like to" because "polling budgets today are a fraction of what they used to be." And journalists don't seem to notice the difference.
"The understanding of basic issues -- like what is a margin of error, or how do you create a good sample -- is dramatically lower than it was back when they had really smart (media) guys ... who knew as much as the pollster."
Last winter, veteran pollster Allan Gregg publicly aired similar concerns in an interview with The Canadian Press -- sparking an angry backlash from some fellow pollsters, including Bricker and Wright.
At the time, Graves proposed the idea of a "blue-ribbon consortium" of public opinion researchers who could apply the highest standards to produce high quality political surveys "as a public service for the industry." No one took him up on it.
"The whole industry might come (out) ahead, much more so than this internecine thing that's emerged in the last year or two," he said.
That internecine squabbling has revolved largely around the different methodologies employed as pollsters grapple with how best to achieve a random sample of the entire population in an age when people vet their phone calls or, in many cases, use only cell phones. Some have moved to online polls, which involve self-selected respondents who sign up to be polled, or, like Ekos, now use automated phone surveys or "robo calling."
Bricker and Wright maintained in their letter that both robo-calling and online polls produce skewed results. But at the conference, most participants agreed that traditional phone and online polls in last May's election produced similar, relatively accurate results. Ekos underestimated Conservative support but, Graves argued, that wasn't a methodological problem so much as it was an inability to anticipate the disproportionate turn-out of Tory voters.
Whatever the drawbacks of the alternatives, it would seem traditional phone surveys are going the way of the dodo bird.
Ipsos representative Mike Colledge argued that no single method can produce a sample that reflects 100 per cent of the population and suggested that mixed polling methods are the way of the future.
Leger pollster Richard Hobbs predicted that by the next federal election, very few pollsters will still be conducting phone surveys.
Graves said he's no pessimist; he's confident the industry will develop ways to ensure quality polling data and develop more poll-literate media partners.
"It's not a high-water point for us right now, let's be blunt. But that doesn't mean we should give up."
Share Tools
Power & Politics' Ballot Box question by Rosemary Barton May. 17, 2013 3:52 PM Should Mike Duffy give up his Senate seat?
Top News Headlines
- WHO concerned coronavirus spreading person to person
- The World Health Organization has issued a blunt assessment of the coronavirus outbreak in Saudi Arabia, acknowledging for the first time that there are concerns the virus may be spreading from person to person, at least in a limited way. more »
- Toronto mayor cancels weekly radio show
- Toronto Mayor Rob Ford will not be hosting his weekly radio show this weekend after explosive allegations that he was recorded on video appearing to smoke crack cocaine. more »
- Senator Pamela Wallin leaves Conservative caucus
- Senator Pamela Wallin says she is recusing herself from the Conservative caucus while her travel expense claims are under scrutiny. Wallin's departure comes one day after Senator Mike Duffy left the Tory caucus amid controversy over his expense claims. more »
- Body found after fishing boat capsizes off New Brunswick
- A man's body has been found after a lobster fishing boat capsized off the eastern coast of New Brunswick. more »
Must Watch
Latest Politics News Headlines
- Senator Pamela Wallin leaves Conservative caucus
- Senator Pamela Wallin says she is recusing herself from the Conservative caucus while her travel expense claims are under scrutiny. Wallin's departure comes one day after Senator Mike Duffy left the Tory caucus amid controversy over his expense claims. more »
- First Nations schools report points to education gap
- First Nations' schools have lower quality teaching, an inferior curriculum and fail to provide proper services for children with special needs — and without further investment these problems could worsen with an expected population spike on reserves, a new federal report warns. more »
- Duffy's Senate expenses may get 2nd look from auditors
- Senator Mike Duffy's expenses may get a second review by independent auditors following media reports regarding expenses he claimed while campaigning for Conservative candidates during the last election. more »
- Chris Hall: Senator Duffy and the little matter of accountability
- A $90,000 'gift' from Stephen Harper's chief of staff to Mike Duffy didn't fix the political problem over the senator's questionable expenses, Chris Hall writes. It just made matters worse and opens the door to questions about prime ministerial accountability. more »
The National
The House
- Questions mount for Harper and chief of staff Nigel Wright in Senate scandal May. 18, 2013 1:15 PM This week on The House, with Senators Wallin and Duffy now out of the Conservative caucus, we get reaction from NDP Ethics critic Charlie Angus. We also hear directly from Senator Patrick Brazeau who says the Conservatives have thrown him under the bus. Plus we speak with B.C. Premier Christy Clark after her stunning victory.
- Senator Pamela Wallin leaves Conservative caucus
- Toronto Mayor Rob Ford denies crack cocaine allegations
- Milwaukee bar wins overturn of bra ban
- Tim Bosma public memorial Wednesday in Hamilton, Ont.
- Public raising funds to buy alleged Rob Ford crack video
- Dennis Oland named as prime suspect in father's slaying
- Sailor fighting cancer says AWOL charges dropped
- Afghan legislators block law protecting women
- RCMP has 'no interest' in discussing harassment suit settlement


