The hunt for a no-telephone poll
By Carolyn Ryan
"A massive Decima Research online survey of nearly 9,000 people provided to the Canadian Press put the Liberals at 34 per cent support, compared with 26 per cent for the Conservatives. The NDP was at 20 per cent and the Bloc Quebecois was at 14 per cent. The Decima poll, of 8,842 decided or leaning voters, is considered accurate to within 1.5 percentage points 19 times out of 20. It was conducted Nov. 25 through Monday."
– News item from the general election file, circa early December.
Decima's online poll is designed to track how campaign developments such as the Conservatives' pledge to cut the GST could affect people's voting decisions. (CP file photo)
A way of conducting accurate, respected online polls has been described as the "Holy Grail" of the market research industry. It would be dramatically cheaper and quicker than conducting phone surveys. It would do away with the problem of declining response rates to phone polls, and it would be less intrusive because it would rely on a pool of people who have already indicated they are willing to answer survey questions.
Canadian market research company Decima has been refining what it calls its eVox panel for the past three years. About 50,000 people have signed up to answer occasional online questionnaires about everything from headache pills to Christmas shopping intentions. In return, they get the chance to win monthly prizes of $1,000. This is no pop-up poll, which allows companies or political parties to skew the results by repeatedly visiting a website. Potential respondents must fill out a detailed demographic survey when they sign up, and they don't know what types of surveys they will be asked to fill out. The company then chooses names from its database at random to represent the full population's mix of age groups, genders, income levels, and geographic distribution.
The process has been so successful that 25 per cent of Decima's survey business is now conducted by e-mail instead of phones. And in the future, the pool of respondents will be getting bigger. The company has just struck a deal with Aeroplan to allow Decima to send an e-mail to about 5 million Aeroplan subscribers, representing a cross-section of Canadian society, offering them a chance to earn Aeroplan points by participating in market surveys. The company hopes to build its panel to 200,000 or 300,000 that way.
Leading up to the start of this campaign, Decima decided to partner with Carleton University's School of Journalism and Communications to showcase its online panel's capabilities when it comes to tracking political issues. CEO Bruce Anderson points out that the sheer size of the sample gives a margin of error that is much lower than traditional telephone surveys can deliver. Or that's the premise, anyway.
Here's an excerpt from an interview CBC.ca conducted with Anderson on Dec. 9:
What are the benefits of using online polling methods instead of telephone polling?
It allows you to do research in a fashion that's more like using a laser than radar … [It's] laser-like research to find the needle in the haystack, rather than having to tear apart the haystack to find the needle.
If people say they intend to purchase something in particular for Christmas, three weeks later we can ask them, "Did you actually buy this?" Also, you can show people things online. And you don't have to ask them their demographic information every time.
The Holy Grail of market research buyers is being able to tap into specific markets, to be able to research those audiences without doing random-digit dialling methods that can be enormously expensive. Response rates for telephone polling are now hovering at 15 to 17 per cent, which means that upwards of 80 per cent of the people we try to contact, we don't actually complete a telephone interview with.
[For the political poll,] we sent out our first tracking questionnaire last night at 6 p.m. [on Thursday, Dec. 8]. By this morning, we had 3,000 completes. We're going to close that off tomorrow at 5 or 6 p.m. in the afternoon. And then we're going to be able to report results on Sunday.
If we tried to do that through our phone bank, we would have had 300 or so people paid an hourly rate to work our phones in a heated, rented office building. We didn't have any that. That's one of the reasons why it is possible economically to provide people with some sort of incentive.
How does your online political poll panel work?
Starting about three years ago, we started gathering e-mail addresses from ads placed on sympatico.ca and other major websites. As well, we always ask people in our telephone surveys whether they'd be interested in joining our online panel, to answer questions on a variety of topics.
Of a sample of 12,000 people chosen at random to take part in the current campaign poll, 75 per cent said they would agree to do a weekly survey.
We make sure our samples are balanced on gender, age, geographic, and are as reflective of the population that they're meant to represent as possible. But we also wanted to over-sample in major urban areas. We wanted to be able to say, "What do we think is happening in Vancouver?" and "What do we think is happening in Ottawa?"
One advantage of the way we're doing this is that we'll be able to see what's happening to their opinion and why. That's because we'll follow the same people all the way through the campaign, to see what specifically is affecting the voters' positions … If some people were telling us they were voting Liberals, what do they think now and why? It's trying to focus more on the explanation for movement.
We happen to think it's pretty accurate.
Doesn't the offer of cash incentives sway the results?
I don't think our incentive will allow people to become professional survey respondents … Incentives have been used for a long time, especially when it comes to recruiting people to take part in focus groups. I don't think there's a philosophical problem there.
Every method has a self-selection bias, though we can pretend to ourselves and everybody else that there isn't. [In telephone surveys,] lots of people self-select to take the call or not take the call, by using their call display feature.
How do keep them interested in staying on your panel?
You have to respect your respondents a great deal. You have to provide them with a reasonable number of surveys – enough to keep them connected, one or two a month, but not so much that you're really oppressing them with requests for information. Each of the surveys will take no more than 15 minutes to complete. And if someone doesn't answer three or four requests to take part in surveys, we send them a n e-mail asking if they want to be off the panel.
How can you keep a party from packing supporters into the study group?
We're not adding people to our campaign panel now. You had to be sent an invitation to participate in it before the campaign began. Given the normal course of what we do [consumer market research], there's very little incentive for parties to target us.
Wouldn't you be an attractive target for hackers?
We have a huge infrastructure to manage the security and the privacy of our panelists.
What about the argument that online polling completely shuts out the 28 per cent of the adult Canadian population that doesn't have access to a computer or doesn't regularly access the internet?
It's important to compare it to the alternatives and not to perfection. You could make the case that no part of the population is excluded from a phone survey that has a telephone, but it's clearly not a perfect sampling system, if it ever was. It's lost some of its strength over time because of declining response rates.
We did take the opportunity to talk with other people who have expertise in online polling, specifically the YouGov polling operation in the United Kingdom.
They tackled this issue head on early. If it's true that some audiences are disproportionately unrepresented in online survey panels – older people, rural residents, those with lower incomes, in some cases female, similar patterns that we see in Canada … they made sure that they had lots of people who fit those demographics in their panel. Then they ran some tests. With the exception of a situation where you're doing market research with regard to technology, you get very, very few differences between online and telephone sample groups.
What's the science on whether people are more or less likely to lie online … for example, by saying they're old enough to vote or eligible to vote when they really aren't?
It's a mug's game to get into this horse race about is online polling more accurate than phone polling... Phone surveyors don't audit age on phone information. There's a certain element of good faith that applies in either of those methodologies.
We did run a couple of parallel studies, conducted online and on the phone. We found negligible differences in the response, and the differences that we did see … are raising the question about whether or not you get a more candid response online.
Even if I intellectually know that a telephone surveyor has no stake in what answer I give, there's a human interaction there. There isn't any instinct to please someone if you're doing an online survey.
The human factors involved in a telephone interview condition people to offer less extreme responses and use more middle-ground answer scales rather than the full range of answer scales.
Another thing, if I'm going to read you a battery of items, you don't know how long the list is going to be, so you can tend to just pick an early answer so that you'll be able to remember one by the time the question comes … Online, that's not a problem. You can show people a screen with 10 different questions and the same answer categories to all of them.
If you ask me what's a more honest or accurate depiction, the more nuances there are in the ways in which they can respond, the more accurate they can be.
Will online polling eventually replace telephone polling?
Our phone research business continues to grow. It accounted for 75 per cent of the one million surveys we conducted last year. [But online polling] is a very, very important thing for us. We're deeply committed to it and investing heavily in it, and we're going to get it right.
We believe in permission-based, incentive-fuelled polling, as opposed to calling people at home at times convenient to us but not necessarily convenient to them … for which we don't compensate them. That model's not going to work quite as well in the future.