A "lot of opinion polling is useless because it doesn’t understand its limitations" writes Graham Young. One of the major limitations of polling is the tendency of respondents to answer questions about things they know nothing about. A series of studies have shown how respondents can be persuaded to answer questions about fictitious issues such as non-existent pieces of legislation.
Pollsters are reluctant to allow respondents to say "don’t know". They worry it reduces the representativeness of responses and they know it annoys journalists and lobbyists who want to tell a clear, simple story about what ‘the public’ thinks. As a result, questionnaires are usually designed to nudge respondents into answering the question.
Times columnist Daniel Finkelstein suspects that polling on climate change has a lot in common with polling on fictitious issues. He suggests that people "don’t have the foggiest clue about climate science" but answer questions about it because they feel they ought to know about it.
If Finkelstein is right, what should politicians do with the results of climate change polls?
Adventures in polling land
I was at a loose end after I graduated from university. So I moved back to Sydney and picked up some work doing market research interviewing. I’d done some political polling when I was in California, so it was a job I knew I could do. So after a quick training session I was spending most of my evenings ringing people up and asking their opinions on things like chocolate bars, freeways, rain forests and politicians.
One day the boss strode into the interviewing room and pulled our supervisor aside. We were getting too many "don’t knows" he said. Other companies were not getting so many and he wanted it fixed. Our supervisor passed on the order and we all got back to work.
The great myth of opinion polling is that ‘survey instruments’ measure something that already exists — attitudes respondents carry around with them in their heads. So the assumption is that when a respondent resists it’s usually because they are not trying hard enough to retrieve the attitude from memory. So you give them a good prod, they put in the effort, and you get your answer.
With a bit of encouragement it’s possible to get many reluctant respondents to choose an answer from a list. You can repeat the question or suggest that there are no right or wrong answers, only opinions. You can ask which way they respondent "leans" on an issue. But in many cases, it’s obvious that respondents have no idea what the question is about.
Something from nothing
Polling experts George Bishop and David Moore argue that: "Pollsters typically produce the illusion of an opinionated public by a variety of techniques that, wittingly or unwittingly, manipulate respondents into coming up with opinions, even when they don’t have them."
In the mid-1980s Bishop and his colleagues at the University of Cincinnati demonstrated this by asking respondents about fictitious issues such as the Public Affairs Act, The Agricultural Trade Act and the Monetary Control Bill (pdf). When interviewers applied pressure in order to reduce "don’t know" responses, it was possible to get over 50% of respondents to give an opinion about a fictitious issue.
One way to boost responses to questions about fictitious issues is to anchor them in partisan politics. Following on from George Bishop’s work, the Washington Post’s Richard Morin constructed a survey asking about the fictitious Public Affairs Act. Half the sample were asked:
President Clinton said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree?
The other half were asked:
The Republicans in Congress said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree?
More than half of those interviewed gave an opinion. Democrats were more likely to support repeal of the fictitious act when asked the first question and Republicans were more likely to support repeal when asked the second question.
More recently researchers Patrick Sturgis and Patten Smith asked British respondents about the fictitious Agricultural Trade Bill and Monetary Control Bill. They found that ‘opinion’ on these issues differed according to which political party respondents supported suggesting that "respondents were anchoring the fictional bills to genuine issues of public controversy."
Sturgis and Smith also found that respondents who reported being more interested in politics were more likely to answer questions about the fictitious bills. And asking respondents about their level of political interest early in the questionnaire increased the number of responses. They reported:
These findings lend further support to the suspicion that self-reported political interest is not a ‘pure’ measure of an individual’s motivation to acquire and retain information about politics. Instead, it appears to incorporate a significant element of socially desirable responding for those individuals and groups which hold civic attitudes and behaviour in high esteem.
These kinds of findings raise doubts about what opinion polls really measure.
Non-beliefs about climate change
Late last year in the UK Ben Webster and Peter Riddell reported that: "Less than half the population believes that human activity is to blame for global warming, according to an exclusive poll for The Times." The poll results were widely reported in the media.
The Times/Populus poll had reported that only 41% of respondents accepted that: "Climate change is happening and is now established as largely man-made". But citing Sturgis and Smith’s study, Times columnist Daniel Finkelstein argues that responses to questions about climate change produce a high level of pseudo-answers:
Climate-change science may be a subject on which most people actually know little but feel they ought to know a great deal. It is thus the perfect subject for people to give a pseudo-answer. And that answer will be in line with their broader political opinions and their confidence in the government.
It’s a reasonable suspicion. The poll’s climate change question seems designed to minimise "don’t know" responses:
Still thinking about climate change and global warming, based on however much or little you know about this, please say which of the following statements comes closest to your own view, even if none of them exactly describes what you think …
Reporting the results of a recent Australian poll, the Institute of Public Affairs announced that: "Two-thirds of Australians now doubt the scientific consensus on global warming". The Daily Telegraph translated that into a dot point: "Two-thirds don’t believe it is real".
According to the IPA’s executive director John Roskam,"These figures reveal that Australians are no longer confident they’re hearing all the facts about climate change"
Roskam’s interpretation suggests that respondents are actively seeking out and weighing up the evidence about climate change. And after doing so, many have concluded that the evidence for man-made climate change is weak. But what’s more likely is that a large number of Australians have paid little attention to the climate change debate and don’t have a meaningful opinion about the evidence.
The poll surveyed 1000 respondents across Australia from 30 April – 2 May 2010 about their attitude to global warming:
- 35% of Australians believe that “The world is warming and man’s emissions are to blame.”
- 26% of Australians believe that “The variation in global temperature is just part of the natural cycle of nature.”
- The largest group, 38% of Australians agreed with the statement that “There is conflicting evidence and I’m not sure what the truth is.”
The question was "Thinking now about global warming. Which one of of these statements best matches your view?" While there was a "don’t know" option available, interviewers were instructed not to read it out.
What should governments do?
Opponents of government action against climate change often cite polling evidence like the IPA’s. Sometimes this is just partisan gloating — right wing partisans tend to see climate change action as one of their opponents’ key issues and are happy to see them saddled with something unpopular. But sometimes it seems as if they are suggesting that governments should delay taking action until they’ve persuaded the electorate that the problem is real and the action necessary.
One test of whether this is a genuine argument is whether its proponents would take the same position on issues like tariff reduction and privatisation. It’s unlikely that a majority of the Australian public were ever convinced on these issues. And most would have known little about the economic arguments behind the reforms. Should the Hawke and Keating governments have waited until the electorate was convinced about the need for reform?
Sometimes governments take action when they believe that most of the public will support the results of action even though they oppose the action itself. As Edmund Burke told the electors of Bristol: "Your Representative owes you, not his industry only, but his judgment; and he betrays, instead of serving you, if he sacrifices it to your opinion."