Poll scepticism & climate change policy

A "lot of opinion polling is useless because it doesn’t understand its limitations" writes Graham Young. One of the major limitations of polling is the tendency of respondents to answer questions about things they know nothing about. A series of studies have shown how respondents can be persuaded to answer questions about fictitious issues such as non-existent pieces of legislation.

Pollsters are reluctant to allow respondents to say "don’t know". They worry it reduces the representativeness of responses and they know it annoys journalists and lobbyists who want to tell a clear, simple story about what ‘the public’ thinks. As a result, questionnaires are usually designed to nudge respondents into answering the question.

Times columnist Daniel Finkelstein suspects that polling on climate change has a lot in common with polling on fictitious issues. He suggests that people "don’t have the foggiest clue about climate science" but answer questions about it because they feel they ought to know about it.

If Finkelstein is right, what should politicians do with the results of climate change polls?

Adventures in polling land

I was at a loose end after I graduated from university. So I moved back to Sydney and picked up some work doing market research interviewing. I’d done some political polling when I was in California, so it was a job I knew I could do. So after a quick training session I was spending most of my evenings ringing people up and asking their opinions on things like chocolate bars, freeways, rain forests and politicians.

One day the boss strode into the interviewing room and pulled our supervisor aside. We were getting too many "don’t knows" he said. Other companies were not getting so many and he wanted it fixed. Our supervisor passed on the order and we all got back to work.

The great myth of opinion polling is that ‘survey instruments’ measure something that already exists — attitudes respondents carry around with them in their heads. So the assumption is that when a respondent resists it’s usually because they are not trying hard enough to retrieve the attitude from memory. So you give them a good prod, they put in the effort, and you get your answer.

With a bit of encouragement it’s possible to get many reluctant respondents to choose an answer from a list. You can repeat the question or suggest that there are no right or wrong answers, only opinions. You can ask which way they respondent "leans" on an issue. But in many cases, it’s obvious that respondents have no idea what the question is about.

Something from nothing

Polling experts George Bishop and David Moore argue that: "Pollsters typically produce the illusion of an opinionated public by a variety of techniques that, wittingly or unwittingly, manipulate respondents into coming up with opinions, even when they don’t have them."

In the mid-1980s Bishop and his colleagues at the University of Cincinnati demonstrated this by asking respondents about fictitious issues such as the Public Affairs Act, The Agricultural Trade Act and the Monetary Control Bill (pdf). When interviewers applied pressure in order to reduce "don’t know" responses, it was possible to get over 50% of respondents to give an opinion about a fictitious issue.

One way to boost responses to questions about fictitious issues is to anchor them in partisan politics. Following on from George Bishop’s work, the Washington Post’s Richard Morin constructed a survey asking about the fictitious Public Affairs Act. Half the sample were asked:

President Clinton said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree?

The other half were asked:

The Republicans in Congress said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree?

More than half of those interviewed gave an opinion. Democrats were more likely to support repeal of the fictitious act when asked the first question and Republicans were more likely to support repeal when asked the second question.

More recently researchers Patrick Sturgis and Patten Smith asked British respondents about the fictitious Agricultural Trade Bill and Monetary Control Bill. They found that ‘opinion’ on these issues differed according to which political party respondents supported suggesting that "respondents were anchoring the fictional bills to genuine issues of public controversy."

Sturgis and Smith also found that respondents who reported being more interested in politics were more likely to answer questions about the fictitious bills. And asking respondents about their level of political interest early in the questionnaire increased the number of responses. They reported:

These findings lend further support to the suspicion that self-reported political interest is not a ‘pure’ measure of an individual’s motivation to acquire and retain information about politics. Instead, it appears to incorporate a significant element of socially desirable responding for those individuals and groups which hold civic attitudes and behaviour in high esteem.

These kinds of findings raise doubts about what opinion polls really measure.

Non-beliefs about climate change

Late last year in the UK Ben Webster and Peter Riddell reported that: "Less than half the population believes that human activity is to blame for global warming, according to an exclusive poll for The Times." The poll results were widely reported in the media.

The Times/Populus poll had reported that only 41% of respondents accepted that: "Climate change is happening and is now established as largely man-made". But citing Sturgis and Smith’s study, Times columnist Daniel Finkelstein argues that responses to questions about climate change produce a high level of pseudo-answers:

Climate-change science may be a subject on which most people actually know little but feel they ought to know a great deal. It is thus the perfect subject for people to give a pseudo-answer. And that answer will be in line with their broader political opinions and their confidence in the government.

It’s a reasonable suspicion. The poll’s climate change question seems designed to minimise "don’t know" responses:

Still thinking about climate change and global warming, based on however much or little you know about this, please say which of the following statements comes closest to your own view, even if none of them exactly describes what you think …

Reporting the results of a recent Australian poll, the Institute of Public Affairs announced that: "Two-thirds of Australians now doubt the scientific consensus on global warming". The Daily Telegraph translated that into a dot point: "Two-thirds don’t believe it is real".

According to the IPA’s executive director John Roskam,"These figures reveal that Australians are no longer confident they’re hearing all the facts about climate change"

Roskam’s interpretation suggests that respondents are actively seeking out and weighing up the evidence about climate change. And after doing so, many have concluded that the evidence for man-made climate change is weak. But what’s more likely is that a large number of Australians have paid little attention to the climate change debate and don’t have a meaningful opinion about the evidence.

The IPA’s media release provided the details:

The poll surveyed 1000 respondents across Australia from 30 April – 2 May 2010 about their attitude to global warming:

  • 35% of Australians believe that “The world is warming and man’s emissions are to blame.”
  • 26% of Australians believe that “The variation in global temperature is just part of the natural cycle of nature.”
  • The largest group, 38% of Australians agreed with the statement that “There is conflicting evidence and I’m not sure what the truth is.”

The question was "Thinking now about global warming. Which one of of these statements best matches your view?" While there was a "don’t know" option available, interviewers were instructed not to read it out.

What should governments do?

Opponents of government action against climate change often cite polling evidence like the IPA’s. Sometimes this is just partisan gloating — right wing partisans tend to see climate change action as one of their opponents’ key issues and are happy to see them saddled with something unpopular. But sometimes it seems as if they are suggesting that governments should delay taking action until they’ve persuaded the electorate that the problem is real and the action necessary.

One test of whether this is a genuine argument is whether its proponents would take the same position on issues like tariff reduction and privatisation. It’s unlikely that a majority of the Australian public were ever convinced on these issues. And most would have known little about the economic arguments behind the reforms. Should the Hawke and Keating governments have waited until the electorate was convinced about the need for reform?

Sometimes governments take action when they believe that most of the public will support the results of action even though they oppose the action itself. As Edmund Burke told the electors of Bristol: "Your Representative owes you, not his industry only, but his judgment; and he betrays, instead of serving you, if he sacrifices it to your opinion."

This entry was posted in Uncategorized. Bookmark the permalink.

10 Responses to Poll scepticism & climate change policy

  1. Chris Phillips says:

    I have always found surveys difficult to respond to, particularly the multiple choice answers. So often there are no ‘black and white’ answers as it depends on circumstances and I nearly always want to rationalise my feelings. Without giving chapter and verse in my replies which would take far too long, I can understand that surveys cannot always be accurate.

    As far as things like Climate Change, I defer to the experts. If they seem to know what they are talking about and are constantly reviewed by their thousands of peers, it seems logical to believe them and all their statistics. I get the impression that the deniers in this case are the people without the expertise and qualifications. So it is obvious to me on which side I come down on.

    It’s a bit like religion, people have faith in things without evidence, so they believe what they want to believe, or because they really can’t think for themselves in a rational way.

    Having said that, I am open to any argument that suggests that there could also be natural causes s well.

    I also believe that Surveys can manipulate the desired answer by the way a question is couched and why any survey on a complicated subject can never be answered in two syllable words or less.

  2. hc says:

    The difficult issue is that people with interest group roles can use polls to advance their cause. Hence on occasion it is convenient to cite outputs without the obvious need for qualification.

    Of course a “don’t know/don’t care” category should be provided – we should not assume an opinionated public. But people may also create “warm glow biases” – we want to appear to be concerned about climate change even if they are not or even if they might not vote for a party which seeks strong action on climate change. You have mentioned above the problem of framing biases with respect to the Public Affairs Act. I also think people need to be provided with background information on the issues they are being asked about.

    Economists will tell you that eliciting preferences isn’t easy and is difficult to do on the cheap. Its an complicated issue of sampling design.

  3. John Barr says:

    One of the major limitations of polling is the tendency of respondents to answer questions about things they know nothing about.
    The other is polling people Green or anti green, who only research their own bias to support their own view.

    Times columnist Daniel Finkelstein suggests that people “don’t have the foggiest clue about climate science” but answer questions about it because they feel they ought to know about it.
    And some people, both sides of the coin, have an opinion based on,”whatever you are for, I’m against.” Much like Politicians & supporters of the major political partys.

    I’m one of the 26% of Australians believe that “The variation in global temperature is just part of the natural cycle of nature.” Having done my research into both sides of the arguement.

    I do believe that their should be a “don’t realy know” & a “don’t really care” box to tick on all Polls. The research is right when it says that Polls can be biased towards a particular result. When I do these polls & there is a comments box at the end, I usually add a comment like. In Q9 I ticked #3 because the poll refused to go on unless I ticked a box. My answer should have been. I don’t know.

    I find some sites such as “Get Up” tend to espouse their opinion as the “Majority of People” when clearlly it’s not. It is the Majority of “Get Up” Members. If you are a member of such a group, or any similar group, your opinion, if it is at odds with the opinion the Group has decided it wants to espouse, is ignored and frownd upon. You are deemed as not to have the right attitude or are being “impolite” I believe that’s another word for not being Politically or Lattecally Correct. Which of course, I’m not. It wasn’t in the corriculum when I went to school.

  4. John Barr says:

    •26% of Australians believe that “The variation in global temperature is just part of the natural cycle of nature.”

    I am in this camp. When polled I am usually asked to support one side or the other. If I go with the 26% It seems that I am against all the things that would reduce carbon emissions. If I go with the 35% then it seems that I accept all the Greenie arguements. I am usually forced into one camp or the other.

    The reality is that I support the Natural Earth Cycle argument, but I would like to see less reliance on fossil fuels & more research into & use of alternate energy sources. None of the Polls on Climate Change, or any other Poll I have taken refelect this view.

  5. Graham Young says:

    Don, you had a very interesting post until you got to the IPA stuff. What interests me most about their poll is that is at odds with all the other Australian polls of which I am aware. Either there has been a sudden collapse in support for action on global warming, or there is something wrong with their questions. Perhaps it is an odd sample. I haven’t quite worked out what it is.

    From my research it is certainly true that there has been a decline in support in Australia for the proposition that manmade climate change will be catastrophic. And when you ask questions designed to find how high a priority it is, you find Australians aren’t prepared to pay or do much to avoid it. But the IPA poll has a much larger collapse than I would have thought possible.

    I think you also misanalyse what the IPA is about. One of the most pervasive arguments in favour of the IPCC version of global warming is that the vast majority of scientists agree – essentially an opinion poll argument. Naomi Oreskes compounded this with her study which purported to show that no-one who published peer-reviewed papers disputed the IPCC version. Then you had Laputian academics such as Quiggin saying that you shouldn’t report views opposed to the IPCC because they weren’t mainstream and that to report them as though they had equal validity was bias.

    Organisations like the Climate Institute are keen to conduct surveys in this area and then use them as arguments for why politicians should act on their views.

    Roskam is just getting in on the act. His position, as far as I can work it out, is that science is not done by consensus (which is a variation of the polling argument). But he’s been on the wrong end of a populist argument for so long he can’t resist turning the hose on his tormentors when he gets the opportunity.

    It’s all politics, and of course politicians want to keep on side with the majority, so if you want to convince them to do something, that is their standard of proof. Which is the reverse of what Mark’s original post, that most of this stemmed from, purported to show, which was that politicians and elites shape public opinion rather than the other way around.

  6. As I noted in the other thread on public opinion, known issues with polls such as those Don discusses means that you can’t take one poll as clearly representing meaningful public opinion. But when you have a lot of different polls producing the same pattern of results I think we can be reasonably confident in our conclusions.

    Except for the last couple of months, I’ve written about most of the published Australian polling on climate change for the last couple of years, and we can clearly see trends towards the sceptic camp. This appears to be largely Liberal supporters shifting. On complex issues people cannot judge from everyday experience they defer to others – the politicians they support, experts, etc.

    I haven’t done a detailed analysis of the IPA’s poll, but its findings are similar to other polls. Its 35% support for a purely human cause of climate change is very similar to the 32% in a Newspoll in 2008. It’s 35% willing to pay nothing for climate change policy is up on 21% in a 2008 Lowy Pol, but the broad pattern is very similar with only quarter to a third prepared to pay the kinds of sums proposed in in the ETS.

  7. James Farrell says:

    An intereing post, Don. Two observations:

    First, it isn’t the levels of opinion on any topic that are interesting, for all the reasons you give, but the changes. Who the hell knows what players’ comfort level’ is, that Tony Greig earnestly reports from pitch-side? But it means something that the index has risen or dropped from the previous day.

    Second, I’m inclined to see these changes, say from 55% to 35%, not as inherently meaningful descriptions of public opinion, but as data calling for some explanation. If the people who changed their minds are at the bottom of the information food chain, what event caused them to change their minds? It might be something as simple as one Alan Jones tirade.

  8. Corin says:

    I accept all these points but politicians need clarity in presentation. This is why climate change backflips have hurt Rudd. You can’t go from being a chief champion to an agnostic overnight without some long conversation over it. I mean it is possible for politicians to change their minds but it often doesn’t help them in the PR/polling outcome: see amendments to WorkChoices – they had almost no effect as people just thought it was a con or worse that the Govt got it wrong the first time in not having one.

    I was working for the Labor party when WorkChoices was amended and it probably helped Howard a bit (poll wise), but I tell you what, it sure made business groups shut up. They had little to campaign for once Howard put the fairness test back on.

    It is the biggest conundrum in politics that you can’t know what the difference would have been if different decisions had been made – which leads to everyone being an expert. My guess is that Howard would have lost by more, but then business would have campaigned harder for him than they did. That business campaign went out with a bit of a whimper and may be a noisy campaign could have helped Howard more with his whole ‘Union bosses’ thing.

    I guess, given my job, we moved Labor more into the centre-ground on workplace issues (i.e. it was more balanced than previous ALP policies on work) and this stranded Howard over the abyss.

    He was over the abyss whether the fairness test was included or not and may be the noisy bloody viscious campaign could have been more effective for him, who knows!

  9. Pingback: Andrew Norton » Blog Archive » Is the IPA’s climate change poll plausible?

  10. Pingback: Club Troppo » Lies, damned lies and opinion polls — the Daily Kos controversy

Leave a Reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.