Brian Schmidt: The Mathematics Does Not Lie: Why Polling Got The Australian Election Wrong

This is a guest post by Brian Schmidt. Actually it isn’t, I’ve cut and pasted. I hope he doesn’t mind. Important stuff. HT: John Walker

Everyone in my office grew sick last week of my continual complaints about the state of the political polls. Not because of any insights into the results they were predicting, but because they were all saying the same thing with a collective similarity that violates the fundamentals of mathematics.

Since the election was called, there were 16 polls that published two-party preferred results ahead of Saturday’s vote. Every single one of them predicted the LNP winning 48% or 49% of the two-party preferred vote, with Labor winning 51% or 52%.

These polls were central to the public’s perception of this election, with everyone, including the media, ignoring the polls’ underlying uncertainties. These uncertainties typically far out-weighed most of the conclusions drawn from the poll results.

In 2019 it’s hard to get a poll right. No longer is there an easy way to phone a random sample of people at home using the White Pages. Those people who are contacted are less likely to agree to be surveyed than in decades past. This means that getting a random sample that really represents Australia is harder than ever.

But the one thing that is almost impossible to avoid is what is called sampling error. This uncertainty in a poll is caused by talking to a subset of people, rather than everyone. And no matter what you do, except polling more and more people (which is very expensive), you are stuck with it.

You can think of the uncertainties in the polls much like what happens when you flip a coin 10 times. You can expect to get the “right” answer of five heads quite frequently, but not every time. It turns out mathematics tells us that you’ll only get five heads 25.2% of the time.

If you do a similar calculation for the 16 polls conducted during the election, based on the number of people interviewed, the odds of those 16 polls coming in with the same, small spread of answers is greater than 100,000 to 1. In other words, the polls have been manipulated, probably unintentionally, to give the same answers as each other. The mathematics does not lie.

I say unintentionally because humans are biased towards liking to get the same answer as everyone else. We often make subtle choices, even in quantitative analyses, to get the answer we expect. Commonly called confirmation bias in science, many of the large experiments in physics and astronomy hide the answers of an analysis from researchers until they are completely done to avoid this effect.

I don’t know why the polls so badly missed the election’s actual result. But whatever led to the five polling companies to illegitimately converge on the same answer, must be a significant contributor. All five need to have a thorough and independent investigation into their methodologies, and all should agree to better reflect uncertainties in their future narratives.

The last five years have demonstrated to me the fragility of democracy when the electorate is given bad information. Polls will continue to be central to the narrative of any election. But if they begin to emerge as yet another form of unreliable information, they too will be opened up to outright manipulation, and by extrapolation, manipulation of the electorate. This is a downward spiral our democracy can ill afford.

• Professor Brian Schmidt is a Nobel Laureate in physics and the vice-chancellor of the Australian National University

This entry was posted in Democracy, Economics and public policy, Politics - national. Bookmark the permalink.

51 Responses to Brian Schmidt: The Mathematics Does Not Lie: Why Polling Got The Australian Election Wrong

1. Nicholas
It should include link to the Guardian newspaper that published the piece

https://www.theguardian.com/commentisfree/2019/may/20/mathematics-does-not-lie-why-polling-got-the-australian-election-wrong

2. I am and will always be Not Trampis says:

Nick,

He is merely saying what Mark the Ballot said some time ago.

you would know that if you read very modest blogs

3. Alan says:

That leads to the slightly worrying question of what proportion of the citrate, if any, are influenced by inaccurate polls.

• Fyodor says:

Yairs, that would be the “acid test”.

4. I am and will always be Not Trampis says:

It alsp leads to another question which will be raised in a modest blog today.

you cannot trust the qualitative polling or focus groups either. So we have no idea of the reasons behind the election results!

Of course there is another question. It did not happen in 2016 only now. why?

• Nicholas Gruen says:

I ran into Andrew Robb at a function of a start-up accelerator and asked him what his main conclusion was of his inquiry into why Malcolm did so badly

I thought that they’d have a good run to the polls and that the 52-48 lead the ALP had could easily be eaten up. But who knows if it was a 52-48 lead and who knows if it was eaten up.

But I was always sceptical that the ALP had booby trapped their chance of getting elected, on the strength of the fact that the tax changes with negative gearing and CGT didn’t seem to do them much harm in 2016. But they were facing a politician who wasn’t much chop at negative campaigning and didn’t want to do that.

5. Fyodor says:

I wonder if the incongruity between polling surveys and the election result had much to do with the “shy voter” effect. Clearly the preferences of UAP and ON were critical in a couple of the seats that shifted to the LNP (Herbert and Longman had sizeable first preference votes for UAP+ON), and it may be that many prospective UAP or ON voters were under-represented in the poll sampling.

It was a relatively close election where the preferences of minor parties really counted in a handful of swing seats. That the polls couldn’t predict this doesn’t bother me overly – sometimes we expect too much from statisticians.

6. I am and will always be Not Trampis says:

nope.

The problem per se’ was too much reliance on landlines when in times of the NBN households perhaps do not even have one. Getting replies to mobiles is also problematic.

robocalls are also inferior to humans on phones in getting reliable information as we found out in the last Presidential election.

• Fyodor says:

The medium for polling (i.e. landline vs. mobile, robo-call vs. human) can only make a difference to the poll results if it skews the sample significantly, to the extent that it generates a misleading/unrepresentative opinion poll. That is, do we have good evidence to expect LNP-leaning voters to have been selected against in the polling process because they are, for example, more likely not to answer landlines or respond to robo-callers? I don’t think there is a good basis for making that assumption.

7. I am and will always be Not Trampis says:

well the experts beg to differ.

They think the samples were representative at all.

The ‘shy voter’ effect has it the sample is representative it is merely the answers of some that is the problem.

Why would some voters be shy this election and NEVER before..

8. Fyodor says:

well the experts beg to differ.

Which experts?

Why would some voters be shy this election and NEVER before.

I don’t know, but I do know that “NEVER before” is an unproven assertion.

The difference between 2016 and 2019 could – stress could – have resulted from the fact that the UAP didn’t exist at the 2016 election and its predecessor party (PUP) received trivial votes relative to the substantial number it received in 2019. In this election the UAP received a +3.35ppt swing, the largest, followed by ON +1.7ppt. In contrast both LNP and ALP saw modest (<1ppt) declines in first preference votes. These are important facts that may have played a role in skewing the 2PP polling estimates if the UAP and ON voters were under-represented (whether by omission or by "shy voter" misleading poll response) in survey samples.

9. I am and will always be Not Trampis says:

take your pick old son, in the paper , on the radio,on TV, even Antony said it on the night, you really need to get out more often.

Err actually we do know the ‘shy’ voter has NEVER happened before because the pols were essentially ( pun intended) right. This time they were well out.

10. Fyodor says:

take your pick old son, in the paper , on the radio,on TV, even Antony said it on the night, you really need to get out more often.

Err actually we do know the ‘shy’ voter has NEVER happened before because the pols were essentially ( pun intended) right. This time they were well out.

More unfounded fallacy. Absence of evidence is not evidence of absence.

11. I am and will always be Not Trampis says:

if you are too lazy to take an interest look it up yourself but apparently to even read what Brian would say.
I will give you a hint, how about a former head of Newspoll, Lowy institute just for startes

the ‘shy’voter theory would have polls underestimating certain party’s polling.
until this election this has not occurred. The pols fully reflected what happened on election night.
Another reason to read what Brian actually said.

12. Fyodor says:

Link, or it didn’t happen. It’s not my job to prove your assertions; it’s yours.

the ‘shy’voter theory would have polls underestimating certain party’s polling. until this election this has not occurred.

That pre-election polls were broadly accurate in the past does not mean that there was no “shy voter” effect in this election.

13. I am and will always be Not Trampis says:

you the one that needs to know although clearly you won’t understand as you have not understood Brian’s article. do some work

The shy voter effect means voters do not tell pollsters about some parties because of diverse reasons. If this theory were correct there would be a discrepancy bwteen polls before the election and the election result. There has been NO discrepancy.

14. Fyodor says:

you the one that needs to know although clearly you won’t understand as you have not understood Brian’s article. do some work

What work? Brian doesn’t even mention “shy voters”.

Meanwhile, you’ve still not produced your homerwork. Chop chop.

The shy voter effect means voters do not tell pollsters about some parties because of diverse reasons. If this theory were correct there would be a discrepancy bwteen polls before the election and the election result. There has been NO discrepancy.

Even more incoherent than usual, Homerkles.

There IS a discrepancy; that is the point. I’ve presented one theory – unsubstantiated, but it’s a theory – that might explain the discrepancy. You’ve provided the usual empty bluster.

15. I am and will always be Not Trampis says:

Brain does not mention shy voters you did. Brian’s article is about representative sampling problems.

That has NOTHING to do with shy voters. The Shy voter myth has never had any evidence here at all. Shy voters occur in random samples. They simply do not tell the pollsters their true voting intentions.

Anyone who has even been to ESL classes knows that did not happen here. Only a person who had not read Brian’s article would have attempted to say so

16. None of the above comments have anything to do with stats, so i feel free to add my bit:
In my electorate as in most of the nation (on a booth by booth basis) the 2PP split seems to be the better off voted Labor and the worse off voted LNP.

Neither of us are shy ( we are professional exhibitionists )and we don’t vote the same, but both of us hang up on survey calls, life is too short.

Maybe the question could be ‘ what kind of bullshit twat’ has the time to waste on responding to surveys these day?

17. I am and will always be Not Trampis says:

Sorry John but it does. Getting a random sample is stats 101

Just for the lazy one Here is Peter Lewis on the failure of the polls.
https://www.theguardian.com/commentisfree/2019/may/21/as-pollsters-we-are-rightly-in-the-firing-line-after-the-australian-election-what-happened

18. I am and will always be Not Trampis says:

here is the best summary from the pollbludger. Kevin Bnham’s summary is highly recommended
https://www.pollbludger.net/2019/05/22/of-swings-and-misses/

I like the fact the polls were wrong. It means the stupidity of short-term poll driven politics is hopefully at least in part diminished.

Also, Brian Schmidt is wrong about the need to increase samples to get better results. In the US, I believe there are companies which get very representative samples of people, and then pay them to participate over the long term. This way you can have smaller samples but get much more accurate results and far better bias estimations.

• Nicholas Gruen says:

I was wondering the same thing – whether longitudinal polling would solve a lot of this.

20. Simon Molloy says:

Maybe it’s the ‘shy voter’ effect. I prefer to think of it as ‘virtue signalling bias’.

Probably people on the left are more likely to be forthright in expressing their preference because they are more likely to be interested in virtue signalling (hypothesis 1). Hypothesis 2: the left – Labor, in this case, presents its policies as compassionate and moral which makes many people uncomfortable with been seen to be disagreeing with them (yes, even in a private poll interview). It’s easier to say ‘I’m undecided’ and wait ’til the truly private ballot booth day – or to say – with some discomfort – ‘Labor’.

If 1 and 2 are right, I’d expect to see the proportion of people in the undecided poll category increasingly composed of Coalition voters.

Why this time and NEVER before? Because, even over just the past 3 years, the extent of virtue signalling, herd mentality, left moralising, political correctness etc has grown significantly (hypothesis 3).

Highly speculative, I know and I take the point about “the odds of those 16 polls coming in with the same, small spread of answers is greater than 100,000 to 1”. That raises an important question about the absence of statistical variability but the more interesting question is why did all those polls converge on a result that was wrong and significantly wrong? Two different things.

• I am and will always be Not Trampis says:

The reaction of the election reminds me a lot of 1993 except of course the polls were okay then.

I think Morrison has a Keating problem with tax so I think the ALP wil win because of this.
More so if they attempt to maintain a surplus ( although most economists would call it a balanced budget) as the economy slows.

21. paul frijters says:

There could have been a late swing which no poll can predict, no matter how many people are in it.
Also, since Oz has become a class society in the last 20 years, you get class-society elements that were not there previously. The result reminds me of the unexpected win by John Major in 2000 in the UK where Labour was destined to win and yet lost, essentially because large parts of the electorate wanted to feel they were winners, not losers.

The left in general has a problem with talking down to people. They don’t get the simple point that they need to protect the self-esteem and the self-image of the electorate, rather than allow constant attacks on it. It alienates so many voters, yet the internal incentives are strongly skewed to keep doing that. The priests (like John Menadue) will just keep preaching, costing the left lots of votes. Only the leadership can effectively push against that tendency.

They need a Bill Clinton who successfully analyses where they lost and who manages to charm the swing voters. If there is one lesson from this election, then it is that Labour needs a charmer to lead them, keeping the message simple but powerful. They really should cast their next leader.

• paul frijters says:

correction: that should have been 1990 when JM won, not 2000.

• Nicholas Gruen says:

But the ALP had a charmer

Bill Shorten …

22. One in a hundred thousand chance but virtually no experts noticed it can anybody explain that?
Re booth by booth: in Queanbeyan there were booths ,all within a easy walk of each other that had swings to, or against ,Labor of around 5 percent can anybody explain that?

23. I am and will always be Not Trampis says:

There was a great irony in the election.
The swings occurred from low income and low educated voters.

These were the very people who would have gained from an ALP victory!!

• Simon Molloy says:

Not if they lost their jobs.

• They are emphatically not the people who would have gained from a labor victory .

‘we know best and, you’d better get used to it’ (or don’t vote for, us)
is not how to speak like a paid up member of the downtrodden is it?

• John,

I’m not trying to get into the substance of the debate you’re having, but, without knowing or caring much about the context, I suspect that Bowen’s now famous quote was not an expression of arrogance towards those he’s courting – take it or leave it.

I read his statement as saying that people have a choice.

It’s a bit like Thatcher’s comment “there’s no such thing as society”. Her comment has to be read in terms of what she was trying to say which is something like “Society” isn’t an actor, only individuals, and organisations thinking of them as collectives in need of coming to a view are actors, and it is to that that our attention should be drawn when we call for something to happen.

But it’s too easy to misrepresent. Red meat to the base and the rest is history.

• While it was obviously not aimed at who he was ‘courting ‘ the way he said it made my blood run cold.

• Nicholas Gruen says:

What was the context.

I can imagine it as a completely neutral response to some claim like that changing the rules was ‘undemocratic’. In fact I am struggling to think of any way in which it is even an arrogant comment.

If you don’t like it, don’t vote for it. What’s wrong with that?

• Retroactive application , Re a arrangement where it was previously completely lawful is unjust a kind of violence .

• I am and will always be Not Trampis says:

really,
The pensioners who would have gotten free dental care, the greater expenditure on hospitals which equally would have benefited pensioners, more expenditure on schools that was needs based which would have benefited their grand children,

I could go on but you get the picture.

• Insulin pumps in Australia cost twice as much as they do in NZ and the UK does that mean they’re twice as good. A Canberra based mental health provider has had to increase their work force by about 30 percent so as to help their clients cope with filling in the “massive amounts of forms” created by the NDIS does that mean there’s a 30 percent improvement in mental health.

24. I am and will always be Not Trampis says:

really, and why were they afraid of that?

25. Nicholas Gruen says:

Responding to John’s latest comment:

If you take the view that changing the refundability of credits after a year’s notice (which, incidentally is asymmetrical, pensioners didn’t object to the changes when they generated a windfall for them) we can never change any of the rules of superannuation in a way that disadvantages anyone.

We can’t introduce higher taxation on the SMSFs that have over \$50 million in them because it would be retrospective.

Doesn’t sound right to me.

And I’m still unsure of why ‘if you don’t like it don’t vote for it’ is in any way objectionable. It’s a way of saying ‘we’ll have to agree to disagree and see who’s view is endorsed by the community’. I simply can’t see any other way of reading it other than as saying “I am trying to persuade you, but if I’m unsuccessful, the effort has to end somewhere and you don’t have to agree with me. We live in a democracy and we’ll go with what the majority think.

• re ‘changing the refundability of credits after a year’s notice ‘ that’s a lot of notice if you are 80 and you and-or your spouse need costly home help etc.

The ALP wanted more money, now, and ‘wealthy’ retirees looked like a small group that could be easily cutout from the mob and squeezed.

• I am and will always be Not Trampis says:

only problem with that is the older voters with low incomes changed votes to the libs. The higher income older voters actually changed to the ALP!

• Not sure that older voters with lower incomes changed their votes and surely there’s not enough of them to really matter either ? It might be that their , extended families changed their votes who knows.
What I do know is that the subhuman redneck that I have got to know do know a snob when they meet one and they also have a good nose for BS.

• I am and will always be Not Trampis says:

I always love a theory without evidence particularly when it is protecting a tax rort

• Sure but the worse off didn’t vote your way and that is a fact.

• I am and will always be Not Trampis says:

That is what I have been saying constantly.

We simply do not know why

26. Tumble says:

Gruen

The way you’re characterizing the comment doesn’t diminish the coarseness of what he conveyed.

If you don’t like it then fuck off

is closer to the mark than your sorry excuse.

27. Robert Menzies says:

Great article by Brian & thanks for posting it Nicholas, It was a sure thing that labour was going to win here in the booming Parramatta which will be Sydney’s 2nd CBD. The polls had us convinced of a labour victory, but factoring in the sampling error it makes sense to why the polls got it wrong.

Maybe it is time to up the sample size to get a more accurate outlook?

It is very interesting contrasting the difference in results for the main Sydney CBD (harbour CBD) and Sydney 2nd CBD Parramatta (Central City). I wonder if the trend will continue in the years to come…

28. This from the AFR sums my feelings

“In an interview with the journalist Craig McGregor in March 1977, Paul Keating, then a 33-year-old spokesman for minerals and energy, was asked how, since he lived on the only hill in Bankstown and had a Mercedes sports parked in his garage, he could claim to remain true to his Labor roots.
Keating’s reply struck at the core of what he believed Labor’s mission to be. “It’s no good pretending we’re working class, down at the club socking it away, out at the footy. I reckon I’m lower middle class; I’ve made the move up which a lot of Australians have. Isn’t that what we’re all after?”
It is a telling reflection on just how far Labor has drifted from its moorings that Keating’s language of aspiration was barely audible in the hydra-headed agenda it took to the recent federal election campaign. Indeed for a party that once made so much of cherishing its history, it seems to have almost wilfully turned a blind eye to its past.”
My dad, I think, would have said that the best thing about growing up in poverty in Bourke st Surry Hills in the twenties was, leaving it.

29. One things for sure as it turned out, the polls gave Labor unjustified confidence.

30. I am and will always be Not Trampis says:

Can I add one last thing.

Bad polling does not just have consequences for quantitative polling. It has consequences for focus group research.

All parties and pundits are in the dark on why the election turned out as such.

Always good to see both Mark the Ballot and Kevin Bonham give the shy voter theory the short shift here. Knowledgeable people already knew this!!