Expert political judgment and the dream team


Mr Rudd by Colin Wicking

The media is inevitably full of predictions about the Rudd/Gillard Labour leadership. What follows is the case for flipping straight to the sports pages.

Because none of the punditocracy have much of a record of accurate judgment in the week after the election of a new leader. If the punditocracy were to be judged on their assessments of new leaders, most would have a closer acquaintance with Centrelink.

Michael Costello, a bruised Beazley loyalist, angrily makes the point in The Australian (New faces not enough).

John Hewson, Alexander Downer, Mark Latham, Rudd. The media reaction in each of these cases has been the same: “a fresh face”, “the dream team”, “a new generation”, positive coverage, polls shoot up. In the case of Hewson and Latham, they not only lost the only election they stood for but their parties went backwards. Downer never made it to an election, despite rave reviews for many months.

Gerard Henderson has been making the same point for years in his Fairfax columns (Predictions that missed their Mark) and at the Sydney Institute’s Media Watch.

But it was Berkeley psychologist Professor Philip Tetlock who really nails the global punditocracy’s record in 2005’s Expert Political Judgment: How Good Is It? How Can We Know? (Amazon link), a book based on 20 years’ assembling of evidence. Louis Menand nicely summarised Tetlock’s book in a New Yorker piece Everybody’s An Expert:

… People who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake …

… No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones.

Or as Tetlock himself puts it:

… The evidence falls into two categories. First, as the skeptics warned, when hordes of pundits are jostling for the limelight, many are tempted to claim that they know more than they do. Boom and doom pundits are the most reliable over-claimers …

… Second, again as the skeptics warned, over-claimers rarely pay penalties for being wrong. Indeed, the media shower lavish attention on over-claimers while neglecting their humbler colleagues.

Tetlock got his 15 minutes, and then nothing changed. Conventional journalistic wisdom stayed concreted in place: bold predictions and fake mind-reading efforts make good copy, or at least the copy that editors want. So, to take an instance at random, Shaun Carney and Michael Gordon in The Age (Back in the game) report Rudd and Gillard are seeking:

“The equivalent of a rebirth of the Labor Party … a comprehensive repositioning of the ALP at the centre of the political debate … the consensus among shrewd thinkers on either side of Parliament is that there is now a serious contest to be had.”

This sort of media content could benefit from a couple of innovations.

The first innovation is a better indication from the journalist about the strength they assign to their own predictions, so that we get writing like this imaginary passage:

“Rudd believes the ALP needs to move closer to the centre of the political landscape (certainty: 95%) … He should perform much better than Beazley at the 2007 poll (certainty: 65% that he will pick up seats).”

The second innovation would be a database of predictions (weighted for certainty, ideally) against which the punditocracy could be judged – a Gruen-style effort to improve the quality of consumer information. Tetlock himself notes that “no society has yet created a widely trusted method for keeping score on the punditocracy.” But it would be instructive to assess the predictions of a Malcolm Farr, a Sean Carney, a Miranda Devine against their previous performance. Suggests Tetlock: “Pundits might adapt to accountability by showing more humility, and political debate might begin to sound less shrill”.

We seem hard-wired to enjoy firm narrative, so don’t expect a media revolution tomorrow; these sort of innovations will take time if they happen at all (certainty: 99.999% that the pundits will not suddenly change their writing styles en masse).

Also possible, though, is that the punditocracy will draw less attention over time, its judgments less important in the face of some innovation that better captures the wisdom of crowds – perhaps a fine-grained system of prediction markets (certainty: no-real-idea-at-all-%).

About David Walker

David Walker runs editorial consultancy Shorewalker DMS (, editing and advising business and government on reports and other editorial content. David has previously edited Acuity magazine and the award-winning INTHEBLACK business magazine, been chief operating officer of online publisher WorkDay Media, held policy and communications roles at the Committee for Economic Development of Australia and the Business Council of Australia and run the website for online finance start-up eChoice. He has qualifications in law and corporate finance. He has written on economics, business and public policy from Melbourne, Adelaide and the Canberra Press Gallery.
This entry was posted in Politics - national, Print media. Bookmark the permalink.
Notify of
Newest Most Voted
Inline Feedbacks
View all comments
14 years ago

Good points, D W Griffiths. I think most people would agree with what you’ve said.

And there we have it. Fertile ground for the imaginitive. In just those simple words, which sound ok, don’t they, what has to be objectified? Which ‘points’ are good, (90% of them, if so, which ones?, and of those, to which degree), and of those points, against what or which other points are they to be compared? And who and how are those comparative points rated? Of the ‘most people’, which most people? Most people who read it? Most people in Australia, the world? In Mozambigue? Agree to which percentage? Compared with what? And what and where is it that ‘you said’ it, having made quotes. Are they agreeing with what you’re saying about the quotes, or about the quotes themselves, or about your assessments of them? And so on.

What happens, then, when there’s a whole op-ed piece to go through, or even just a mad commenter suffering the odd (very odd, say, 93%, against the normal type of odd) grammatical error?

It’s a great idea (12% – 86%, depending, see above), but you’d want to have to love numbers (0% – 100%, depending).

Nicholas Gruen
Nicholas Gruen(@nicholas-gruen)
14 years ago


Instead of “I think most people would agree with what you’ve said” I think (I’m 65% certain) that you should have said “the consensus among shrewd and hard-headed thinkers on either side of Parliament is as you have said”.

14 years ago

Nicholas, I think (22%) the consensus among shrewd and hard-headed thinkers on either side of Parliament is as you have said.

As a general matter of course, otherwise, it is amazing how poorly we human beings communicate, even in direct conversation.

Take the innocuous two sentences in the first comment above – two sentences typed in daily on blogs all over the place – and straight into the opening gambit confusion and uncertainty are the order. What comes after that compounds it all into a verifiable state of madness. (Ah.. no? What do you mean, how verifiable?) We really do talk amongst each other in our own heads, for much of our lives.

And if you believe that, and anything else written here or elsewhere, along with this, you’re crazy. Me, I’m having a party.

Nicholas Gruen
Nicholas Gruen(@nicholas-gruen)
14 years ago

I have just come across Peter Brent’s column on the changeover. At least based on his own factual claims which I have no reason to doubt, the MSM should be beating a path to his door as the person who got it right on Latham all along – a beacon of calm amid a storm of febrile media hysteria.

A few days before the 2003 caucus vote that installed Latham as federal Labor leader, I posted, on my little website, estimates of the chances each prospective Labor leader had of taking their party to victory the following year. Unlike most, I believed Simon Crean, if he kept the job, stood a fair chance of defeating John Howard – one in three. Beazley, I reckoned, would have two chances in three of winning an election and Rudd almost as much three in five. Latham I gave one chance in 11 because his every utterance showed he had no idea about what makes people change their vote. He wanted to out-Howard Howard.

Throughout the next 10 months I stuck to my prediction that, come polling day, Latham would take his party backwards, which was what happened. Pardon my trumpet-blowing, but comparisons can be illuminating and we are now in the identical position in the three-year cycle.

Beazley was the Labor Party’s best chance for 2007, with Rudd the next most likely. You wouldn’t know it from the almost criminally incompetent journalistic coverage, but opinion polls over the last six months pointed to a federal Labor victory.

14 years ago

hey I like kevins new hat…I want one just like it

Joshua Gans
Joshua Gans(@joshua-gans)
14 years ago

We can’t test counterfactual predictions – Crean would have won in 2004, Beazley in 2007, etc. – so they don’t count for much. Brent’s prediction on Latham earns him some points, especially when you consider that someone as experienced as Alan Ramsay was likening Iron Mark to a Mack truck bearing down on a stupified Howard (though he changed his mind pretty soon as Latham’s weirdness quickly manifested itsef). But I would want to see a few more good ones before pronouncing Brent the New Oracle. A useful website, though.

Nicholas Gruen
Nicholas Gruen(@nicholas-gruen)
14 years ago

Ramsey’s hopelessly mislead by his own foaming at the mouth.

Nicholas Gruen
Nicholas Gruen(@nicholas-gruen)
14 years ago

DW, I’ve just read the Louis Menand piece you link to and it’s bloody marvellous. Up there with Wisdom of Crowds for stuff to think about. And may I pose this thought to you. I know you are a Popper fan but the material re-inforces my own view that the important thing to talk about in methodology of science is not the line of demarcation between science and non science. (It would be useful to develop and enforce it if one could draw that line, but one can’t. The project was a failure, and so on Popperian grounds the Popperians should move on – as I gather from Rafe’s comments he has).

But what methodological discussion and awareness might be able to do is to give us more insight into our own and our own scientific cultures’ foibles as we attempt to ‘do science’ (by which I mean to include social science and disciplined discussion more generally). If this is the case then as well motivated as Popper’s search for a criterion is, and as useful as some of his other work has been – for instance pointing to falsification rather than confirmation – it might actually be counterproductive.

I recall one weblog I visited for a while which had a line from Karl Popper emblazoned on its headpiece something along the lines of “We might disagree but I might learn from you and you might learn from me”. Trouble was it was a fractious and arrogant weblog in my opinion. Karl – one of the worlds great hedgehogs – is supposed to have had a similar problem. One person suggested that his great political tome should really have been called “The Open Society: by one of its Enemies” ! A good line n’est pas?

I don’t say this out of disrespect to Popper. As the Menand article to which you linked argued (and as I was thinking as I read it before it said it) hedgehoggery tends to characterise a lot of people of great achievement.

Still there you have it – some thoughts preliminary to a post that I might never manage on the subject (but I’m hoping to!)