As Philip Tetlock so powerfully showed, most expertise isn’t worth nix if the criterion of expertise is whether you can demonstrate superior predictions about what will happen in the future. As he showed, most experts can’t predict any better than tolerably informed non-experts and some experts – particularly the hedgehogs who “know one big thing” – are substantially worse than your average Joe. By contrast, some expert ‘foxes’ who know many things are a tad better.
And of course prognosticating about the future is a human foible – we’re always doing it whether it’s useful or not. It can also involve status displays. I recall being invited to a dinner hosted by one of Australia’s noted businessmen during the GFC with 15 or so people around the table. No-one really wanted to discuss the nature of the events. Rather we were told by our host that he’d been talking to the head of Goldman’s in the US or some other great and powerful operator and he said so and so which involved some casual prediction. But it was painfully obvious to me – and I kept quiet about it because it would have been raining on others’ parade – that no-one really knew what would happen next and these corporate guys were less likely to know than a canny economist. But the conversation rolled on.
I’ve always thought it strange that when journos interview experts and discuss their predictions – for instance about the dollar – that they don’t also ask them for some casual variant of a confidence interval. Mightn’t the expert on such matters be expected to have an expert view on the value of his expertise? For instance there’s a literature on the extent to which past forecasts improve on simple rules (like predicting that today’s value will be tomorrow’s value or that tomorrow’s value will revert to the long run mean). If experts don’t volunteer such insights, shouldn’t they be asked for them? Oh but wait, the interviewer and the interviewee are all in it together – along with the listeners. The bullshitter, the bullshittee and the bullshitted.
Anyway, it’s a nice fantasy – one which might cut swathes through the industry of being and broadcasting talking heads.
In the meantime, I came upon this nice visual illustration of the ideas here.
The diagram being appealed to is the diagram above.
When you have any scientific theory, it has a range of validity. Think about that phrase for a minute: range of validity. What does that mean?
Let’s start with a small idea first to illustrate this: the idea that heat rises. Sure, you put a hot thing like a burning candle in a cool room, and the flame will heat up the air around it, and the hot air will rise. Sounds simple, until you ask the question, “how will the hot air rise?”
This doesn’t mean that there’s anything wrong with your theory that hot air rises, though. It means that your theory is incomplete, as are all scientific theories.And the answer to that one is more complex. Under some conditions (above right), the hot air will rise smoothly and the air will flow in a laminar fashion, while under other conditions (above left) the air will rise turbulently.