Category Archives: Methodology

If our models are correct, then people are smarter than we realised!

Posted on by
7

Whilst making pies yesterday I happened to recall a sentence I read 7 or so years ago, which suddenly struck me as very silly. So I just looked it up to make sure I hadn’t imagined it.

I didn’t.

Here’s the whole paragraph.

A final point worth noting on gang wars is that their strategic
aspects are not lost on the participants. Gangs use violence on
their competition’s turf as an explicit strategy for shifting demand
to their own territory. As one former member of the rival gang put
it during a gang war:
See the thing is they [the gang for which we have data] got all these
places to sell, they got the numbers [of sellers], you know. It’s not like we can
really do what they doing. So we gotta try get some kinda advantage, a
business advantage. If we start shooting around there [the other gang's
territory], nobody, and I mean it you dig, nobody gonna step on their turf. But
we gotta be careful, ’cause they can shoot around here too and then we all
f——. But, it’s like we ain’t got a lotta moves we can make, so I see shooting in
their ‘hood as one way to help us.
In fact, in some cases, a gang engages in drive-by shootings on a
rival’s turf, firing into the air. The intention is not to hurt anyone,
but rather to scare potential buyers. It is interesting to note that 
the gang member understands the game-theoretic consequences
of such actions corresponding to retaliation by the rival, in which
case both parties are worse off than if no violence had occurred.  (my emphasis)

Some of you might recognise the paper, given it was popularised in Freakonomics. It’s “An Economic Analysis of a Drug-Selling Gang’s Finances” by Levitt and Venkatesh.

On reflection I might be unfair in calling it silly, but it reflect a strange pathology I think I often see in economics. Economists attempt to understand the consequences of human behaviour, so they build models. Models are necessarily simplified, so we include things like the assumption of rationality, which is then described in terms of optimal behaviour from an individual standpoint. We then use these to try and understand the world.

Except somewhere along the road economists take a detour, and rationality stops being a simplifying assumption, and starts being a way of thinking clever and properly educated people (i.e economists) do. Subsequently when economists observe real life behaviour like the drug gangs, the conclusion isn’t;

Looks like our game theory models are useful for understanding strategic behaviour in drug gangs. Cool.

Rather it is;

Drug dealers understand Game Theory?! Amazing!

Which is a very silly way to go about trying to understand human societies and economies.

I also recall a post on the Freakonomics blog [fn1] in which Levitt remembers fondly a conversation with Milton Friedman early in his career. Levitt had said he was careful to save money on his early salary, and Friedman chided him for being irrational and neglecting to take into account that Levitt’s higher salary in the future would allow him to smooth out his consumption. Here Friedman was clearly drawing on the Permanent Income Hypothesis, his “best scientific work” and one of the cited reasons for his Riksbank prize.

So we have Friedman chiding people for not acting the way his models said they would. A very smart man and a positivist no less!

It is a very strange discipline that is surprised when its models are useful, and cranky at reality when they are not.

 

[fn1] Which I can’t find. I guess they wanted to make sure all memory of the blog prior to the past few years of hissy fits is damned.

 

 

 

Economists as engineers and humbler, better scientists

Posted on by
34

Here’s a paragraph I wrote about fifteen years ago.

The culture of economic expertise places inadequate weight on integrating insights from multiple perspectives, that it frequently places an unreasonably high ‘burden of proof’ on heterodox views, and that it has a penchant for spelling out the normative implications of its analysis by way of ‘peremptory rules’ which narrow the scope for dialogue.

This is just a sketch of a post, because I don’t have time for more, but for some time I’ve wanted to put down a marker about one of the ways in which results from Kaggle are instructive. The winners of Kaggle competitions often win not by trying to build the One True Model of the phenomena they’re trying to model, but rather by building a large number of models all of which have some explanatory power and which are independent of each other and then aggregating their insights. Random forests approaches often perform very well. One might call it a ‘wisdom of algorithmic crowds’ approach.

In any event, this is the antithesis of much economic modelling which involves reaching for the One True Model and then parameterising it.

I was reminded of this by David Colander’s latest piece urging humility on economists (pdf). His proposal? That instead of thinking of themselves as analogous to dentists (which Keynes suggested) economists should think of themselves as engineers:

An engineer’s approach to modeling such a complex system as the macro economy would likely focus much more on statistical models and methods of pulling patterns out of the data. It would explore a wide variety of formal models to gain analytic insight, and then would integrate the many variety of models with the statistical models to interpret the patterns. That would involve a fundamentally different way of doing macro and of thinking about macro problems. Similarly, with micro. Economists focus much of their applied micro policy discussion on Pareto optimal solutions even though we know that all actual policies will violate Pareto optimality. We can contort our micro policy models designed to provide Pareto optimal solutions to provide insight into non-Pareto optimal solutions, but, generally, that contortion comes at a cost. It means that we spend less time discussing other models that betting fit not Pareto optimal solutions, but “reasonable person solutions” that more closely reflect society’s value judgments. An engineering applied microeconomics would likely have an entire branch devoted to measuring society’s value judgments and integrating those judgments into applied policy instruments. Our scientific applied micro leaves the topic almost totally undiscussed.

Quite.

Other required reading on the point is this paper entitled “Identification Problems in the Social Sciences and Life”.

I made the latter point in a discussion of discursive collapse a while back.

 

Counteracting our biases

Posted on by
6

In an earlier post, and one of a series by me and subsequently Ken as well, I suggested that an important part of any professional education should be a kind of counter-narrative in which those who learn a profession are also made familiar with that profession’s cognitive biases, with a view to lessening them in practice.

Nice to see this kind of thing is beginning to be taken seriously in management books. Actually it might have been taken seriously before now, I wouldn’t know because I don’t read management books. But I occasionally browse them and it hasn’t seemed to prominent in my browsing. In any event Daniel Kahneman et al have a long article in the HBR on the behavioural economics of business decision making.

And a useful check-list for when an organisation is making big decisions. Viz: Continue reading →

Why good thoughts block better ones: Cognitive biases and the psychopathology of knowledge

Posted on by
27

Keynes famously said that the hardest part of coming up with the General Theory was not coming up with the new ideas so much as escaping from the old ones. I’ve just run into a great article on the implications of happiness research for making policy (and yes there are implications) at least according to John F. Helliwell. Read the whole article if you wish – it’s not very long, not technical and very interesting and it’s here (pdf). Anyway, here’s the first half of it’s conclusion which struck me, perhaps because I’ve been thinking about the Keynes quote a lot recently not in the intellectual but in the organisational context. Why is it that ideas that are clearly good ones, which have negligible risks and are clearly worth giving a go, are not given a go? Laziness? Inertia? The fact that people are busy and fully engaged in existing routines? Undoubtedly. But at least the chess example brought it home to me in a concrete way (and of course as a chess player it describes perfectly why I’m not very good – well one reason!)

There seems to be sufficient evidence already in hand to encourage policy field trials and policy experiments implementing what is already known from subjective well-being research. If this is so, why has so little changed? The relatively slow progress from accumulating evidence to even experimental changes in policies and procedures is partly due to the human predilection, evident in medicine and all sciences (Nickerson 1998), to adhere to old ways despite the arrival of contrary evidence. Even chess masters unconsciously stop looking effectively for better strategies once they have something plausible in hand, enough so to drag the quality of their play down by three standard deviations in the skill distribution (Bilalić et al 2008, 654). [this article is here btw, though there are better formatted, possibly later versions behind a paywall.]

Caution has its own rewards, however, as the inherent conservatism of science can at least reduce the likelihood of running off in all directions. But if taking subjective well-being more seriously has the potential for increasing the quality of lives while reducing pressures on available resources, should there not at least be a stronger commitment to broaden the range of policy alternatives to include those with a strong chance of improving subjective well-being?

Postscript: Here’s the problem in the experiment.

(a) 2-Solution problem; (b) 1-Solution problem (these positions are based on an idea of Pertti Saariluoma). White to move in both problems. In (a) the familiar smothered mate solution is possible: 1. Qe6+ Kh8 2. Nf7+ Kg8 3. Nh6++ Kh8 4. Qg8+ Rxg8 5. Nf7#. The shorter optimal solution is: 1. Qe6+ Kh8 2. Qh6 Rd7 3. Qxh7#, or 2.. . . Kg8 3. Qxg7#. In (b) the smothered mate is no longer possible because Black’s bishop now covers f7. The optimal solution is still possible. 1. Qe6+ Kh8 (If 1..... Kf8, 2 Nxh7#) 2. Qh6 Rd7 3. Qxh7#, or 2.. . . Kg8 3. Qxg7#, or 2 . . . Bg6 3. Qxg7#. The crucial squares for the familiar solution are marked by rectangles (f7, g8, & g5) and the optimal solution by circles (b2, h6, h7, & g7) in (a).

The experimental procedure then involved comparing the ease with which people presented with the first problem saw the most efficient solution – given that they would almost certainly have seen the less efficient (smothered mate) solution first.

And from the article:

Across a range of skill levels, the presence of a familiar solution that first came to mind [the smothered mate in the first problem] reduced the problem solving performance of the experts to that of players about three standard deviations lower in skill. Similar results were obtained using different problems and the more naturalistic instruction to find the best move (Bilalic´ et al., 2008). Three standard deviations is a gulf in skill level. The chance of a player being beaten by one 600 Elo points lower is close to zero. And yet the Einstellung effect temporarily reduced the problem solving ability of the experts to that of the less skilled players. It is a very powerful effect. . . .

We show, by measuring players’ eye movements, that the mechanism by which the first idea prevents a better idea coming to mind can be demonstrated. Crucially, we find that players believed that they were actively searching for better solutions when in fact they continued to look at aspects of the problem related to the first idea they consid- ered. This is why the Einstellung effect is pernicious – peo- ple do not realize that it is influencing their thoughts.

It was (I read somewhere) to avoid this (Einstellung) effect that Freud refused to read Nietzsche because he thought his ideas would pre-empt and so displace his own emerging ideas.

More generally while there are sub-disciplines variously entitled the ‘methodology’, the ‘sociology’, the ‘philosophy’, the ‘history’ and the ‘theory’ of one’s discipline, one of the most interesting and useful of these sub-disciplines might be called the ‘psycho-pathology’ of one’s discipline.  Such a sub-discipline, which I think there should be courses in which could be very valuable for the health and usefulness of the discipline would seek to generate greater self-awareness amongst practitioners of the psycho-pathology of their own discipline. Certainly there’s a dark psycho carnival going on in economics. Wouldn’t it be worth making some awareness of the psychological foibles of their discipline a routine part of the intellectual apparatus of economists?  Ditto, (mutatis mutandis) for lawyers, doctors and other professional practitioners.  Shouldn’t all people seeking to become ‘experts’ of one kind or another be familiar with all the foibles of expertise, as documented for instance by Philip Tetlock?