Why good thoughts block better ones: Cognitive biases and the psychopathology of knowledge

Keynes famously said that the hardest part of coming up with the General Theory was not coming up with the new ideas so much as escaping from the old ones. I’ve just run into a great article on the implications of happiness research for making policy (and yes there are implications) at least according to John F. Helliwell. Read the whole article if you wish – it’s not very long, not technical and very interesting and it’s here (pdf). Anyway, here’s the first half of it’s conclusion which struck me, perhaps because I’ve been thinking about the Keynes quote a lot recently not in the intellectual but in the organisational context. Why is it that ideas that are clearly good ones, which have negligible risks and are clearly worth giving a go, are not given a go? Laziness? Inertia? The fact that people are busy and fully engaged in existing routines? Undoubtedly. But at least the chess example brought it home to me in a concrete way (and of course as a chess player it describes perfectly why I’m not very good – well one reason!)

There seems to be sufficient evidence already in hand to encourage policy field trials and policy experiments implementing what is already known from subjective well-being research. If this is so, why has so little changed? The relatively slow progress from accumulating evidence to even experimental changes in policies and procedures is partly due to the human predilection, evident in medicine and all sciences (Nickerson 1998), to adhere to old ways despite the arrival of contrary evidence. Even chess masters unconsciously stop looking effectively for better strategies once they have something plausible in hand, enough so to drag the quality of their play down by three standard deviations in the skill distribution (Bilali? et al 2008, 654). 1

Caution has its own rewards, however, as the inherent conservatism of science can at least reduce the likelihood of running off in all directions. But if taking subjective well-being more seriously has the potential for increasing the quality of lives while reducing pressures on available resources, should there not at least be a stronger commitment to broaden the range of policy alternatives to include those with a strong chance of improving subjective well-being?

Postscript: Here’s the problem in the experiment.

(a) 2-Solution problem; (b) 1-Solution problem (these positions are based on an idea of Pertti Saariluoma). White to move in both problems. In (a) the familiar smothered mate solution is possible: 1. Qe6+ Kh8 2. Nf7+ Kg8 3. Nh6++ Kh8 4. Qg8+ Rxg8 5. Nf7#. The shorter optimal solution is: 1. Qe6+ Kh8 2. Qh6 Rd7 3. Qxh7#, or 2.. . . Kg8 3. Qxg7#. In (b) the smothered mate is no longer possible because Black’s bishop now covers f7. The optimal solution is still possible. 1. Qe6+ Kh8 (If 1….. Kf8, 2 Nxh7#) 2. Qh6 Rd7 3. Qxh7#, or 2.. . . Kg8 3. Qxg7#, or 2 . . . Bg6 3. Qxg7#. The crucial squares for the familiar solution are marked by rectangles (f7, g8, & g5) and the optimal solution by circles (b2, h6, h7, & g7) in (a).

The experimental procedure then involved comparing the ease with which people presented with the first problem saw the most efficient solution – given that they would almost certainly have seen the less efficient (smothered mate) solution first.

And from the article:

Across a range of skill levels, the presence of a familiar solution that ?rst came to mind 2 reduced the problem solving performance of the experts to that of players about three standard deviations lower in skill. Similar results were obtained using different problems and the more naturalistic instruction to ?nd the best move (Bilalic´ et al., 2008). Three standard deviations is a gulf in skill level. The chance of a player being beaten by one 600 Elo points lower is close to zero. And yet the Einstellung effect temporarily reduced the problem solving ability of the experts to that of the less skilled players. It is a very powerful effect. . . .

We show, by measuring players’ eye movements, that the mechanism by which the ?rst idea prevents a better idea coming to mind can be demonstrated. Crucially, we ?nd that players believed that they were actively searching for better solutions when in fact they continued to look at aspects of the problem related to the ?rst idea they consid- ered. This is why the Einstellung effect is pernicious – peo- ple do not realize that it is in?uencing their thoughts.

It was (I read somewhere) to avoid this (Einstellung) effect that Freud refused to read Nietzsche because he thought his ideas would pre-empt and so displace his own emerging ideas.

More generally while there are sub-disciplines variously entitled the ‘methodology’, the ‘sociology’, the ‘philosophy’, the ‘history’ and the ‘theory’ of one’s discipline, one of the most interesting and useful of these sub-disciplines might be called the ‘psycho-pathology’ of one’s discipline.  Such a sub-discipline, which I think there should be courses in which could be very valuable for the health and usefulness of the discipline would seek to generate greater self-awareness amongst practitioners of the psycho-pathology of their own discipline. Certainly there’s a dark psycho carnival going on in economics. Wouldn’t it be worth making some awareness of the psychological foibles of their discipline a routine part of the intellectual apparatus of economists?  Ditto, (mutatis mutandis) for lawyers, doctors and other professional practitioners.  Shouldn’t all people seeking to become ‘experts’ of one kind or another be familiar with all the foibles of expertise, as documented for instance by Philip Tetlock?

 

  1. this article is here btw, though there are better formatted, possibly later versions behind a paywall.[]
  2. the smothered mate in the first problem[]
This entry was posted in Chess, Economics and public policy, Methodology and tagged . Bookmark the permalink.
Subscribe
Notify of
guest

30 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Ben Tremblay
12 years ago

Replying to your Subject: in context of sofware design (specifically: computer system to faciliate public engagement in discussion of policy issues) I’ve found it true that “it’s hard to make things easy” and “creating a simple system calls for a lot of complexity”. (To go on in a more Zen mode: profound truths can be easy to know, but hard to understand. “The perfect answer becomes donkeys’ hitching-post.”)

We’re cognitive misers. “Good enough” works … and it’s adaptive to cling to that! (I just this morning added a “soup” snippet from Dr. Anders Ericsson‘s short essay on expertise with emphasis on “The superior quality of the experts’ mental representations allow them to adapt rapidly to changing circumstances and anticipate future events in advance.”) This applies to my work; the subjective clinging to a convenient / familiar idea is the ground I have to work with. That we can detach from that subjectivity for the sake of better outcomes is my (reasonable) hope. It’s the ” slow progress from accumulating evidence to even experimental changes in policies” that is my central concern.

p.s. In 2011 we’re using means and methods I considered ineffective more than a decade ago; it’s naive to think that everyone smiles on innovation and progress in technology. I’m sure the sophists (May I make passing reference to the psychopaths present among us?) are perfectly happy with the way “social sofware” shatters the public space.

Patrick
Patrick
12 years ago

I agree strongly with your postscript. My work actually does do some training around ‘framing’ and (although I hadn’t heard it called that) Einstellung. Unfortunately it is rather obscure optional training that I estimate about 1% have done!

I absolutely agree that such training on one’s own inbuilt weaknesses of thought process is essential for people who are, in essence, specialists in given thought processes.

Pedro
Pedro
12 years ago

Certainly for lawyers. We rely heavily on precedent documents, including ones from other firms. At Mallesons they (used to) put “Do not follow this precedent slavishly” on the front of each one. My experience is that people tend to be afraid to think for themselves, or at least to deviate from the herd. Perhaps part of the problem is that few people are capable of seriously original thought.

Ben Tremblay
12 years ago

What Pedro writes reminded me of early days in my project, when I looked at jurisprudence as paradigmatic of how precedence couldn’t be applied mechanically. What I looked into was how knowledge in that domain (like economics?) couldn’t be treated simply with the scientific method, where falsification wasn’t sufficient, where there had to be some synthesis. This all moved me towards a method I describe as dialectical, where what came before seeds what follows. (Is it only psychology that’s familiar with “cognitive priming”?)

p.s. I took Protension as a domain name with the thought that good thinking magnetizes and energizes good thinking.

Pedro
Pedro
12 years ago

Nicholas, Accountants don’t just look stuff up? Doctors for that matter? We lawyers are surely not the only lazybones.

trackback

[…] Nicholas Gruen’s post about Einstellung (a person’s predisposition to solve a given problem in a specific manner even though there are “better” or more appropriate methods of solving it) has given me an idea.  I would like to devise a couple of seminars for undergraduate Law students to be delivered as part of the subject Jurisprudence that I am next teaching at CDU in semester 2 2012 (so there’s plenty of time to work out how to do it). […]

Paul Frijters
Paul Frijters
12 years ago

regarding the post-script: we have just abolished the kind of self-analysis you call for in economics. It used to be part of the ‘History of economic thought’ courses and the courses on scientific methodology which are no longer part of most economic curricula. I think the reason we did away with those courses is that they were unpopular with the best students and with fellow economists. Perhaps this is ultimately attributable to the fact that all the incentives are now towards the seduction of our peers, who enjoy flattery and unquestioning application of their thoughts as much as anyone else prefers flattery over truth.

Regarding the question of the why of conservatism, I think Machiavelli had the right of it when he observed that any situation that lasts has people who benefit by that situation and who will defend it with all the vigour of an animal defending a territory, whilst those that stand to gain from change are merely lukewarm. What is amazing is just how hard people are willing to fight with their equals over what seem to be minute bits of territory.

trackback

[…] Nicholas Gruen’s post about Einstellung (a person’s predisposition to solve a given problem in a specific manner even though there are “better” or more appropriate methods of solving it) has given me an idea.  I would like to devise a couple of seminars for undergraduate Law students to be delivered as part of the subject Jurisprudence that I am next teaching at CDU in semester 2 2012 (so there’s plenty of time to work out how to do it). […]

trackback

[…] nice picture.   Good and bad things happening in the life of Lorenzo […]

TerjeP
TerjeP
12 years ago

I love the photograph shown with this article.

Incurious and Unread (aka Dave)
Incurious and Unread (aka Dave)
12 years ago

Conscious reasoning is deductive: it must proceed from assumptions or axioms. It is natural to base those axioms on conventional wisdom. If not that, what else? It takes brilliance and bravery to create a new axiom (eg “the speed of light is constant in all inertial frames”) and proceed deductively from that.

Ben Tremblay
12 years ago

My project explores the swampy ground that surrounds folk philosophy and conventional wisdom (“We know things we don’t understand.” –John Ralston Saul “On Equilibrium”) in the hope of building bridges from one axiom to another, building on what can be agreed on. The problematic I’m trying to dis-solve is that some practitioners operate by throwing sand in others’ eyes; winning the argument is all. If I went back to studies I’d drill into “epistemic responsibility” and try to extend it to “epistemic pessimism” or even “epistemic cynicism”.

Bob Moles
12 years ago

This is an interesting topic. The issue seems to be taking a less efficient route to a legitimate goal, rather than a more efficient route. This topic and ‘cognitive bias’ relates to an issue I am working on for a new book.
In my previous book Forensic Investigations and Miscarriages of Justice (2010)
http://netk.net.au/ForensicInvestigationsHome.asp
I set out details of cases from Australia, Canada and Britain where forensic specialists and lawyers not only took the less efficient route, but took one which was incorrect or inappropriate. In other words, they broke the rules (legally and ethically) to obtain a result which they preferred, but which was entirely unjust.
In Britain the Criminal Review Commission has now led to the overturning of some 320 convictions. My question is, why do so many well-educated forensic specialists, lawyers and judges not only behave inefficiently, but engage in conduct which they know or ought to know is wrong?
By this I mean ‘fraudulent’ evidence as opposed to mistaken evidence. And judges refusting to allow appeals as happened in the Derek Bentley case for nearly 50 years. When the appeal was eventually allowed, the grounds were so blindingly obvious that it beggars belief that the grounds for the appeal were not recognised right away.
This ‘cognitve bias’ if that is what it can be called – occurs in all common law legal systems. I’d be interested to hear what insights people have into this form of institutionalised error which prioritises ‘systemic’ values over those of the individual whilst proclaiming the reverse.
Bob

Paul Frijters
Paul Frijters
12 years ago

Bob, Nick,

you raise a fascinating issue. At heart I think your examples show the importance of ‘face’ to individuals and organisations. ‘Face’ is a truly remarkable thing, exceptionally hard to define but something organisations will go to great lengths to defend and keep up, despite all overt evidence to how untenable a particular ‘position’ is. The best definition I can give of ‘face’ is a professed self-image as to what is important to the organisation (its ideals) and as to how it intends to deal with violations of its own ideals (punishment and reward). The interesting thing about the concept of face is that it is not necessary at all for either those who keep up the face or those who regard it to truly believe it or even to adhere to the professed punishment schedule. It is perfectly well understood at many levels of the organisation that the face is not the same as the truth. That makes it a kind of ‘outward’ story, but one to which individuals and organisations fanatically cling.

Reflect on how ‘face’ fits both your examples: Nick’s college clung to the idea it was really fairminded and Christian despite clear evidence to the contrary and like Bob’s legal organisations and legal practitioners will have stoically professed a self-image of incorruptability despite clear evidence to the contrary. And I bet the people involved to this day have kept that face up. The both of you got punished for rallying against the face, i.e. calling it out into the open and saying it was a false face. That is why you were the enemies of the organisations, needing to be resisted despite any monetary costs.

Ben Tremblay
12 years ago

I have to say that I really enjoyed my studies in cog- and social-psych. (Actually I was seduced by historiography!) I could easily have followed any of the many tracks that opened up, even clinical methods / statistics. “Law and Psych” (which was for more criminology than law was fascinating; a ton of craft involved. But I had something like a personal mandate, a mission statement.

“Why has so little changed? The relatively slow progress from accumulating evidence to even experimental changes in policies …” I quote this to draw attention to Bob Moles’ submission. (#15) What I’d like to suggest is that the problematic is not restricted to any one field of study or practice. (I consider The Innocence Project to be one of my project’s target audiences.)

What I bit into (35yrs ago) was, “How do we make such bad decisions?” Even the most well intentioned of principled practitioners gets caught in error. But when error mode is systemic and even tradition … just how did we decide to overthrow the democratically elected government in Chile? (That’s my test case; there’s far more material for our war in Iraq.)
I’m no scholar. So as a communications technologist I set aside one academic fascination after another with a single goal in mind: to facilitate decision making in hopes of enabling discourse.

As Harvard’s Michael Sandel puts it, the study of ethical dilemmas estranges us from the familiar. (see his “Justice: What’s the Right Thing to Do?“) I hope my system makes it clear that convenient thinking exposes us to catastrophic consequences.

Bob Moles
12 years ago

thank you Nick and Paul for your comments. I must say that I was quite shocked when I first came to South Australia and twigged on to the fact that there were some serious miscarriages of justice. It was the very basic errors in the case which stunned me. We did a program with ABC 4 Corners (2001 – I can send you a copy of the DVD). The response from the Attorney-General was that he had asked those whose conduct had been impugned (the chief prosecutor and forensic scientist) if he should hold an inquiry. Apparently, they said there wasn’t anything in the criticisms, so no need. The AG then told parliament that the ABC were just being ‘mischievious’ or deceitful. I then wrote the State of Injustice book (2004) which sets out clearly the errors in each of the cases (I can send you a copy) – still nothing is done. In the meantime, the Medical Board / Tribunal and now the Supreme Court judges have said that it does not amount to unprofessional conduct if forensic experts fail to disclose the result of an exculpatory scientific test. The interesting thing about that judgment is that it does not cite any legal authority for that proposition – and it does not deal with extensive legal authority in Australia the UK and Canada which says the exact opposite. In fact, there are no legal references in the judgment at all – http://netk.net.au/Keogh/Keogh104.asp
At some stage of course national and international pressure will bring about some review of those judgments. But in the meantime, innocent people must remain incarcerated to maintain ‘face’.
In fact, within the Sikh tradition, the concept of ‘face’ is very important. I was married in a Sikh temple in Malaysia some years ago.
The irony is of course, that I am seen as someone who is trying to subvert ‘the system’ whereas I see myself as someone who is trying to improve it. My strong feelings about this type of injustice stem from my time in Belfast as a student during ‘the troubles’.
Any system which turns a blind eye to injustice usually has to pay a greater price down the track. However, the thing which concerns me is those officials who misapply the principles in the short term – are they not interested in their reputations in the longer term? Or do they think that we’ll never get to that.
I’m keen to follow up on the ‘group-think’ issue and if you have any further thoughts on individual or systemic factors in cognitive bias please tell me about them. The book I’m working on is “Cognitive Bias and Miscarriages of Justice – Psychology and Human Rights”. We’re using cases from Australia, UK and Canada, and we have a Canadian Superior Court judge as a joint author – he was a clinical psychologist before he took to the law.
Its early stages yet, but I’m keen to look at false confessions and unreliable eye-witness testimony as well.
Thanks to you both for firing me up again this week, much appreciated,
Bob

Ben Tremblay
12 years ago

p.s. re: Nicholas Gruen’s “Groupthink: the enemy within” I hope you’re familiar with the story behind the decision to launch that doomed STS-107, Space Shuttle Columbia’s final mission. In short, those attending the mid-night management teleconference knew that Morton Thiokal’s tank seals allowed blow-by, especially when cold. (see “Telecon Meeting (Ethical Decisions – Morton Thiokol and the Challenger Disaster” and “Ethical Decisions” by NAE’s “Online Ethics Center”) And they all knew that the Shuttle was in a “deep chill” condition. The tipping point? “Take your engineers’ hats off, and put your managers’ hats on.” Case closed.

I don’t call this group-think. What I encountered mid-70s with regards to Canada’s GATT policy I would call paradigmatic group-think. I suggest that the above is something else. In cases like this I don’t want just a transcript, or a well considered essay. I want forensic analysis of the logic. I want “diamond cutter” logic. I want an ice pick!

Bob Moles
12 years ago

One further point I meant to make – Nick’s comments about the jury – if I had been on the jury in the Keogh case I would have supported a conviction. The same is true of most of the miscarriage of justice cases I have looked at. If the jury are given wrong information, we cannot criticise them for acting upon it. We should bear in mind that in the Chamberlain case the jury was told there was foetal blood all over the front of the car. That turned out to be wrong – as was every other element of the scientific case. So too with the Splatt case in South Australia. Every item of the more than 20 scientific components of the case was found to contain errors. It took 11 days to convict him – and 192 hearing days in the Royal Commission to identify the errors.
As we pointed out in the Forensic Investigations book, the rules about admissibility of scientific expert evidence are pretty clear and sensible. Its just that they are not applied in practice – and this means that the prosecuting, defence counsel and judges are at fault. The question is whether they are willing to own up to error when the issue is raised subsequently?
There are also serious issues around criminal law being state based. This means that in South Australia all Supreme Court judges there were trained at just one law school – the Uni of Adelaide. They all practiced in this one (small) city for say 20 years – then become judges in that same small city. That could not happen in the UK for example, where a judge becomes a judge for the whole of England and Wales. Maybe the parochialism of the jurisdictional basis compounds the groupthink mentality. After all, what pressure might there be upon a solicitor-general to recommend a case not be referred to the appeal court?
Should it really take 4 years to answer a petition on such a question?
Bob

Dwight Towers
12 years ago

Thanks to Nicholas for the initial post and continued comments (the college story sounds pretty par for the course). Thanks also to the other commenters – loads of ideas and links to track down

@ Ben. On the Columbia (I think you linked in error in your comment at 22), Edward Tufte has some great work on the consequences of using Powerpoint to communicate important information (like the tiles having been scrapped off at launch).
http://timothyblee.com/2009/11/12/bottom-up-thinker-edward-tufte/

@Bob (21) “Any system which turns a blind eye to injustice usually has to pay a greater price down the track.”
I’ve just been reading a lot about resilience and ‘panarchy’ and so on. The suppression of feedback/variance in the search for “maximum sustainable yield” simply means that crashes – when they come – are bigger and nastier. Perhaps a useful analogy there?

Best wishes to all

Dwight Towers

trackback

[…] an earlier post, and one of a series by me and subsequently Ken as well, I suggested that an important part of any […]

Tom
Tom
12 years ago

In reply to Bob Moles Belfast experience as a student: This is now much easier since local niche directories like accountants belfast allow to separate the wheat from the chuff. But I understand that his studies were many years ago when such help was unavailable.

trackback

[…] even here it’s not system information, but information between buyer and seller). But as on other occasions, I’m addressing myself to the psychopathology of the […]