Uncertainty, Part 1: McGurk

As one the best illustrations of the way our minds deal with uncertainty, consider the following video. Please listen and watch at least 30 seconds so you can experience the three sequences of spoken words.

Pretty much all humans who watch the video will first hear “ba ba” in the first sequence, then “da da” in the second sequence, and “va va” in the third sequence. Yet if you play the video but close your eyes, everyone will hear only “ba ba” in any of the sequences. Hearing “da da” or “va va” is an illusion caused by the fact that the video shows the lips moving as if “da da” or “va va” is said, but the actual sound is “ba ba” throughout.

It is a wonderful example of how the human subconscious recognises uncertainty and resolves it without the conscious being involved at all: in the second sequence the auditory part of the brain deduces that what is said is “ba ba”, whilst the visual part of the brain deduces that what is said starts with a “d”. These conflicting pieces of information are then combined such that the visual information dominates and the conscious is told the sound is “da da”. The conscious brain is not even alerted to the uncertainty as the sound “da da” is relayed in real time with no hesitation.

What makes the example especially interesting is that conscious awareness of what is going on does not change what the brain tells us about the sound: you can watch the video thousands of times and try and train yourself to hear “ba ba” all the time, but even scientists who studied the illusion for decades still hear “da da” when seeing the mouth move as if a “d” is uttered. The subconscious resolves the uncertainty in the same way regardless of how the conscious mind tries to direct it.

This “McGurk effect” is also a case of where more information actually leads to the wrong perception. Normally, visual information adds to auditory information to improve the processing of spoken language, but on this occasion more information leads to a conflict between information at which point the “correct” information gets disregarded.

It turns out that our subconscious does something similar with everything we see and hear (or sense in any other way): uncertainty about what is sensed is resolved many thousands of times every second to produce a sensation of certainty around what is going on. I am at this very moment hence “seeing” a room with chairs, a bench, a clock, a labtop, a tv, etc. There is no hint at all of uncertainty, such as whether I am seeing a pen or a chopstick, a chair leg or a lamppost, the side of a window glass or the side of a piece of paper stuck on the window glass: my subconscious simply paints a picture of what I am seeing, with no role for uncertainty whatsoever.

The pretense of certainty in what is seen and heard thus occurs entirely automatically and is the case for every normal human: we have a brain equipped to deduce certainty, not to live with radical uncertainty on everything we see and hear. What is amazing is that that ‘certain’ view emerges even though in actuality my eyes scan but a very tiny amount of the true visual field: nearly everything I think I see is made up from extrapolation, involving lots of little uncertainties. The brain thus deduces something is “a table” from a few glimpses, combined with the expectation that there is a table, culminating in a picture in our minds full of details that are not really seen at all (like the contours of the whole thing).

There seems an obvious reason for this penchant of our subconscious to present our conscious mind with the pretense of certainty: it allows for quick decision making without distractions. I don’t need to spend energy on the tiny probability that the house cat is actually a devouring tiger and can concentrate on my typing because my subconscious mind rejects out of hand that the house cat is a devouring tiger. My conscious mind can tell me there might be an escaped tiger from the zoo nearby (which then creates a hint of anxiety) but in the normal cause of events such thoughts never enter the mind for the excellent reason that it wastes time and distracts from what we are doing.

Our self-pretense of certainty is thus evolutionary efficient, a ‘hack’ you might say: by simply not even alerting our consciousness of the thousand and one uncertainties, our conscious mind is kept in reserve for more rewarding problems to think about. Being aware of uncertainty slows our decision making down immensely and thus needs to be particularly rewarding to even contemplate.

Let me give two more example of how our mind “fills in the blanks” in a way that is efficient but strictly speaking totally wrong. Read the following sentence as quick as you can:

“Arocdnicg to rsceearch at Cmabrigde Uinervtisy, it deosn’t mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the rghit pcale.”

Now, with this ‘illusion’ it turns out not everyone reacts the same, unlike the McGurk effect above which pretty much 100% succumbs to. Still, the majority, me included, will be able to read the sentence above very quickly and ‘see’ that the sentence is meant to say

“According to a research at Cambridge University, it doesn’t matter in what order the letters in a word are, the only important thing is that the first and last letter be at the right place.”

This example is illustrative of the fact that the majority of people do not truly read every letter of a word but essentially guess at the order of the letters in the word on the basis of what the mind expects to read, helped with some information on the more important parts of the word (like the start and the finish). Our minds simply fill in lots of the blanks, partly on the basis of the overall shape of the word (ie we do not truly look hard at the individual letters either but make them up from more limited information too), and even because of what we expect to read in that part of a sentence. So after deducing in the sentence above that we read “Cambridge”, any subsequent word starting with “U” is going to almost automatically be guessed to be “University”.

Once again, filling in the blanks on the basis of deductions so far is efficient, even though the deductions are strictly speaking untrue. Our minds are not truly reading “Cambridge University” when the letters are “Cmabrigde Uinervtisy”. Yet, once again, our consciousness is not told to worry about the actual information even though in this case our conscious mind does get alerted that something is not quite right, which is probably because a bit of higher-order reasoning is needed to unscramble the words quickly. Yet, since there is no obvious alternative candidate way in which the sentence was “supposed” to be read, our conscious mind is not told there is a problem and is kept in a state of blissful certainty: we are not motivated to ‘check’ whether there is another possible sentence hidden in the same letters. All this happens at pretty high speed (I read the sentence above in 3-5 seconds).

A third example goes into the question of how our minds deal with uncertainty about the future. Consider the following survey question on which many macro-economic forecasts are based:

“Do you think your business will expand or contract the next 3 months?”

Note how this question only allows answers to be a definitive expectation of “expand” or “contract”. To a purist statistician this question is nonsense because one cannot know what is going to happen the next 3 months and one thus “should” have a “probability distribution” in mind on all the events that might cause the business to expand and contract. A statistician would thus like to ask business leaders questions like

“what probability do you assign to the event that your business will expand 10% or more in the next three months”.

The problem with such probabilistic questions is that many respondents will not be able to answer. Most people do not think in terms of probabilities of future events, let alone probabilities of broad categories like “10% or more”: it takes enormous training to think that way about something. That’s why the ‘incorrect’ version of questions about the future dominate surveys.

The question on business expansion thus divides neatly into two scenarios: up or down, and simply asks people to say which one they believe holds. This is how people think about the future: in terms of storylines, ie scenarios. Most of us do not really think in terms of uncertainties but more in terms of competing “scenarios”, where everything within each scenario is taken to be certain. A statistician would call a single scenario a “point estimate”. So via allowing multiple scenarios people can grasp a few point estimates. To a purist statistician any exact scenario occurs with a probability of zero and is thus useless, yet in politics and business, having several scenarios to mull over is about as sophisticated as one can talk about the future.

The general insight is hence that our very thought processes are set up to resolve uncertainty such that the conscious mind deals mainly in total certainties, even when considering the future. That appears to be an evolutionary optimal strategy: allowing more than a tiny bit of uncertainty about the present or more than a few scenarios for the future just distracts us too much and paralyses our ability to take action. In order to be decisive, we are set up to be extremely bad at openly thinking in terms of uncertainty.

This is also how we should think of uncertainty-resolution in social groups: as something that usually is efficient and necessary to be able to decide quickly. No politician can afford to sound uncertain, let alone say something as scientific as “With my policies there is a 5% higher chance of economic growth”. Group leaders must exude certainty lest they be seen as weak and not leaders at all. The inability of individuals and groups to live with much uncertainty is then not a sign of how backwards they are, but a ‘hack’ that allows faster decision making.  This is even more true in a crisis: the more that humans sense quick decisions need to be taken, the less able they as individuals or group can allow for the possibility of uncertainty. They need ‘to run’ with whatever presents itself as the certainty of that moment.

A major question for large groups is then how to avoid the trap of having the group as a whole become totally certain about something that just isn’t so. In terms of the McGurk effect, the question is how the group as a whole can have mechanisms to ensure that the “truth” about the sound is recognised by some people, combined with some mechanism to convince the others despite conflicting information. One obvious answer is that you want some people in the group as a whole that only listen (and do not use their eyes), for they will then hear something totally different in the McGurk video to what those who see as well as listen think they hear. So one answer to the trap of certainty is organised radical diversity in perception.

This and many other group considerations around uncertainty are for a future part though. They involve the issue of how emotions lead to the search for agency stories in situations of uncertainty. They involve the issue of how a leader must respond to the demand to exude certainty and control. They involve mechanisms like independence, public inquiries, and ‘conscience’ that very large groups develop to maintain and reward diversity of perspective. They involve the impossibility of representative leaders to openly maintain perspective if very strong group emotions get involved. The bigger idea in the background is thus that our societies need and already have lots of ‘counter-hacks’ to limit the damage of the many “usually efficient hacks” we as individuals and social groups have.

This entry was posted in Economics and public policy, Life, Media, Philosophy, Political theory, Science, Social, Theatre. Bookmark the permalink.

21 Responses to Uncertainty, Part 1: McGurk

  1. My experience of that McGurk vid is that the second and third parts feel ,muffled unclear enough that I’d say I’m not sure what he actually said. Also felt unease, he is obviously straining not at ease.

  2. Conrad says:

    I’m obviously glad you’ve taken an interest in low-level language stuff :). I think the precursor problem which you seem to have somewhat given up on is trying to get people to understand that more or less everything they consciously perceive is just a construction their brains have stitched temporarily together. If people had more diversity of thought, or even realised that their realities wern’t necessarily correct, then perhaps you wouldn’t need such radical diversity of groups.

    • paul frijters says:

      indeed that is somewhat the precursor problem. Rather than say I give up though, it is more accurate to say I am trying to understand all aspects of this issue myself, and generating material that allows others to engage with bits of it without adopting the whole.
      But yes, part of the motivation was questions of people like Nick on how one could be sure of something. This post goes to the issue of what certainty actually means to people. As you say, it is all “a construction their brains have stitched temporarily together”.

      Is this your area of expertise?

      • Conrad says:

        I mainly work on stuff do to with cognitive science of language (especially but not exclusively on how people read). I don’t work on the McGurk illusion, although it is a useful example for people to think about for reasons you discuss (funnily enough, I currently have a video linked to it for some first year teaching I do).

        If people haven’t thought about it, more or less everyone has hearing damage after about 30, and we don’t typically talk in silent rooms. So most of the time we don’t hear everything due to car’s passing, coughing etc. . Using multiple cues fixes the signal up, which is really useful given you typically want to get to a known category (a sound or word) in the end.

        The other famous effect where you don’t even need multiple cues to perceive things as a category is known as phoneme categorisation, where you basically only hear the subset of sounds (phonemes) in your language — which everyone who has learnt a second language late in life should know. You learn these in the first year of life. There are in fact quite a few effects like this known to exist in other aspects of things people do (e.g. categorising facial expressions), and some are known to exist in animals too. So they are fairly pervasive.

        I don’t know enough about decision making to meaningfully evaluate it , but when I read the Gigerenzer stuff on decision making, I suspect a similar warping often goes on with issues where there are points in decision space (i.e. typical opinions) that people gravitate towards (obviously this would be beliefs and not perception).

  3. Saupreiss says:

    Looking forward to Part 2.
    Just for the record, it is a long way from optical illusions to cognitive illusions / systematic biases. Which is where you seem to want to go …
    As I said, looking forward to Part 2.

    • Conrad says:

      At least for things which can be described in relatively concrete categories, I imagine that’s less true than you might think. In this case, the categorical boundaries we create are often pretty arbitrary and often based on perceptual attributes, but they become instantiated in the way we think about things. Many of the racial stereotypes are based on these. For examine, people who are “black” vary widely in their genetics (indeed there is more genetic diversity in Northern Africa than the rest of the world combined — and there are groups which are clearly separate, like Africans and people form PNG), yet they get treated similarly in many places.

      Indeed, lots of decisions are based on categories, not continuous variables, and so if you use biased categories you will clearly will end up with biased decisions.

      • Saupreiss says:

        Interesting point but … . The set of decisions where that becomes relevant seems a pretty small one. Getting out of bed, navigating the way to my office, working on a manuscript, going home, shopping, having dinner, … all of these mundane decisions seem not affected by these kind of concerns. Importantly, the work by Gigerenzer and various collaborators has demonstrated that cognitive illusions can be turned on, off, and even reversed systematically. They are essentially not as robust as some optical illusions are.

        • Conrad says:

          Does it really matter that much whether they are as robust as the perceptual ones (which is unsurprising ) ? If people still do them a lot and there is not much you can do to stop people doing them, the main benefit would mainly be collecting the losers and trying to change their behavior, generally after the fact.

          For example, I used to work with a semi-reformed gambling addict, and his mental hack to not going broke was to go down to the TAB each day at lunchtime and only spend $20 (don’t bring a credit card). There are probably lots of ways stop to gambling like this, but this doesn’t stop any number of people succumbing to sunk-cost effects even when they know that they will be the statistical loser (c.f., less obvious situations like sunk-cost in human relationships etc.).

          The problem is even more difficult in less constrained domains where the likely outcomes are less obvious. Just look all the vaccination mess in Euroland at the moment (still a pretty constrained domain). Clearly even reasonably smart people don’t always understand what base-rates are (something we apriori knew). But the damage is done already. Where was the cognitive hack to stop this?

    • paul frijters says:

      Hi Andreas,

      if you think of them as illusions and biases, then I have failed in the post to make you realise the central point, which is that we do not think in the way science (or economics) advocates one should think. We think totally differently. It is then basically silly to talk of that in terms of biases and illusions. Rather, rationality, probabilistic thinking, bayesian updating, etc., are deviations, only maintained at huge cost with often very limited payoff.

      Behavioural economics has gotten into a cul de sac by adopting a total delusion as the standard model, and saying all deviations from that delusion are illusions and biases. It leads to a huge waste of research effort because no-one even remotely resembles the standard model. Its akin to trying to convince a monkey that it is a fish, apart from a few biases.

      The language of mental hacks and counter-hacks is much more useful because it makes one look at what combinations of mental hacks and counter-hacks are useful for what type of problem or situation.

  4. ianl says:

    >” … a construction their brains have stitched temporarily together” [reality, that is]

    Driving daily in traffic.

    It seems almost everyone who does this has stitched together the same construction. Just as well …

    • conrad says:

      It might seem like that to you, but you are wrong. You have noticed something that seems intuitively obvious and hence that is what you quite reasonably believe, and over-generalised it based on that, just as I suspect Paul would predict here.

      The physical reality is that when driving to work, you are driving with about 7% of people who are stereo blind. These people can have entirely normal looking eyes, but for whatever reason they may not be able to use things like retinal disparity cues that everyone else uses. So their depth perception (which is of course really important for driving), is really quite different to yours (or 93% of people are quite different to you if you are one of the 7%).

      • paul frijters says:

        very true, but Ian’s point is also important, which is to note just how well this all works and how in many settings so few costly mistakes get made. It is amazing what natural and social evolution comes up with in terms of behaviour and interaction that ‘just works’.

        One reason to find this so fascinating is that the sheer complexity and diversity of how realities and behaviours get constructed mean you just have no chance beforehand to predict a lot of social solutions to complex problems. Many social solutions can really only be found by trial and error, lots of groping in the dark and accidental paths. And yet if that trial and error were presented to others as groping in the dark….

        • Paul
          “the sheer complexity and diversity of how realities and behaviours get constructed “
          You might yet become an artist 😊

        • Nicholas Gruen says:

          Thanks Paul,

          Good post.

          I think I agree with the point you’re making, and certainly with the point you’re making re behavioural economics.

          As you know I also wrote up ‘hacks’ in my piece on embodied cognition. Everything is a hack, hacked together with other hacks and this is how we should be thinking of perception and action – and how the pragmatists did try to think. And he whose name cannot be spelled – Nietzsche – thought.

          I think it’s strange to describe this as groping for ‘certainty’. Certainty is a strong state. What one’s system is groping for is coherent cognition. So it jumps to conclusions from fragmentary evidence. Using the word ‘certainty’ seems to project the events you’re depicting into the same kind of reductive and falsifying vocabulary as the behavioural economics folks.

          People need perception that’s good enough to act in the world. They’re not looking for certainty any more than they think of themselves as ‘Bayesians’ (even if they knew what that meant). They know they can be wrong from time to time.

          • paul frijters says:

            yes, they know they can be wrong from time to time, but in a way that is a counter-hack to how their perception tells them about the outside world.
            So yes, ‘coherent cognition’, but also that sense of certainty. What I said in the post, perhaps not strongly enough, is that cognitive awareness of uncertainty is extremely taxing cognitively, something the system seems to try and avoid. So in order to be coherent, our mental systems seems to ‘resolve’ uncertainty and simply present our conscious mind with a perception that comes with a sense of certainty (ie, without the motivation to self-doubt). So the counter-hacks do not get activated too often.

            • KT2 says:

              Paul, I fear your pivot from panic & hysteria, to neuroscience – and I’d laid London to a brick – the second installment will track back your own McGurk effected” blindness. When it is applied to you it actually weakens your case.

              Are you also going to invoke Van der Waals forces to say we don’t actually ‘touch’ anything? So how could we get the virus? 

              The only new thing here is your ability to throw everything to prop your panic.

              NG said “People need perception that’s good enough to act in the world. They’re not looking for certainty any more than they think of themselves as ‘Bayesians’ (even if they knew what that meant). They know they can be wrong from time to time.”

              Too true. Blind? Can’t decide?


              “The McGurk effect in the time of pandemic: Age-dependent adaptation to an environmental loss of visual speech cues

              ….” This implies that adults adapt their speech perception faculty to an altered environmental availability of multimodal cues, and that younger adults do so more efficiently. This finding demonstrates that besides sensory impairment or signal noise, which reduce cue availability and thus affect audio-visual cue reliance, having experienced a change in environmental conditions can modulate the perceiver’s (otherwise relatively stable) general bias towards different modalities during speech communication.”
              Kateřina Chládková, 
              Václav Jonáš Podlipský, […]Šárka Šimáčková 
              Psychonomic Bulletin & Review (2021)

              Paul said “So in order to be coherent, our mental systems seems to ‘resolve’ uncertainty and simply present our conscious mind with a perception that comes with a sense of certainty”

              “Brain Decides, Then Tells You Later

              “That a great deal of our thought process isn’t conscious isn’t big news. Estimates vary as to the percentage of subconscious activity in our brains, but one common estimate is 95%. And it’s not just problem solving – the article notes that decision-making happens subconsciously before we are aware of it:…”

              “Toward A Bayesian Theory Of Willpower
              “P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.

              “Guyenet describes various brain regions making “bids” to the basal ganglia, using dopamine as the “currency” – whichever brain region makes the highest bid gets to determine the lamprey’s next action. “If there’s a predator nearby”, he writes “the flee-predator region will put in a very strong bid to the striatum”.

              “The economic metaphor here is cute, but the predictive coding community uses a different one: they describe it as representing the “confidence” or “level of evidence” for a specific calculation. So an alternate way to think about lampreys is that the flee-predator region is saying “I have VERY VERY strong evidence that fleeing a predator would be the best thing to do right now.” Other regions submit their own evidence for their preferred tasks, and the basal ganglia weighs the evidence using Bayes and flees the predator.

              “This ties the decision-making process into the rest of the brain. At the deepest level, the brain isn’t really an auction or an economy. But it is an inference engine, a machine for weighing evidence and coming to conclusions. Your perceptual systems are like this – they weigh different kinds of evidence to determine what you’re seeing or hearing. Your cognitive systems are like this, they weigh different kinds of evidence to discover what beliefs are true or false. Dopamine affects all these systems in predictable ways. My theory of willpower asserts that it affects decision-making in the same way – it’s representing the amount of evidence for a hypothesis.”…

              Hint for McGurk Mk2. Lots of other brain systems. Thanks as always.

              • conrad says:

                More pseudo-neuroscience. Here’s an obvious problem with linking things like dopamine (or more or less all neurotransmitters and hormones to complex behavior): The amount which people have varies across individuals a lot, and varies within individuals a lot across time. Most of the time this predicts nothing about any of the sorts of complex behaviors you might be interested in.

                The basic problem is that people impute a straight line from the extremes when no such line exists. For example, if I have some nice addictive substance, I will generate lots of dopamine. Alternatively, if I lose 80% of my dopamine I start getting Parkinsons.

                But this tells me more or less nothing about what this is doing to any complex behavior in between these points. If it did, all we would need to do to work out human behavior would be to know what levels of transmitters people had at any point of time.

  5. KT2 says:

    Conrad – as with Paul’s discovery of McGurk “More pseudo-neuroscience.”

    With some people as per your astute comments, another illusion allowing straight lines to look as though they are trending upwards. Some people manage to see beyond illusion, some need training, and some are flummixed.

    But most decide on lockdowns, masks, or the gt Paul thesis, without needing to invoke McGurk.

    Are you sayung Conrad, Paul’s panic is revealed or altered via just the McGurk effect?

    How many systems other than visuaal auditory are utlised in decisiin making?

    • conrad says:

      Paul didn’t mention anything to do with covid, so I took the post at face value. If Paul wants to say how pervasive some things to do with language and cognition are, that’s fine by me! Many are, and that’s much better than many areas of psychology.

      The McGurk illusion is good because it is so pervasive that it allows people to understand that what they expect (a sound process), is more complicated than they think. This is often very hard to get people to understand.

      The reading example, which I didn’t comment on, is actually something that one might take as bad-science given how it is sold in the popular media versus the reality (the actual stuff by Keith Rayner didn’t sell it like this). The interesting thing about it that no-one talks about is that it is actually a good example of how information theory can be used to quantify problems. This is because if you remove the _vowels_ you can still read it. But if you removed the same number of _consonants_ you couldn’t. Why is this? Well, consonants typically give you far more information about what a word might be than vowels (which you can quantify statistically if you want) and so changing them has more effect. So if you removed some level of information, you can still reconstruct what you want, but not if you remove too much (no surprise there!). Indeed, some lesser used scripts don’t even have vowels but people can read them (e.g., unpointed Hebrew), but I know of no scripts that only represent vowels. So it’s news but not news at all.

      In terms of how many systems are used in higher level decision making? All of them I suspect, just like most complicated problems. Obviously some will have different computational properties, and so working out how they integrate and if you can find more general effects is obviously worthwhile. In this respect, as far as I can tell, cognitive-neuroscience and psychology are ahead of the assumptions Paul complains about in behavioral economics, because the stuff is really looking at the underlying computational properties of systems rather than the assumption that they are somehow optimal (although there are some people that start off with bayesian frameworks) — for example, it would be a pretty fruitless pursuit trying to describe the effect of subcortical systems like that.

Leave a Reply

Your email address will not be published. Required fields are marked *