John Quiggin’s objection to self-reported happiness data

subjective_well-being.jpg

In an earlier post I listed the main contentions of the happiness research program, and invited readers to contribute to the critique. The response was gratifying, and the student found them very helpful — in the comments here, in further posts by John Quiggin and Don Arthur, and in the comments on those posts (John even took the debate to Crooked Timber, which generated more discussion still). I didn’t follow the comments up because it was all too daunting, but John and Don have both returned to the topic, so they’ve spurred me to action.

Most of the respondents elaborated on the standard criticisms I listed rather than added to the list, so there weren’t many surprises, though I didn’t expect quite such hostility, from almost every point on the ideological spectrum. Ken Parish detected a plot to confiscate his car and computer and force him to attend line dancing classes, and most of the other comments also focused on the paternalist overtones of the project. I can understand this as a basis for healthy scepticism, but I was taken aback by the enthusiasm with which commenters clambered over each other (especially on Don’s thread) to dismiss the whole idea of enquiring into happiness and its causes as somehow shallow and nonsensical.

The one objection I hadn’t heard before was John Quiggin’s. Although I had put ‘Happiness can’t be measured’ first on the list of criticisms, and had also linked to an earlier post of his on the measurement issue, this particular challenge, as he reformulates it here, turned out to be one I hadn’t encountered:

Ive long argued that these questions cant really tell us anything, and an example given by Don Arthur gives me the chance to put it better than Ive done before, I hope.

Suppose you wanted to establish whether childrens height increased with age, but you couldnt measure height directly.

One way to respond to this problem would be to interview groups of children in different classes at school, and asked them the question Don suggests On a scale of 1 to 10, how tall are you?. My guess is that the data would look pretty much like reported data on the relationship between happiness and income.

That is, within the groups, youd find that kids who were old relative to their classmates tended to be report higher numbers than those who were young relative to their classmates (for the obvious reason that, on average, the older ones would in fact be taller than their classmates).

But, for all groups, I suspect youd find that the median response was something like 7. Even though average age is higher for higher classes, average reported height would not change (or not change much).

So youd reach the conclusion that height was a subjective construct depending on relative, rather than absolute, age. If you wanted, you could establish some sort of metaphorical link between being old relative to your classmates and being looked up to.

But in reality, height does increase with (absolute) age and the problem is with the scaling of the question. A question of this kind can only give relative answers.

I’m not convinced by this, and I don’t think John has thought it through.

Obviously subjective height data is inferior to objective height data, but that’s not the fundamental reason why the height-age hypothesis is rejected in John’s example. It’s actually because, in reporting their height, the children just happen to be adjusting for the very variable whose relationship with height we are investigating. Each child is using his class average as a yardstick, and of course children in a given class are all around the same age. One might just as plausibly suppose that each child will interpret the question to mean ‘are you tall for your age?’, because of course that’s what we usually mean when we comment that this or that child is tall. If they interpret the question that way, obviously no corellation between height and age will show up at all, not even in the class-level statistics. The only reason why there is some corellation at the class level in John’s example is that there is typically about eighteen months’ age variance in a given classroom.

But is there any danger that people will make an adjustment for income when you ask them how happy they are? For John’s analogy to work, it must be the case that if I’m asked to rate my happiness level on a ten-point scale, I say to myself: ‘Well, the question must be asking if I’m happy relative to other people on a similar income. Now, it’s true that my family is living on the street, that my children haven’t had a bite to eat for three days, and that I’m miserable compared to rich people, but compared to other destitute people I’m actually pretty happy, so I’ll report a score of 9.’

To avoid misunderstanding, this is not to say that people don’t compare themselves with a relevant reference group. Indeed, one of the findings of the happiness reseachers is that income and wealth are valued in large part because they confer status within one’s own peer group or socio-economic category (Don’s most recent post deals with this aspect). So, the person who lives in a smart three bedroom flat in Rooty Hill is happier than the person who has a modest five bedroom house in Dover Heights.

But to say ‘I’m happy because my income is high compared to others in my soci-economic category’ is not the same thing as saying ‘I’m happy compared to other people in my socioeconomic category.’ In the first case, you may indeed be happy relative to the population average. In the second, you are unhappy realtive to the society as a whole, but you are comparing yourself to a group of mostly unhappy people and finding that you are actually happier than they are.

If the ‘subjective’ nature of happiness data is a problem for happiness researchers, I think this applies more to its time-series claims than to its cross-sectional claims. The key finding, as John himself said (quoting me), is that, beyond a certain point, higher per capita income in a given country doesn’t seem to produce greater happiness. The standard explanation for this is that favourable or adverse changes raise or lower our happiness level only temporally: that is, we eventually become ‘habituated’ to an improvement or deterioration in our circumstances.

Habituation is normally taken to mean that, after some good or bad thing happens to a person, he reverts to the old level of happiness — the same point on the same scale. But what if it’s the whole scale that’s moving up or down — that what he previously called moderately happy is now very happy, and so on? This could happen unconsciously, in which case we might just as well speak of ‘resignation’ as of habituation; or it might be unconscious, that is, we might simply forget that what we now call happy is what we once regarded as only moderately happy. The unconscious case is a real problem for researchers because, even when the subject promises to be objective as possible, his reported increase or decrease in happiness may be illusory. A whole generation of respondents mistakenly report that they are no are happier than they were twenty years ago, and the reesracher walks away think he has evidence that happiness doesn’t rise permanently with income.

On reason why economists are wary in general about data on self-reported happiness is that our their teachers drummed into us that utility is incommensurable – that we’re not allowed to aggregate utility across individuals or make ‘interpersonal comparisons’.

But this is just an excuse. After all, we make interpersonal comparisons all the time, whether we’re calculating welfare loss triangles, conducting cost-benefit appraisals of public projects, or measuring GDP or aggregate consumption to gauge a nation’s performance. The real problem is that we are so accustomed to measuring welfare in money that substituting a less quantifiable variable like happiness seems like a retrograde step. How much money people spend, or would be Willing to spend, is not a bad guide if we know for sure that the spending translates into utility. But if we suspect that such spending is rash and myopic, or that it has external costs that can’t easily be quantified in money, then we must try harder to measure utility itself.

This is where the philosophical and political objections come in, and the accusations of paternalism. How dare you assume that people don’t know what’s best for them? That’s fine, but you can argue that measures of happiness are unnecessary or that they betray a authoritarian mentality, without necessarily arguing that they are meaningless.

Yes, self-reported data on psychological states are subjective and imprecise, but the same applies to data on pain. As far as I know, nobody is demanding that medical researchers quit using self-reported pain statistics on those grounds.

This entry was posted in Economics and public policy. Bookmark the permalink.

11 Responses to John Quiggin’s objection to self-reported happiness data

  1. Bannerman says:

    Refer my post of 8 June, wherein I addressed this question of ‘happiness’. It’s etheral, it’s individual and it cannot be adequately or accurately measured without a benchmark measure being defined. Said benchmark can’t be set because as individuals, we’re all entriely different.

    Oh, and that post wasn’t a rant. It was a thought exercise.

  2. Ken Parish says:

    Yes, it’s predominantly the assumption that it is possible in any meaningful sense to make a personal comparison of one’s own subjective happiness across time that I find inherently implausible. I simply have no idea whether I’m happier now than (say) 30 years ago, and I don’t believe most others are able to make a meaningful evaluation either. I’m sure I’m mostly less prone to the wild, hormonally-driven fluctuations from elation to despair that often characterise teenagers and young adults, although even that wasn’t true only 3 or 4 years ago when my marriage was breaking up.

    I’m no longer full of the optimism and sense of boundless possibility of youth, but then I’m also no longer driven to compete or prove my masculinity or even in many respects to worry very much what most people think of me. I have a more realistic sense of my own abilities, limitations, needs and desires, but conversely my abilities are less than they once were, at least physically. I will never play rugby for Australia or be a concert pianist or run a sub-two hour marathon.

    But can I tote up all these pluses and minuses and evaluate my overall happiness level comparative to 30 years ago? Of course I can’t, and I reckon anyone who claims otherwise is either lying or deluding themselves. Moreover, as the above discussion implicitly suggests, most of the factors that impact personal happiness arise from stages of one’s life cycle and have little to do with material wealth per se.

  3. John Quiggin says:

    “But to say Im happy because my income is high compared to others in my soci-economic category is not the same thing as saying Im happy compared to other people in my socioeconomic category.”

    Exactly. But how can you tell which of these two responses you’re getting?

  4. James Farrell says:

    I knew you’d say that. But isn’t the onus on you to show why we would have any a priori expectation that the second response is intended? As I said, when it comes to a child’s height, we know that tall means ‘tall for his age’. There is no such convention when it come to happiness. Of course, if you are asked by a friend, who knows the background, ‘how are you going’ and they reply ‘fine’, we might take this as a shorthand for ‘fine, all things considered’. But there is no reason why a researcher, in the context of a survey, would or should interpret the response as having an unstated qualification attached.

  5. John Quiggin says:

    As regards onus of proof, I think it’s on the person who wants to derive an inference. I don’t know whether people who answer happiness questions are comparing themselves now to themselves in the past, or to other people they know or to everyone in the world. I suppose you might ask them, but I’m not aware that this has been done.

    The only thing that is clear to me is that the answer is inherently relative to some set of people. Otherwise, as an answer to “how happy are you?”, “7” makes no more sense than “Belgium”.

  6. Geoff Honnor says:

    “I will never play rugby for Australia or be a concert pianist or run a sub-two hour marathon”

    There’s always line dancing, Ken.

  7. Just Me says:

    I have to make sense of peer-reviewed literature in medicine that includes (explicitly and implicitly) the question of ‘happiness’, in one form or another.

    Utter nightmare. So subjective and methodologically difficult that it is largely impossible to determine objectively to any useful degree.

    It is quite possible to change the self-perception and verbal behaviour of a large percentage of people so they report being ‘happy’, or at least at lot less unhappy, but without any concomitant statistically significant change in objective measures such as disease morbidity or mortality, medical resource consumption, work and income status, personal relationship status, etc. I have seen mainstream medical therapeutic studies that claim entirely on the basis of patient self-report to have cured serious physical symptoms, yet primary objective measures (such as blood parameters, activity levels, workforce participation rates, etc) either didn’t improve, or got worse. All it proves is that people’s self-perception and verbal behaviour is often highly manipulable and has no necessary immediate relationship to the reality they live (at least in the short term, in the long term reality catches up with all of us).

    Some things are just never going to be amenable to solid objective assessment, and I suspect ‘happiness’ is one of them.

  8. Ken Parish says:

    Geoff

    Yes, there’s always boot scootin’. And lawn bowls. Closely followed by Phil Nitschke’s peaceful pill … exit stage left happily singing Sinatra’s “My Way” very badly … regrets I’ve had a few … but then again too few to mention …

  9. Paul Frijters says:

    these arguments have all been mulled over dozens of times already, including on this site. Its a rearguard debate not worth going into again.
    The more interesting question is why the sceptics allow themselves to disregard the same problems in the measurement of other socio-economic factors. We for instance measure capital on the national level now, but in the 50s had debates raging about whether you could really just add buildings, coffee machines, bridges, and computers into one single number as if each component is pefectly interchangeable and as if the unvalued forms of capital (institutions etc.) dont count. Similar objections can be raised with respect to the concept of ‘labour utilisation’ where it is presumed that the people who dont work are just as productive as those that do (ludicrous). Even income is a concept one can only measure by pretending that all the access to services we get that are not captured in money wouldnt be worth something on the open market (such as household production). The list is endless and the only reason we dont talk about it any more is that the profession has moved on and simply accepted that we’ve got to work with the best we can measure rather than moan and complain like old women.
    One reason theorists like John in particular are so sceptical is that they’re used to simply presuming they themselves know what the utility function looks like. By measuring the thing one takes their toy away. Sorry John, but the days when you could shoose one utility function for this paper and another one for another without having to defend it on empirical grounds are hopefully coming to an end.

  10. James Farrell says:

    John

    I don’t know why you insist that people compare themselves with other people when quantifying the intensity of their psychic experiences. If I ask you how funny, embarrassing or scary some experience was, on a ten point scale, you don’t say to yourself ‘Let me think how funny a typical person would find that joke? I’ll call that 5. Unfortunately I have a more sophisticated sense of humour, so for me unfortunately it was only a 3’.

    It’s in the nature of subjective experiences that they can’t be verified or measured againts a standard yardstick. But it’s also in the nature of being human (except for autistics) that we try to quantify each other’s experiences, and to some degree succeed.

    The idea is that we have our own internal scales, and as long as the end points are meaningfully defined, one person can identify points along the scale for the benefit of others. I mentioned pain data at the end of the post. These also involve a ten-point scale, and typically the subject is told that zero means no pain and ten means the worst pain you can imagine. In the HILDA survey, the overall satisfaction question is: ‘All things considered, how satisfied are you with your life? Again, pick a number between 0 and 10 to indicate how satisfied you are.’ Respondents are shown a card indicating that 0 means totally dissatisfied and 10 means totally satisfied.

    Of course we can never know exactly how another person’s scale corresponds to ours — in the case of pain, some people may be able to imagine a worse upper limit of pain than others, and we can never be sure whether a given individual has in mind a linear or an exponential scale. But if you accept that people every day succeed in communicating the grades of their pain, sadness, embarrassment or whatever, then it’s not a huge leap to assign some numbers to the gradations. If someone rates their life satisfaction at 7, it’s pretty clear that this means: significantly closer to very happy than very unhappy, but still with a lot of room for improvement. That’s a lot more information than Belgium.

    Of course the data should be treated with care, and I’m sure it’s the case that answers are sensitive to all kinds of factors, like the context of the survey, the wording of the question, and the other questions in the survey. But this just means there is an art to conducting good surveys, and practicioners have a duty to implement best practice.

    Just Me

    The symptoms of many diseases, including cancers and muscular-skeletal disorders, can treated in ways that make the patient more comfortable without affecting mortality or morbidity, so I don’t see why this throws the subjective measurement into doubt. I’d be interested to know where pain fits into the equation. The studies you refer to would only be puzzling if the subjects report that their wellbeing has improved while also reporting unchanged pain levels.

  11. James Rice says:

    John Quiggin wrote: “As regards onus of proof, I think it’s on the person who wants to derive an inference. I don’t know whether people who answer happiness questions are comparing themselves now to themselves in the past, or to other people they know or to everyone in the world. I suppose you might ask them, but I’m not aware that this has been done.”

    On my admittedly cursory reading of this debate, the most controversial issue is whether, among rich people, increasing income leads or does not lead to increases in happiness. Longitudinal happiness studies (but not cross-sectional ones) suggest that increasing income has little effect on happiness among the rich.

    So, happiness studies support the conservative, cautious belief of no (or little) effect of income on happiness. It seems to me that those who question how happiness is measured in these studies do so in order to bolster their own inference that income actually does have a significant effect on happiness. It is those who question happiness studies that seek to support an inference, not their opponents. The onus of proof lies with the former.

    (Note that, despite the problems associated with the measurement of happiness, happiness studies do continually find significant relationships between happiness and more sociological, less economic variables, like marital status.)

Leave a Reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.