Measurement in social science: hard, but worth it

Posted on February 16, 2013 by David Walker

A video and an essay all on the same subject: measurement in the social sciences. Summary: It’s really worth doing and doing better, even though it’s really hard.

First, health statistician and visualisation expert Hans Rosling, co-founder of Gapminder and justly famous for his presentations on the past 200 years of human development. In the video below, Rosling explains how the world’s poorest countries are developing:

Second, Bill Gates, now well into his second career as a development entrepreneur, makes an eloquent argument for more innovations in measurement, and explains why it matters (or at least his PR staff do):

Given how tight budgets are around the world, governments are rightfully demanding effectiveness in the programs they pay for. To address these demands, we need better measurement tools to determine which approaches work and which do not …

… I think a lot of efforts fail because they don’t focus on the right measure or they don’t invest enough in doing it accurately …

… As 2015 approaches, the world is taking a hard look at how it is doing on the [Millenium Development] goals. Although we won’t achieve them all, we’ve made amazing progress, and the goals have become a report card for how the world is performing against major problems affecting the poor. The MDG target of reducing extreme poverty by half has been reached ahead of the deadline, as has the goal of halving the proportion of people who lack access to safe drinking water. Living conditions for more than 200 million slum dwellers have also improved – double the target …

… It is the kind of good-news story that happens one life at a time and so it often doesn’t get the same visibility as a big setback like the outbreak of a new epidemic. From time to time we should step back and celebrate the achievements that come with having the right goals – combined with political will, generous aid, and innovation in tools and their delivery,

Both Rosling and Gates single out the example of Ethiopia, which they argue has made huge improvements in effective primary health care. Their message is that we can see various development initiatives succeeding in such countries.

I’m sceptical of their Ethiopian conclusions, because measurement is one thing and determining causation is another. Development experts have a history of claiming credit for improvements which turn out to have flowed simply from rising incomes (and which reverse when incomes stagnate or turn down). In many African countries, including Ethiopia, incomes have risen in recent years because of higher prices for resource exports. Ethiopia’s biggest export is coffee, and coffee prices have soared in the past decade. They’ve had a terms of trade boom since 2004, a phenomenon Australians should understand. (Update: As commenter Patrick notes below, they’ve also stopped fighting a war with Eritrea. Yep, that could help …)

Still, for whatever reasons, we’re seeing measurable improvement.

I’ve always been in awe of Tycho Brahe, who compiled detailed measurements of the movements of the planets, and Johannes Kepler, who analysed those measurements and made the mental leap necessary (from circular to elliptical orbits) necessary to create the laws of planetary motion. Brahe’s story breaks your heart: all that painstaking work, and he did not live to see where it led. But Brahe’s work mattered. We need great data sets in order to find truth, though creating them is a tough and often unrewarding pursuit.

It’s particularly tough to do decent measurement in the social sciences. I’d argue that far too much social science starts with poor data sets and plows vainly on through poor analysis to pointless conclusions. But good measurement in the social sciences is not impossible, and we should try to do more of it.

Footnote: It turns out Rosling’s latest video is embedded in Gates’ online essay, making it more likely that they are part of a calculated effort to create a powerful narrative around development. Make your own mental adjustments.

You can survive on Newstart but you can’t live on it

Posted on January 6, 2013 by Ken Parish

Troppo readers may have noticed a Christmas “silly season” debate about an ill-advised assertion by Minister for Families, Housing, Community Services and Indigenous Affairs Jenny Macklin to the effect that she could live on Newstart Allowance (aka “the dole”) if she had to.

The assertion was in response to a typical media “gotcha” question of the sort that inevitably reduces political debate to the sub-puerile level. In a strictly logical sense Macklin’s answer was clearly correct. As Samuel J points out at Catallaxy:

All of the 330,000 people receiving Newstart are able to survive on it, otherwise they would have moved to the local cemetery. There are an estimated 182,000 long-term Newstart recipients – surely they are surviving on Newstart, or are we now paying Newstart to dead people?

The point was to lure Macklin into providing a response that would enable her to be painted … as arrogant, insensitive and out of touch with “real” people and their concerns.

Given that Jenny Macklin is almost certainly more knowledgeable and resourceful than the average Newstart recipient, there really isn’t any doubt that she could re-arrange her life and priorities and budget to survive on Newstart if circumstances required it. But of course that wasn’t the point of the question. The point was to lure Macklin into providing a response that would enable her to be painted (no doubt spuriously) by the tabloid media as arrogant, insensitive and out of touch with “real” people and their concerns. Having regard to subsequent coverage, the glib journo’s gambit succeeded brilliantly.

However, although this is just the sort of meaningless “gotcha” journalism that is sadly now typical of Australian media including what were once the “quality” broadsheets, it actually raises some important issues: Is Newstart allowance fixed at an appropriate rate? Should it be much less than long-term pensions and benefits like Parenting Payment or the Age or Disability Pension (as it currently is)? By how much? Should parents be forced off the Parenting Payment when their youngest child reaches eight? Should there be exceptions to such a general rule?

Continue reading →

Informality as a mode of official communication

Posted on December 1, 2012 by Nicholas Gruen

Get a load of the UK Cabinet Office Minister’s delivery.

It’s fabulously low key, informal, indeed intimate compared with the formal bullshitting mode of almost all political utterance, and straightforward. It is of course ‘spin’, as it couldn’t be anything else. The Gettysburg Address was spin. But what I find thoroughly delightful about it is the way in which it simply dispenses with the entire genre of the public statement.

Of course there’s a reason for the genre of the public statement, because a public statement is not an intimate statement to a single person. However it is now so thoroughly debased by Orwellian corporatespeak “The Government is committed to a fairer Britain for all Britons”, that it’s a breath of fresh air to start again.

Reminds me of this issue I drew attention to in a previous post:

One of the things that intrigues me about the world is that acting is never ‘realistic’. For instance whenever you listen to a documentary and some scene is ‘reconstructed by actors’, you can always tell that they’re actors. They say their lines like they’re in a play or a movie, yet they’re acting real life. Strange isn’t it? They’re professionals at feigning life, and yet, when their only job is to feign life, not to ‘put on a play’ which is understandably a kind of hyper-real-life, they can’t do it. I’d like to understand why this is so. I’m sure it’s not a reflection on actors that their acting is not fully ‘realistic’, just as a TV presenters speech to camera is not like they speak normally, and just as when we give a speech to a group it’s not the same voice we use to speak to each other. Still I think it is a very telling reflection on actors that they show little sign of doing something completely realistic on the rare occasions when it’s called for.

Thoughts on “Thinking, fast and slow”

Posted on August 23, 2012 by Paul Frijters

I couldn’t resist buying a copy of Daniel Kahneman’s best-seller when returning from holidays. Several friends and colleagues told me it was a great book; it got great reviews; and Kahneman’s journal articles are invariably a good read, so I was curious.

Its general message is simple and intuitively appealing: Kahneman argues that people use two distinct systems to make decisions, a fast one and a slow one. System 1, the fast one, is intuitive and essentially consists of heuristics, such as when we without much thought finish the nursery rhyme ‘Mary had a little…’. The answer ‘lamb’ is what occurs to us from our associative memory. The heuristic to follow that impulse gives the right answer in most cases but can be lead astray by phrases like ‘Ork, ork, ork, soup is eaten with a …’. Less innocuous examples of these heuristics and how they can lead to sub-optimal outcomes are to distrust the unfamiliar, to remember mainly the most intense and the last aspect of an experience (the ‘peak-end rule’), to value something more after possessing it than before possessing it (the ‘endowment effect’) and to judge the probability of an event by how easily examples can come to mind.

System 2, the slow way to make decisions, is more deliberative and involves an individual understanding a situation, involving many different experiences and outside data. System 2 is what many economists would call ‘rational’ whilst System 1 is ‘not so rational’, though Kahneman wants his cake and eat it by saying that System 1 challenges the universality of the rational economic agent model whilst nevertheless not wanting to say that the rational model is wrong. ‘Sort of wrong sometimes’ seems to be his final verdict.

Let me below explore two issues that I have not seen in the reviews of this book. The first is on whether or not his main dichotomy is going to be taken up by economics or social science in the longer-run. The second, related point, is where I think this kind of ‘rationality or not’ debate is leading to. Both issues involve a more careful look at whether the distinction between System 1 and 2 really is all that valid and thus the question of what Kahneman ultimately has achieved, which in turn will center on the usefulness of the rational economic man paradigm.

Continue reading →