A video and an essay all on the same subject: measurement in the social sciences. Summary: It’s really worth doing and doing better, even though it’s really hard.
First, health statistician and visualisation expert Hans Rosling, co-founder of Gapminder and justly famous for his presentations on the past 200 years of human development. In the video below, Rosling explains how the world’s poorest countries are developing:
Second, Bill Gates, now well into his second career as a development entrepreneur, makes an eloquent argument for more innovations in measurement, and explains why it matters (or at least his PR staff do):
Given how tight budgets are around the world, governments are rightfully demanding effectiveness in the programs they pay for. To address these demands, we need better measurement tools to determine which approaches work and which do not …
… I think a lot of efforts fail because they don’t focus on the right measure or they don’t invest enough in doing it accurately …
… As 2015 approaches, the world is taking a hard look at how it is doing on the 1 goals. Although we won’t achieve them all, we’ve made amazing progress, and the goals have become a report card for how the world is performing against major problems affecting the poor. The MDG target of reducing extreme poverty by half has been reached ahead of the deadline, as has the goal of halving the proportion of people who lack access to safe drinking water. Living conditions for more than 200 million slum dwellers have also improved – double the target …
… It is the kind of good-news story that happens one life at a time and so it often doesn’t get the same visibility as a big setback like the outbreak of a new epidemic. From time to time we should step back and celebrate the achievements that come with having the right goals – combined with political will, generous aid, and innovation in tools and their delivery,
Both Rosling and Gates single out the example of Ethiopia, which they argue has made huge improvements in effective primary health care. Their message is that we can see various development initiatives succeeding in such countries.
I’m sceptical of their Ethiopian conclusions, because measurement is one thing and determining causation is another. Development experts have a history of claiming credit for improvements which turn out to have flowed simply from rising incomes (and which reverse when incomes stagnate or turn down). In many African countries, including Ethiopia, incomes have risen in recent years because of higher prices for resource exports. Ethiopia’s biggest export is coffee, and coffee prices have soared in the past decade. They’ve had a terms of trade boom since 2004, a phenomenon Australians should understand. (Update: As commenter Patrick notes below, they’ve also stopped fighting a war with Eritrea. Yep, that could help …)
Still, for whatever reasons, we’re seeing measurable improvement.
I’ve always been in awe of Tycho Brahe, who compiled detailed measurements of the movements of the planets, and Johannes Kepler, who analysed those measurements and made the mental leap necessary (from circular to elliptical orbits) necessary to create the laws of planetary motion. Brahe’s story breaks your heart: all that painstaking work, and he did not live to see where it led. But Brahe’s work mattered. We need great data sets in order to find truth, though creating them is a tough and often unrewarding pursuit.
It’s particularly tough to do decent measurement in the social sciences. I’d argue that far too much social science starts with poor data sets and plows vainly on through poor analysis to pointless conclusions. But good measurement in the social sciences is not impossible, and we should try to do more of it.
Footnote: It turns out Rosling’s latest video is embedded in Gates’ online essay, making it more likely that they are part of a calculated effort to create a powerful narrative around development. Make your own mental adjustments.
- Millenium Development[↩]
I’m a big fan of measurement. After all, a great many countries have had terms of trade booms and massive increases in GDP/GNI per capita without necessarily having a large improvement in early primary education.
But you are right that it is not causation, not so long ago Ethiopa was gripped by war and then famine which would have both ripped the guts out of early primary education. So maybe they are just no longer suffering from those afflictions!
I agree, there is a lot more that we can and should be doing in measurement. However, as the history of project/program evaluation shows, measurement is not free. If measurement does make it into a budget, it will be underfunded and it will probably end up being poorly implemented or then cut entirely. Lots of people want measurement, but surprisingly few are willing to pay for it.
In Australia, there is an obvious weakness in understanding of even basic principles of social science. It is sad how few people understand what data need to be collected and from whom and even less of those people can take an ideal design and adapt it to organisational realities. There are very few people with expertise in developing practical designs that will produce reliable and valid data over time within reasonable budgets. Those who do know how to do this kind of work are often met by woolly thinking from clients / decision makers who think it is all common sense and who sulk and shout when told that what they really need is not A but B+C+D+E+F and that will cost quite a lot of money (unless they are planning on conjuring it out of thin air, which some of our dodgier compatriots are quite happy to do for, oh, roughly half the price). Apparently, big data will save the day ;-)