The limits to evidence based policy.

Evidence-based policy is a buzzword that conjures up images of responsible government: difficult decisions taken after a careful examination of the evidence, tailored local experiments, and then implemented using the best advice available. Sounds good, no? As a buzzword, it is a clear winner and something we all want more of.

But how much can we really expect of this buzzword and what will it actually lead to? Let’s have a look at the dangers and benefits.

The dangers and limitations

The first thing to note about the buzzword is that its main association is with rational control. In order to have evidence based policy, you must gather evidence and have a policy process. That means hiring more bureaucrats to do the gathering and the processing. Hence the buzzword is first and foremost a means for a bureaucracy to get their hands on more resources.

The buzzword also works to legitimise more power to a centralised bureaucracy: if you are going to have evidence-based policy, you must be able to implement the centrally-decided policy. Hence it is an excuse to take independent decision-making power away from local actors to the benefit of a supposedly rational centre. One might note the inherent contradiction here: with less power to local actors comes less local experimentation, meaning less policies are tried out and less possibilities for learning. Evidence-based policy is hence an engineer’s view of how an organisation learns (top-down). An alternative view could be that an organisation learns by following successful examples within the own organisations, which is a more organic but far less ‘evidence-based’ view. One might not know why something works and have no evidence that it will work in other places, but by simply following previous successful examples one can nevertheless reasonable hope to improve over time. That is how evolution works (what accidentally happens to fit its environment gets to procreate) and how many markets work: not by a rational gathering and analysing of information but by an almost blind mimicry of accidental success.

These initial thoughts mean that adopting the word ‘evidence-based’ policy implies increased bureaucracy, increased centralised control, and decreased local experimentation.

Then, let us consider the limits of ‘evidence’. New events that have no clear precedent but to which one must react are clearly outside the scope of evidence. What is one supposed to do then, wait until evidence is gathered? For instance, the recent Global Financial Crisis had no clear known precedence. It only vaguely resembled previous recessions. Should this then mean that all kinds of policies that were not implemented before and for which there was no evidence should not have been implemented? Should there have been no fee for the bank guarantee? Should we have stopped migration? Should we have started government work programmes (which were deemed so successful during the Great depression)? Or should we have done nothing for 50 years during which we gather more evidence?

It is clear that in the case of new major shocks hence, evidence-based policy has no meaning. We react to these based on vague rules-of-thumb and a vague understanding of how the whole economic system behaves. Such vague knowledge, based on vague historical evidence, can hardly be called evidence-based, but it is all we have to go on in such situations.

Then, onto the plus side:

The usefulness of evidence

‘Evidence’ to aid better policy is of most use in areas where government involvement is long-term, where the number of alternative policies is large, and where the issues concern complicated systems for which it is basically impossible to tell which policy will work in advance.

What areas are we then talking about? Primarily we are then talking about the delivery of major services like education, health, law enforcement, taxation, and social security. How to organise universities, schools, hospitals, medical insurance, tax payments, police forces, etc. are things for which there is a lot of time available. These services will still be here in 50 years time and it will be up to the government and its bureaucracy to determine how to organise its delivery. It is there that best-practise, discovered anywhere in the world, can in principle be identified, experimented with, and enforced.

The questions on which evidence can be generated include the question of whether we should have a voucher system for education. Should hospitals be paid on the basis of the diseases the patients comes into the hospital with or the number of actual procedures? Should we introduce an automated ‘default’ for tax returns? Should the internal affairs of a police force be connected to outside scrutiny (such as independent legal institutions) or should they be self-contained? Should teachers be paid according to the progress their students make or should they be paid a fixed rate in order to keep the peace within the teachers’ guild? Does harassment of people in welfare make them more likely to find jobs or make them more entrenched?

Questions like these are almost unknowable beforehand, but once they have been tried in a variety of settings, we can hope to learn which one works.

Do’s and dont’s in evidence-based policy.

A couple of obvious points come out of the above for how, in the best of worlds, ‘evidence-based’ policy can be institutionalised. One is that in order to learn about the evidence, major spending departments must have an analytical unit that retains the memory of previous ‘evidence’ and of ‘best-practise’. Such an analytical department must be somewhat separated from the political process, because governments react to the madness of the day and hence by design cannot really react to evidence most of the time. The knowledge of best-practise must thus be insulated against the political system and be allowed to come forward as independent advice every time a politician of a different colour asks for it. Partially, this knowledge is kept outside government entirely, such as when it is housed inside universities. For major spending areas though, the level of specialisation is so immense that it would seems a good idea to have independent research institutes inside particular departments.

A second point is that in order to learn from experiments, there have to be incentives to have open experiments. At the moment, nearly all the incentives of politicians and departments are towards secrecy, particularly for governments who have been in power for a while. This is because of the lop-sided nature of rewards in the competitive world of politics and senior civil service: any verifiable mistake you are associated with is used to crucify you by your opponents, whereas any ‘success’ has many fathers and usually belongs to the previous administration. Hence in order to create incentives, governments partially have to bind their hands via policies on openness (so that outsiders can criticise bad policy on the basis of internal data), would benefit from having scientific units inside departments who would have a vested interest in experiments, and should perhaps also bind their hands in terms of having regular open advise from outside scientific bodies, such as, in Australia, the Productivity Commission.

A third point is that evidence based policy would have to be kept out of those areas which are not long-term, clearly recurring, and complicated. In the areas of macro-economics and trade, for instance, it makes little sense to demand evidence-based policy and it instead is better to have individuals well-schooled in economics and in the history of macro-economics running around the policy makers.

Evidence based policy also needs to be kept from affecting the overall balance of power until there is a clear case for something, i.e. no jumping the gun. If you do jump the gun in those cases where that alters the balance of power, it will be almost impossible to reverse course because you will have created a political ally for the new situation. In the case of teaching for instance, one should first have experiments on teacher-pay and league-tables before setting up a huge measurement bureaucracy to enforce such policies nation-wide. Once in place, that bureaucracy will be very hard to get rid of.

On balance

Evidence-based policy is useful in a political culture that values reasoned input and where the balance of power between various actors (commonwealth, states, councils, business, etc.) is already clearly defined. In a political culture where the balance of power is not clearly defined however, it would be naive to expect the buzzword of ‘evidence-based policy’ to lead to better policy. More likely, it will be used to justify centralisation and the expansion of the civil service.

In the case of Australia, where the commonwealth is slowly gobbling up power from the states, evidence-based policy will probably mean centralisation. Real evidence will have little to do with it.

This entry was posted in Uncategorized. Bookmark the permalink.
Notify of
Newest Most Voted
Inline Feedbacks
View all comments
Nicholas Gruen
Nicholas Gruen(@nicholas-gruen)
11 years ago

Thanks Paul,

Another great conversation starter which doesn’t seem to have started a conversatoin :(

I only discovered the post today.

A few quibbles. I have never thought of evidence based policy as a grab for centralised power – I just thought of it as spin from our former PM who shortly after saying that he wanted all decisions evidence based handed out $25 mil to a rather startled Toyota to help build hyrbid cars which they were going to do anyway.

Though I can see how you can argue it. As I think you imply lower down, real evidence based policy is not necessarily centralised – it’s as much about learning and learning at the periphery as much as the centre. That’s certainly how Toyota goes about evidence based building of cars. And as you say it involves permission to experiment and it also involves the close measurement of performance. And in Toyota that’s not so much for external scrutiny, but to assist teams optimise their performance.

Paul Frijters
Paul Frijters
11 years ago

dont worry about that, Nick. There were some comments at Core econ on it, and it is going to be reprinted in the ‘Public Sector Informant’ in August. Besides, it is mainly a note-to-self.