Here’s a presentation to the annual Communities in Control conference run by the amazing outfit Our Community established in the 1990s by Denis Moriarty who had previously been a Deputy Secretary in the Victorian bureaucracy. (If you prefer audio, it’s here). The slides are here). It is as it were, the presentation of this lengthy essay on the PC’s Indigenous Evaluation Strategy.
The transcript is below the fold, this time produced by Our Community and so not the usual kind of machine produced transcript I’ve posted up here a few times. The transcript does not contain the Q&A session which was a good one I thought, so if you want to read the transcript and skip me taking you through it, if you click on this link you’ll be taken to the point in the video at which Q&A starts.
oooOOOooo
I was introduced as someone who had something to do with innovation. I don’t much like the term “innovation” – it means so many different things to so many different people. I want to talk today about escaping the iron law of “business as usual”, and I want to use a fairly simple idea – that is, the valley of death.
In private sector innovation in Australia, the valley of death is the gap between a technology being developed in a university and its proper commercial exploitation and growth. We do it terribly. Most countries, in fact, do it pretty badly. Even in the United States, most universities fail to make it across the valley of death.
That’s relevant to our community because I think there’s an even worse valley of death in social policy. The way it works is fairly simple. Policy discussion is on the basis of their being a market. We all know it’s not a perfect market, but there is a process going on in which the government provides money for social programs, there are lots of providers of social programs, and good social programs will be purchased more than bad social programs. That’s the pretty simple logic of it.
There can also be pilots of new approaches to things. Lots of NGOs will be doing exciting new innovations in social policy, and we all know we need that. It’s pretty obvious, though, that we’re not doing a very good job of social policy. and I distinguish between thin and thick areas of policy.
Australia’s been typically very good at thin policy. What I mean by thin policy is the kind of policy where you change a tax rate, or you change a welfare payment rate, and the whole system immediately scales, because we already have a system that organises tax or welfare around the country.
Thick policy problems, on the other hand, are problems like trying to protect neglected and abused (or suspected neglected and abused) children, Aboriginal welfare, mental health, just teaching kids well at school, and keeping them healthy. Those are thick policy problems, and we fancy we’re good at them. We’re not terrible at it, I guess, looking around at our peers in the world, but we could be so much better.
What it always gets down to is a mistake – a category mistake. I’m now going to explain to you how to play chess better than Magnus Carlsen.
1) You learn the rules.
2) You make moves that are better than Magnus Carlsen makes.
You might not think that’s a very funny joke, but in social policy that’s the sort of thing we do all the time.
When we’re dealing with a difficult area, the difficult question is “How do you do those things?” It’s knowhow. We default back all the time – because it’s more familiar territory, because it makes us feel more competent – to talking about knowledge, not knowhow. We don’t face up to the fact that we don’t know how.
And so we’re back at the rather cruel joke that people tell about economists.
A physicist, a chemist, and an economist are stranded on a desert island. And then a can of soup washes ashore.
But, famished as they are, our three professionals have no way to open the can. So they put their brains to the problem. The physicist says “We could drop it from the top of that tree over there until it breaks open.” And the chemist says “We could build a fire and sit the can in the flames until it bursts open.”
Those two squabble a bit, until the economist says “No, no, no. Come on, guys, you’d lose most of the soup. Let’s just assume a can opener.”
And that’s more or less what we do.
Here’s an example from the Productivity Commission’s most recent report on indigenous evaluation strategy. They think it’s an important – a core – contribution. They don’t look at how to do it, or look at who has done it well and how they’ve managed to do it well, and who’s been not doing it well but continuing to get paid to do it. They don’t look at any of those things with any great focus. In fact, they don’t know very much about evaluation.
What they did do, however, is to present us with a chart with some very pleasing words on it. They told us that the principles of evaluation, in this national strategy that they were releasing, was for evaluation to be credible, useful, transparent and ethical. And the question of how to do each of those things somehow fell into the background, as if the task was to say these words. There’s obviously a 500-page report to back these words up, but I still think it would be pretty easy to come up with four other words that were just as pleasing.
I hope that I can put that question into the mind of people in the audience, so that when you see a chart like that at your next corporate away day you’ll just ask yourself if something similar is happening.
So what matters, what’s hard, what’s fundamental in getting a system with knowhow – a system that knows how to deal with difficult problems, that has skills that work – is getting a healthy relationship between the arteries and the capillaries. Ant there we need to know about Lord Acton’s fault line.
A century or so ago Lord Acton joked that rowing is the perfect preparation for public life, because it enables you to go in one direction while facing in the other. Gough Whitlam quoted it, which is where I heard it.
I’ve told this joke for 20 years as a throwaway line, but in a recent essay on the Productivity Commission’s report into Aboriginal evaluation, I made a deliberate decision early on in the essay to bring it into centre stage. Because this is what’s driving the system crazy. It’s driving out of our deliberations the capacity to focus on what matters.
We’ve all seen that approach sent up on Yes Minister or Utopia, but what matters here is its subtlety and depth. We all know – and this is the plotline of many episodes of Utopia – that if you give a bureaucracy a task it will pretend to do it even if it makes no sense at all. A brave bureaucrat may say to their minister, “Minister, we’re not sure what this means. Minister, we don’t believe this is a good idea,” but once there is a policy there, the bureaucracy will not say “We don’t know what we’re doing.”
So here’s a quick illustration of Lord Acton’s fault line in action. It’s from an area which has relevance for the social sector but isn’t the social sector. It’s our habit – which we’ve now had for 34 years, ever since regulation impact statements were announced in 1986 by Bob Hawke’s government – of announcing every five or 10 years that there’s too much red tape and we’re really going to get on top of it this time.
That’s what’s possible. That’s what the system constantly defaults to. That’s what drives us towards a situation which we used to think characterised only the developing world – the cult of announceables. The problem that it’s nice to announce new things, and that’s what the system likes to do, but actually making the things that have previously been announced work (especially if they’ve been announced by your predecessor)? Well doesn’t go over nearly so well on the telly.
And here’s Lord Acton’s fault line at work. The New South Wales government announced that it was going to really have a big push for evidence-based policy. A few years later, the Auditor-General did a report on that push and said that it was “Largely ineffective. No information is provided on the performance of programs that have been evaluated.” And on we go.
A couple of years ago the Institute for Government in the United Kingdom produced a remarkable report – called All Change, if you want to look it up – which simply documents the way in which new eight and 10-year plans were announced. If you look down the bottom there at “industrial strategy”, they began being announced every four years, then every three years, and then every year. So it’s good for the announcements, but this is not a functioning system. And here in Australia, we do something pretty similar. Here’s the former Secretary of the Department of Prime Minister and Cabinet, for example, talking about the degree of churn in Aboriginal affairs.
The challenge here is that we have policy up the top and we have delivery down the bottom. Learning from the field, nurturing knowhow, protecting what is known by the system and having it survive all the way up to the top, and – who knows? – speaking truth to power, all that is very antithetical to a hierarchy where all the power is at the top. Adam Smith, taken to be the founder of free-market economics and a close personal friend of mine from the 18th century, made this remarkable statement: that “the disposition to admire and almost worship the rich and the powerful, though necessary to maintain the order of society, is at the same time the most universal cause of the corruption of our moral sentiments”.
You can take a line directly from that to what Peter Shergold said in 2005; everyone knows in the public service that the status goes to those who are in charge of policy – the people who do the nice charts – not those people who have learned how to do a difficult thing in the community, or even the people who have built a system that might be able to do it.
I often use an example that I became familiar with when I was chairing the Australian Centre for Social Innovation. We were asked to build an early intervention program to stop families falling into crisis. It was a very radical program, but, of course, it was just motherhood and apple pie. It arose out of lengthy discussions with the families themselves, and it worked by having families who self-identified as at risk of falling into some kind of crisis being matched with another local family, preferably one that had been through tough times themselves. Just to give you a sense of the texture of the program, it was the family we were seeking to help who chose their mentor family. A bit like a dating service, if you like.
I’m just going to guess now, but I suspect that there is no human services department in the country that actually actively matches caseworkers to families. A family will just get a caseworker. That’s how bureaucracies work. And our program wasn’t just a mentoring program. It was highly articulated, and trained family coaches took the families through the program.
I think the first roll-out of that program would have been in 2010, maybe 2011, at the latest 2012. It’s still running. It ran in New South Wales. It ran in South Australia. But there’s not much point in that. It should either have been expanded or it should have been closed down. Neither of those happened. To quote Peter Shergold now, eight years after saying that he would like this to change, here he is saying that, of course, it hasn’t changed, and that policy just cycles through these processes again and again.
So let’s get back to that social policy valley of death. I suddenly realised that it’s not really even a valley, it’s a chasm. It’s impossible to get across that valley in social policy, and that’s because it’s not really even a valley, it’s a Catch-22. So this is what we do. The government imagines itself as a shop. It’s got all this money. Then an NGO comes along and says, “We’ve got this fantastic program. It will lower recidivism. It will ensure that kids who have to be removed from their families are – they’ll be 50% better off than they would be in the way we’re treating them now.”
Often, the Department of Human Services says, “This is a nice idea. Thanks for telling us – but we don’t have any particular money for you at the moment.” If it gets to the Department of Finance or Treasury, then the Department will say, “Yeah, well everyone says that. You need an independent evaluation.”
So off you go and you get yourself an independent evaluation. That’ll cost you a few hundred thousand dollars. You’d go to Deloitte or PwC or, if you want a better report at a slightly lower price, you’ll come to Lateral Economics, and then you go back, and then they throw it in the bin, because, of course, it’s not an independent report. Everyone knows that Deloitte and PricewaterhouseCoopers make their money from providing the best compromise between absolutely independent rigour and what the client wants, and there is, in this area, plenty of room to give the client what they want. The Departments of Finance are right to throw those things in the bin, but what they don’t do is look at themselves in the mirror. No one is paying attention to the fact that we are essentially building an imaginary vehicle. This is not a market. It can’t function as a market. And no one’s really noticed that, and I think the sector needs to try to identify that and try and work out some kind of strategic response to it. “Insanity is doing the same thing over and over and expecting different results” – Einstein may or may not have said that, but it’s not a bad thing to say anyway.
I want to talk about what we might do, then, to escape “business as usual”, and I’m going to do it in two tranches. I think of this as a minimum viable product, if you’ll pardon another bit of jargon from the world of innovation. What’s the minimum necessary to escape Catch-22? “We want much more clarity about what it is you’re looking for in evaluation.” The sector needs to say to the system, “Where’s the colour of your money? If that evaluation identifies what works, will it lead to a change in current programs? And will these changed programs be compared, on some kind of level playing field, with what’s being done now?”
We need to remove conflicts of interest in commissioning those studies. That sounds like a scary thing for the sector, of course, because the sector will want to have an influence on those studies. But there you need to know about an economic principle called Gresham’s Law, named after a plutocrat and adviser to Elizabeth the First, and that law is that bad money drives out good. If you have gold coins in circulation along with lead coins, everyone will try to pass on the lead and keep the gold. If you can’t make those evaluations good money, you are wasting your money doing them. It has simply become another regulation — another cost. It isn’t achieving anything beyond that but you’re all on a level playing field with everybody else who’s doing those ‘independent’ evaluations.
The sort of thing that we’d need to do, the minimal thing that would be necessary, would be some government funding of evaluation. Now, that’s implicitly there anyway. If they fund people and then require an evaluation from them, that needs to be rearranged and unpacked, and then there needs to be a process of commissioning evaluations according to clear ideas about what you want out of those evaluations, with NGOs and service providers being consulted and having some role in how those evaluations – who they’re allocated to,* and how they’re done.
And then the question is, is it possible to use the system we have at the moment to place some pressure on that system? Because this system will not reform itself without quite a lot of pressure. I think that’s possible, and I think we need to talk about it and strategise it and so on, but here I’m simply putting up a sketch of the beginnings of the sort of thing that might be asked by the sector of the government.
I suggest a biennial Auditor-General stocktaking report. That would report, in a fairly standardised way, on the state of pilot programs, and on other exploratory innovation at the edge of the system, and would say how the best innovations are being identified, what’s being upscaled, and downscaled and why.
And here’s a big one – I should put it in flashing lights or in a different colour. Don’t imagine that the people at the top can suck the knowledge out of the bottom of the system, out of the edge of the system, and codify it and control it. If the system is to acquire knowhow, then we want to see the people who are doing it, the people who are showing how to get together all of those difficult things necessary to achieve something like this. We want to see them getting more agency in the system, not the people sitting there at the top. The top people can keep doing their job, which is to make sure this process happens. That’s hard. That’s way beyond the idea of a ‘nudge’ unit or a ‘what works’ centre.
I want to conclude by telling you about a meeting that took place in 1947, a meeting of people who thought the world was going to hell in a handbasket. They fancied themselves as liberals. Some of you will know of a person called Friedrich Hayek, who put the whole thing together. This was the beginning of neoliberalism. They actually called themselves neoliberals way back then. And they understood that, to get things done you need to have a strategic idea of what you’re doing which is a bit ambitious. If some bits aren’t accepted, you don’t just take them out, you just keep working.
You keep working on all of the important agendas you have. The power centres you’re dealing with they don’t go away, they just continue, but you start to build an intellectual consensus around critical issues, and that is how changes always happen. We’ve now seen the extraordinary amount of impact that that group had – a little of it for good, and most of it, it turns out in the end, not for good. We’ve had deregulation in all directions. In Australia, it’s been exactly the same formula for change. Here’s a quote from Professor Max Corden, in 1968: “The idea of lower tariffs in Australia is only of academic interest”. But by 1973, of course, it had started to become the consensus of informed opinion, and by 1988 it was all over.
The preconditions are the identification of the problem, intellectual work proposing a response, and a broad consensus behind that work. It has worked for tariffs, tax, the NDIS, Landcare, lots of things. Those are the good things. Not all have been good; I suppose there are bits and pieces that people mightn’t be happy with. And it’s worked for lots of straightforwardly bad things as well.
It also worked for greenhouse gases, until it got derailed, and I want to invite the sector to do the same thing for itself. I want to invite anyone who would like to in the audience to join me and some others who I’ve nominated to strategise our way out of “business of usual”. If you did want to do that, you certainly wouldn’t represent your organisation. You’d just be a person trying to think through the issues.
Thank you.
* I realise on reading I didn’t make it sufficiently clear in my presentation that the sector should be extensively consulted on the design of all these things, but it should have no specific influence on who is chosen to do an evaluation on any given project. A major point of evaluation is to produce information that is objective and so can function as a source of learning as well as comparison between projects. Accordingly those whose projects are being evaluated should not influence the choice of individual evaluations.
“If the system is to acquire knowhow, then we want to see the people who are doing it, the people who are showing how to get together all of those difficult things necessary to achieve something like this. We want to see them getting more agency in the system, not the people sitting there at the top. ”
This reminds me a bit of the Chinese political system of the 80s and 90s, the era in which it was working very well. What happened in that period is that the bosses in Beijing would announce to party leaders in the country what kind of thing they wanted to see. A ‘functioning health insurance system’, a ‘modern school system’, ‘an export strategy’. That sort of thing: what you call thick problems requiring a lot of innovation, experimentation and integration of systems.
What then happened was that ambitious party leaders at various levels (towns, provinces) would experiment with things, trying to assemble packages. They would normally fail. But now and then some well-oiled team would pull it off and introduce something useful.
The bosses in Bejing functioned like your Evaluator General. Indeed, that was pretty much the only thing they did: they promoted the successful local party bosses up the food chain and declared the systems they built ‘a new national initiative’ (or some such phrase). In stead of mandating other towns and provinces did the same thing, they then basically allowed provinces and towns to study and copy the supposedly successful example, or key bits thereof. So in a way the experimenting units were whole political systems of towns and provinces. A little bit how Western countries learn from each other, but then a lot faster.
Now, of course, different incentives are in place in China so it doesnt work like this anymore. But its a great system of institutional innovation when it works well. Very quick.
Thanks for that – do you have any sources on China I can follow up?
tough one. Most depictions of the system are either far more detailed or they look at a very different aspects like the formal structures.
https://fas.org/sgp/crs/row/R41007.pdf
makes the basic point I made above about local experimentation and national policies.
https://fas.org/sgp/crs/row/R41007.pdf
is not too bad as a starter on the political system, the issue of factions, meritocracy, and the promotion system. A lot has changed since then, but that’s not the issue here.
sorry, that first reference should be this one:
https://www.jstor.org/stable/20066378?seq=1