The Evaluator General

I recently sent a couple of emails explaining the Evaluator General and also did an extended interview explaining the ideas in the context of Matt Jones’ Public Policy class at Melbourne Uni. The first email below is the one I sent him proposing that we explain the Evaluator General in terms of the course of my own thinking in developing it.


Given the subtlety of the idea of the Evaluator General – most people think of it as one idea when it’s several – one way to understand it is to go through the way in which it was the product of my own history in thinking about certain problems.

Steven Jobs talked about how life is like joining the dots backwards to go forwards and the Evaluator General is a response to these points.

  • In 1983 working for John Button as Industry Minister encountering of the Toyota production system and its extraordinary radicalism – the kind of thing for which (for once) it’s not an exaggeration to describe as a paradigm shift. For some of the flavour of this check out the productivity chart below and this video by an American Toyota engineer.
  • Narrated back to myself through a few decades of thinking, I see this as standing for
    • The importance of building accountability from the ground up from the perspective of those who are being held accountable, not those holding them to account.
    • The gravitational force of the latter (wrong) way of doing it being almost of black hole magnitude – we are close to the event horizon. Warren Buffett has a term for it from his point of view which is “the institutional imperative”. He’s talking about the institutional imperative to grow – to aggrandise the business and its managers, rather than to husband capital to the advantage of its owner. In government the institutional imperatives are different – but they contain an institutional imperative common to business and government which is the institutional imperatives of bureaucracy. This is summarised in my little aphorism “if truth is the first casualty of war, candour is the first casualty of bureaucracies”.
    • The resulting tendency for systems of accountability to become systems of accountability theatre. In that regard, this essay is intended as a practical ‘prequel’ to the idea of the Evaluator General with this speech to the Australian Evaluation Society being the philosophical prequel though reading that one is only optional :)
    • Be that as it may, there are some miraculous cases where the institutional imperative has been avoided (as Warren Buffett has avoided it). They include
      • Open-source software;
      • The Toyota production system

Not coincidentally, in both, the profound, subtle and pervasive problem of truth-telling from the bottom to the top of the hierarchy appears to have been solved.

  • Arriving at The Australian Centre for Social Innovation (TACSI) in 2009 I discovered human-centred design and co-design, a powerful tool in seeking to deliver services from the perspective of the people you claim to be helping. But, despite wider claims being made for it, it is only that – a tool. It is not the system and the system is broadly correct in thinking of it as just one possible way to improve services. Neither human-centred design (or co-design) nor any other new tool contains within itself any clear recipe for the system to perform the tasks it must perform to
    • nurture good practice
    • learn from it
    • preserve and endlessly improve and optimise that learning
    • scale what works, change and if necessary abolish what doesn’t and progressively fit the relevant parts together as their roles and the division of labour between them change with learning how to improve them.
  • The Evaluator General is my attempt to build a system that might
    • Help innovations like TACSI’s be introduced;
    • Validate them against the evidence in such a way as to protect them from the institutional imperatives of managers further up the hierarchy;
    • Expand successes and improve or contract failures.
  • I did this by reference to modern political principles within the Westminster system which is to structurally separate doing and knowing. Thus Treasury is the line department responsible for advice and action to optimise growth and the ABS measures how well we’re doing in a way that’s independent of the Treasury – but nevertheless closely collaborative with it.
  • Another principle I’ve realised in retrospect is of great significance here I came across when thinking about political questions. That is the ancient Athenian term isegoria or equality of speech (or “ισηγορια” if you’re Plato, Aristotle or you’re just trying to be a smartarse)

Toyota was my first engagement with isegoria, but it rumbles on through my life – and is of great significance to public policy.

In summary, though people typically think of my Evaluator General as a top-down compliance type mechanism – using independence to browbeat the system towards addressing the objectives given it from the top, it’s actually two things and neither works very well without the other.

  • Independence
  • That independence is there not to perform ‘accountability theatre’[1. or to be generous to sceptics to accept the inevitability of accountability theatre but to try to change the terms on which it takes place.] by imposing it from the top, but to build an accountability system (as Toyota did) based on the self-accountability of those in the field. This is what science does. And as Richard Feynman says, “The first principle is that you must not fool yourself and you are the easiest person to fool”.[1. This is also Adam Smith’s idea when he talks of the impartial spectator as the foundation of morality (and implicitly of knowledge).]
    • In Toyota that’s workers on the line (and beyond them suppliers and customers).
    • In government programs it’s ‘street level bureaucrats’ (and the communities they service). So it’s teachers, their students and their communities, nurses and doctors, their patients and communities, prison warders etc.


The second email explaining the Evaluator General from scratch:

As usual, I learned much from your speech on economics and the third sector. The one thing among the measures I wanted to see in the speech was some commentary on our failure to properly build evidence-based policy and practice. The cost/benefit analysis you guys have done is obviously important and even this is missing in much of the interface between government and the charitable sector.

But I think there’s something much more fundamental which receives virtually no attention because I think the people who should be leading the debate – particularly policy economists – think they know what evidence-based policy and practice are, but in fact they do not.

The most dramatic way I can suggest the potential significance of what I’m proposing is by contrasting the labour productivity achieved with the usual top-down approach to evidence-based practice with a bottom-up evidence-based practice developed in business – by comparing US to Japanese automotive productivity over the 1970s and 80s.

My argument is as follows.

  • In the charitable sector and amongst many of the social services funded by government, we are not even at the level of top-down evidence-based practise, because, as you acknowledge in your speech, we make far less use of cost/benefit analysis than we should
  • Part of the rhetoric of contracting out and purchasing services from the charitable sector involves the idea of innovation – we tell ourselves that we’ll expand the most successful projects and strategies and scale back those that don’t work.
  • But mysteriously, we’ve been saying this for at least two decades and it’s remarkable how little of this actually takes place.
  • I think I have the beginnings of a quite powerful explanation for why that is – there’s a catch 22 at the heart of this learning system that those running it don’t really acknowledge even to themselves. I document that here.
  • Further, why are we so bad at learning from what works out in the field – with or without ‘what works centres’? We think of accountability as essentially a top-down activity – imposed by those above on those below.  (Or alternatively ‘accredited’ by our researchers using the tools of their trade – CBAs and RCTs and then propagated into practice by tools of ‘translation’ such as What Works Centres). But if this is really how it should be done, why are full blown CBAs and RCTs so rarely used in business (as opposed to much lighter weight experimentation and measurement with things like A/B testing)?
  • The system must build accountability to the facts and possibilities revealing themselves in the field. But that knowledge can’t travel upwards in the hierarchy while the system is engaging in accountability theatre and those above are holding those below ‘accountable’. How can they know what those below should be improving, what innovations will be most promising to try if they do not understand the conditions in the field and if those in the field may be penalised on the basis of information they pass up the line. In these circumstances, candour about what is and is not working is replaced by the white out of accountability theatre.A proper accountability system needs to:
    • be focused not just on measuring the system, but principally measuring it with a view to learning and improving it. [1. For instance, in my experience, the wellbeing agenda has often been measurement heavy but learning light – and this is true of the latest poster boy of New Zealand’s wellbeing budgeting. The measurement system they’ve built aims to be able to tell you Maori wellbeing in Rotorua but is not being built to ensure that those measurements help identify how to improve it].
    • learn from the field (or the bottom of the hierarchy) where most of the existing knowledge will be and most of the learning needs to take place,
    • have that learning objectively validated, so that ‘experts’ and the domain knowledge on which they draw remain accountable to the emerging evidence,
    • have that validated learning given appropriate weight against senior managers responding to institutional imperatives. For it is in this step that what I call ‘accountability theatre’ actively displaces true accountability for understanding what’s going on.
  • Believe it or not, this is what Toyota achieved in its development of a new way of managing car manufacture. It did so by spending literally ten times the industry standard amount on employee training, training shop floor workers to understand and manage the CNC (Computer numerical control) machine tools and then building the company’s accountability for its own productivity on the self-accountability of shop floor teams.
  • My own proposal for an Evaluator General tries to build the same system for the more complex, and ‘social’ world of delivering services to improve social wellbeing. It’s based on
    • Structurally separating doing things from knowing how well they’re performing. This occurs at the agency level within government where an agency like the statistical office will measure inflation and unemployment independently of politicians or the agencies whose performance will be measured by reference to those numbers.
    • Seeking to do this not just agency by agency, but in principle to anywhere where an agency’s work is.
    • Seeking to build close cooperation as well as structural independence between knowing and doing and from that
    • A system of evidence based professional knowledge and accountability built from self-accountability in the field with learning built from that. (As the great scientist Richard Feynman put it “the first rule of science is that you must not fool yourself, and you are the easiest person to fool”.


This entry was posted in Economics and public policy, Ethics, Isegoria. Bookmark the permalink.

15 Responses to The Evaluator General

  1. paul frijters says:

    Hi Nick,

    I think about very similar things now, so am interested in “getting” what you mean.

    Some bits I get, some I don’t. You are being very abstract here, which makes it tough going for those who think in a different way. Standard economists will thus totally miss the concept of motivation and, implicitly, power. You seem to say things about both, such as when you talk about self-accountability. You seem to say that public servants are more motivated when they are more autonomous rather than told top-down what to do. Is that what you mean?

    If it is, an unfortunate reality in the UK has been that top-down systems have increased massively the last 20 years, even in private organisations, so in that sense things are moving in an opposite direction. Same in Australia. Hierarchy has been on a roll. More institutional imperatives and more performances.


    I think I get the reality of accountability theater and the institutional imperative. I see a lot of it :-) And I know how a system could work where the workers are far better trained and responsible. In a broad sense what you sketch is how I think the Dutch public sector works already. This is difficult for you to refute as you have not lived there :-) Why don’t you though? Go live in an egalitarian place and see if it works how you imagine the Evaluator General system to work.

    However, there are bits of the writing above I just have no idea what you mean. I really don’t get the significance of your separation between knowing and doing. Don’t we know by doing? And doesn’t everyone change what they do because of what they know and observe? It just seems such a strange thing to propose to seperate, like dividing people into two. Heads to the left, hands to the right.

    I also really don’t get the Toyota reference. I have studied the Asian collectivist systems for years, which includes Japan, and they are not without institutional imperatives at all. Or without their own forms of theater. In many ways, they are worse than the Anglo-Saxons in terms of hierarchy. True, they do also have a community spirit which can be coopted for productive use, a bit like having teams operate as small villages, which is how I understand the Toyota system, but that is a very culturally-specific model that you cannot just advocate in other countries. The Americans could never work like that.

    So how much do you know about Japan? Have you lived and worked there?

  2. Nicholas Gruen says:

    Thanks Paul,

    A lot of questions and a lot to unpack. And then there’s the question of whether you need a ‘Nick Gruen translator’ or I need a Paul Frijters one!

    Perhaps we should Skype about it – I’m sure we should actually – but it will be useful for me to try to respond to your points. To start off with a statement which is (alas!) far more abstract than anything in the post, (in fact so abstract I may be having a lend of myself) you seem to be in a discourse of episteme (science) whereas I’m in one of phronesis (practical wisdom).

    That is, you are interrogating the material as if it’s a scientific theory about something with causation being the most important question. Of course, what I’m saying won’t be much use if it gets causation wrong, it’s not trying to be some discourse that somehow absolves itself from being about reality in some fit of rhetorical excess. But the way I’m speaking follows from the fact that, as I’ve argued previously, the professions are disciplines of doing first and their relation with reality should be refracted through that lens:

    The point of the natural sciences is to understand nature, and that might be driven by numerous motives – from curiosity to the desire to discover knowledge to exploit for private gain and/or social benefit. By contrast, the aim of what I’ll call the professional sciences is not some abstract notion of knowing more about some domain, but rather to serve a diverse range of human purposes in the field.

    And context is very important in that regard in ways I’ll go on to elaborate on.

    In your first substantial paragraph you say this:

    Standard economists will thus totally miss the concept of motivation and, implicitly, power. You seem to say things about both, such as when you talk about self-accountability. You seem to say that public servants are more motivated when they are more autonomous rather than told top-down what to do. Is that what you mean?

    There’s not much I can do about ‘standard economists’ missing points about motivation and power if they’ve so surrendered their commonsense and experience about life to their disciplinary tools. In the world intimated by their tools, there is no motivation other than self-seeking, and power has no motivational or informational significance beyond its distortion of prices. As Adam Smith understood, this is a pretty impoverished way to look at even most market behaviour but it’s a completely nugatory way to look at power in bureaucracies where there are no prices!). So all I can do is try to speak as commonsensically as possible to see if I can tempt them to use their understanding of everyday life to understand what I’m getting at. But I think you’ll agree, I can’t ‘translate’ it into their language. That would be like trying to teach a frog to fly an Airbus A380.

    You ask this “You seem to say that public servants are more motivated when they are more autonomous rather than told top-down what to do. Is that what you mean?”

    Close, but your way of putting it leaves out whole layers of meaning that are fundamental to understanding what I’m talking about. The way you’ve put it is very mechanical. If I were understood through the words you’ve used I could be understood to be saying that, by some mechanical and causal relation, performance increases wherever there’s more autonomy. But we know that that’s not a law. Greater autonomy is consistent with higher performance, but it’s also consistent with lower performance.

    A team in Toyota has much more autonomy than workers on the line used to be given. But it’s autonomy for something – autonomy to achieve a purpose. The art of management is to create a situation in which that autonomy can be granted and it will be used more effectively for the purpose that it’s been given than the people being ordered around by their superiors who (we assume) have the purpose.

    Toyota is, you will be unsurprised to learn, no free for all for workers. It is a highly articulated system. They have found a way to get their workers to ‘buy-in’ to the mission. But this is a highly structured process. Toyota factories are full of propaganda on the wall saying things like “people come to work every day to do a good day’s work, not a bad days work”. That’s addressed both to the workers and to the managers. But it’s not just rhetoric.

    It’s informationally rich. Workers receive literally ten times the amount of training that American workers did before these innovations. And that training is in self-accountability. They are the company’s eyes and ears on the line. They are the innovation engine on the line – it’s up to them to work out how they’re going, how they can improve – including by involving other teams in the system – including outside the factory amongst suppliers or at least in principle anywhere else in the whole system responsible for designing and building the car. Each substantial innovation isn’t really an innovation until it’s been systematised – built into the (management and information) systems – so that the team and the organisation can move on to the next improvement.

    This uses human beings’ desire to belong, to think well of themselves, to improve themselves as part of the engine of production. But it’s not naïve about it – it’s embodied in a highly articulated, responsive, information-rich system. And, as Toyota has proven at least in that context, one can build a firm-wide system of accountability on the self-accountability of teams on the line. And a system built on self-accountability is vastly superior to one built on top-down accountability for all manner of reasons both obvious and non-obvious that I explored here.

    Why does a worker buy-in? Why does the CEO buy-in? A rich mix of internal and external incentives. And no-one says it’s perfect buy-in (though these days I’d trust the workers to be more fully bona fides on that score than the CEO who can head off for his next gig any time.

    You write this:

    If it is, an unfortunate reality in the UK has been that top-down systems have increased massively the last 20 years, even in private organisations, so in that sense things are moving in an opposite direction. Same in Australia. Hierarchy has been on a roll. More institutional imperatives and more performances.

    I’m intrigued by your use of ‘hierarchy’ here. I agree with what you say, but I expect the system wouldn’t see it that way. I think it thinks of KPIs as just measurements, not as ‘hierarchical’ measurements, though of course, they are. Do you have any references or work you’ve done trying to demonstrate this claim that things have become more hierarchical? It would be tricky to measure not least because in most systems I know of, the number of levels in the hierarchy has reduced – often by quite a bit. But in the old system, people were left much more to muddle along on their own – they weren’t managed (well faux managed) as intensely as they are today. But then that was as much because things were more relaxed and there was less accountability theatre, not because their autonomy was particularly valued or their skills nurtured.

    I’ve paid close attention to your descriptions of Northern European countries as more ‘egalitarian’ than Anglo ones. I think that’s right, but I’m unhappy with the two poles of the dichotomy you’re painting. I don’t see it as hierarchical and non-hierarchical. The way I’d put it is using Michael Polanyi’s terms (which I’m currently writing up further) of monocentricity and polycentricity. Both Anglo and Northern European systems would think of themselves as devolving reasonable autonomy to people in the field (teachers in a school or nurses in a hospital). Both would say they want to ’empower’ their professionals and all that malarkey. But the Anglo ones would then cake their ‘professionals’ in a thick layer of paperwork and KPIs (just so everyone’s accountable ;) while the Northern Europeans would trust their professionals a lot more. They’d also probably require them to be more highly trained.

    The Northern European systems are thus polycentric (or more polycentric) because, in addition to having command from above, there’s the autonomy of and respect for the profession and the knowhow that the institution has is thought of as being delivered through those professionals (not the senior manager heroes) and the professionals’ knowhow and their capacity to learn so as to improve that knowhow.

    Tell me if you think I’ve got any of that wrong. Of course it would be great to live in the Netherlands or somewhere in Northern Europe and see this at closer hand. But I don’t think you can just say “they’re doing the Evaluator General”. As I wrote here, the Evaluator General model (and there may be other ways to deliver on the principles) is designed to nurture and protect professionalism in the countries where it’s been eclipsed by the institutional imperatives (accountability theatre imperatives) of senior managers and was never as strongly supported as it was in Northern Europe. But it’s also meant to strengthen those professions’ commitment to evidence-based policy.

    Of course, professions can hold themselves more and less accountable to their collective professional ideals (which include the truth – though not quite as passionately as scientists talk about the truth). But just as I think it’s just good hygiene for there to be structural separation between knowing and doing at the agency level (Treasury and its Treasurer seeking to deliver low unemployment and inflation as measured by a separate and independent agency) so I’m very confident that structural separation between knowing and doing would have something to offer the Northern Europeans. It would put anti-bodies in the system to make a slide towards Anglo managerialism harder, but I think it would also improve European professionalism. I can’t really see how it couldn’t.  It’s just the principles of science – as Richard Feynman says, “the first rule of science is that you must not fool yourself and you’re the easiest person to fool”. In this sense this aspiration of transparency through self-transparency is an institutionalization of Adam Smith’s idea of the impartial spectator.

    Now for responses to these comments/questions:

    Don’t we know by doing? And doesn’t everyone change what they do because of what they know and observe? It just seems such a strange thing to propose to seperate, like dividing people into two. Heads to the left, hands to the right.

    Does the end of the last paragraph help? In the bureaucracies I’m complaining about, they mostly go through the motions. Of course we must learn by doing, but that doing must be compared with objective information on how we’re doing.  And bureaucracies endlessly tell themselves they’re doing OK. They endlessly role play knowing what they’re doing when they don’t know what they’re doing. So yes, they’re ‘doing’ but they’re not learning, because learning by doing is learning from reality, not from fitting into a social structure that role plays knowing what it’s doing. You sense that there needs to be close collaboration between knowing and doing which I agree with. So that’s the challenge of the Evaluator General and I expect it not to live up to its potential where that collaboration can’t be delivered. But ABS and Treasury get on well and both understand the need to collaborate.
    I also really don’t get the Toyota reference. I have studied the Asian collectivist systems for years, which includes Japan, and they are not without institutional imperatives at all. Or without their own forms of theatre. In many ways, they are worse than the Anglo-Saxons in terms of hierarchy. True, they do also have a community spirit which can be coopted for productive use, a bit like having teams operate as small villages, which is how I understand the Toyota system, but that is a very culturally-specific model that you cannot just advocate in other countries. The Americans could never work like that. So how much do you know about Japan? Have you lived and worked there?
    You seem to me to be imposing your own, dare I say very Western notions on what I’m saying. They’re also very crude. I hardly think that Japanese firms don’t’ have institutional imperatives. They’re institutions after all! The Japanese are more hierarchical than the Anglos (“in some ways”) as you say. But then we need to be more specific. Japanese senior managers take much lower pay than Western ones. The Toyota guys don’t (or are very reluctant to) impose decisions on their underlings. It’s a very different system.

    “That is a very culturally-specific model that you cannot just advocate in other countries”. People have been saying this for decades. And there is oodles of evidence that it’s mostly not true. People said just what you have – “The Americans could never work like that”. And then they did. The NUMMI plant, run by Toyota for GM from 1984 or thereabouts to 2009 got its productivity way above the other American plants very quickly and it was consistently 50 percent above all other GM plants for many, many years without having more technology. I believe Toyota’s plants in the US are comparable in productivity and quality to their plants in Japan today, but I’ve not been able to locate good sources on that. Here’s what an old source from 1988 says:

    THE RESEARCH FINDINGS REPORTED in this article will help to overturn a common myth about the auto industry: that productivity and quality levels are determined by an assembly plant’s location. In reality theme/tins a wide range of performance levels among Japanese, North American and European plants. Corporate parentage and culture do appear to be correlated with plant performance; the level of technology does not. Plants operating with a “lean” production policy are able to manufacture a wide range of models, yet maintain high levels of quality and productivity.

    Here’s a chart from the study. (I couldn’t embed it sadly.)


    Now for that Skype call …

    • paul frijters says:

      yes, we should skype-chat about this, its interesting.

      Some reflections. I am coming more to the view that useful episteme is just another form of phronesis. That’s also how I now think about embodied cognition and that kind of jazz, ie good theory really just is a practical tool, simply for another type of problem.

      However, that’s not the main thing. Let me do two things here. One is give you the key reference on the hierarchy thing. The second is explain a bit more about the way the Northern Europeans think because it really goes to this whole buy-in, belonging, etc. stuff.

      On Hierarchy, I take my cue from the 2017 Skills and Employment Survey here in the UK. Its a recurring survey that has measured similar things for decades sampling British Workers. The key measure is “discretion as to how you do your job”. The proportion of workers who have a lot of discretion over how they do their jobs has declined to 38 per cent in 2017, down from 62 per cent in 1992.

      That really is an enormous drop: from 62% (the majority) to 38%.
      There is also other, corroborating, data, but this survey is the strongest evidence I have seen. Additional clues are that workplaces and organisations in general have gotten bigger in the UK (though there has been a recent growth in self-employment, which is often hard to categorise as it is often a hidden from of employment for a very large company, like uber). Income and wealth inequality (rewards) has also grown, which I see as one of the results of greater hierarchy.

      Then how the Northern Europeans really work, and I mean the Dutch, Danes, and Scandinavians, less so the Germans.

      Firstly, the notion of choice and bosses is very different. Many workplaces in those countries do not have bosses in the way Anglo-Saxon workplaces have bosses. If the CEO throws her weight around too much, she will just be ignored and quickly deposed by a coalition of other managers, boards, unions, worker-collectives, or, if necessary, by politicians. The power of those up higher is far lower and whilst they have more say over how the organisation is run, there is not that sense that they own and run the place which you have here in the UK. So it is not up to bosses to decide to be more “information rich”, “better trained”, “getting buy-in from workers”. Even the notion that that is a choice bosses can make is a strange one in those countries.

      Rather, one is taught from a very young age to slot in with others in cooperative teams, whereby one buys into the goal of that team. Later in life, one thus adopts various goals depending on the jobs that come round. If a Dane becomes a journalist, he becomes passionate about independent news and the ability of the whole journalist staff at his outfit to produce a good picture of what is going on in the world or some area of that world. If a Swede becomes an engineer working on a bridge project, she becomes really invested in the idea of a good bridge and will openly work towards that with her 20 colleagues doing various other aspects of that whole project, slotting in where she herself sees the best role for herself. She will also do lots of small jobs and initiatives that help her colleagues, often without those colleagues ever seeing that she helped them out in an unseen way.

      So these Northern Europeans really buy into the goal of their team and organisation. They dont just think they co-own that goal, but they even think they are the organisation. That’s why there is not such a thing as a real boss: like a village can have a major who does not own the village, an organisation in Northern Europe often would not suffer an Anglo-style boss because that kind of arrogance and presumption of being in charge would not be tolerated.

      This type of working together is somewhat similar to how you sketch Toyota to work. I am happy to be proven wrong about the Americans not being able to work that way, though I am surprised I must say, precisely because Toyota bosses make far less money than American bosses typically do. The pressure to become a “regular American firm” must be huge in those Toyota-plants, I imagine. How on earth have they kept that tendency out?

      Now, to be clear, Northern Europe is not quite as idyllic as sketched and there are more top-down organisations there too. They grew a lot in the 1990s and 2000s, though I think (but am not totally sure) the Anglo-model is receding a bit there. The whole labour market preparation and general culture is certainly still of the type I sketch: people are raised to be co-owners of the organisations they work in, not merely buying into its goals, but even co-owning the process via which goals come about.

      Within that kind of culture, use of data is much more integrated. One cannot so easily cheat because other co-workers would notice the cheating. The whole business of a conscience also goes towards this: one regulates oneself to a large degree.

      I think you would it fascinating to see it in action.

      • Nicholas Gruen says:

        Thanks Paul – it was good that I wrote it all down and that you have!

        Can you point me to a link to the survey and that result. It’s spectacular.

        In the idyl that you present of Northern European work, I think the Evaluator General (and I really need a new name for it because while it captures the independence, it sounds so pompous and top-down) would fit in really well. They’d be the person with a lot of expertise on how practitioners can mark themselves to market as it were – how they can move through problems patiently testing hypotheses rather than jumping to conclusions or putting excessive reliance on their discipline. This is addressed at this point of the video I linked to below.

        The Evaluator General would help everyone in the group and the group as a whole reach their highest aspirations of self-transparency – to understand what they were doing, what was working and why and what could be done better, and trying to produce a public record of that process. Of course, some people will read this as a bureaucratic nightmare, and it could certainly be implemented as such. So you need buy-in on both sides. And like we always say here “pretend is never good enough … except at Club Troppo”.

        • I remember seeing a graph with the changes in autonomy, and these figures I quote have been in various magazines. However, the whole thing is on a government website:

          I am trying to think of how this Evaluator General could work in the Netherlands. It certainly wouldnt work with that title, indeed. But some kind of training-providing institute plus collector of examples and methods might work. Maybe they already have those things in various sectors. I wouldnt be surprised.

          The covid-19 crisis is a good example of what the Dutch population does: the smart ones start to figure out how it all works and after a month have overtaken the medical “specialists” and then start to push those supposed specialists out of the public media.

          This the best-known of these quick thinkers who is now already recognised by many in the population as far ahead of those public sector medical specialists:

          (most is of course in Dutch, but this guy made the effort of putting some stuff into English. He runs a polling organisation, go figure).

          • Nicholas Gruen says:

            Thanks Paul,

            I’ve decided after some reflection that I definitely need a new name for the Evaluator General.

            But it’s a tricky question as choosing names always is. After all, words are a finite collection of little lego blocks of meaning and it’s not particularly surprising that sometimes they don’t perfectly line up with some new thought. Much more so with the small number of words used to describe public institutions.

            The name works very well for two reasons. It conveys the
            1) independence and
            2) status
            that’s necessary for the role.

            Unfortunately it doesn’t convey the bottom-up nature of the exercise – and indeed to the smart money sitting atop the system, suggests the opposite – another thing they can get their hands on and develop a new dimension of subservience to impose upon their underlings.

            But the problem is that it’s almost impossible to come at it from the other perspective because if I call the EG a ‘critical friend’, that conveys to most people that it’s all entirely consensual – that it’s all just another routine and a pretty poncy one – which means the people at the top won’t have any interest in it, and it will have no chance against the soup of BS that envelopes things as they rise through the hierarchy.

            You’ve done this yourself. Suggesting my idea simply become a “training-providing institute plus collector of examples and methods” completely misses the essential rigour of what’s proposed which is the separation of knowing from doing (with a view to their being reunited in close collaboration so that accountability is built on self-accountability with the focus always on improvement rather than have measurement sucked into the obsessive performativity of the system – its need for accountability theatre).

            You know Dutch society far better than me, but I refuse to believe that the Dutch are a super-race: That no-one inside a Dutch institution has ever had a conversation with a superior which began with the superior saying something like “yes, I know the results from this situation haven’t been as good as we’d have liked (or are better than the results of the minister’s favourite program), but I’m just concerned about the ‘messaging'”.

            I’m sure – on your say-so – that this isn’t in plague proportions in Dutch culture the way it is in Anglo culture, but the principle stands as Richard Feynman articulated it. It’s a fundamental tenet of science that one is curious and builds systems that are as immune as practicable from one’s will-to-believe’ certain things. And it would be embodied I am sure in the independence of Dutch statistical collections from Dutch line agencies which are judged by what those statistics say. It’s just basic hygiene.

            The only legitimate question I think is the practicality of the cooperation between people with different ultimate reporting requirements. I think that can be handled well in most cases. But we should try it and see where it can be made to work well (if it can be made to work well), where it works less well and why.

            • paul frijters says:

              sure, there have been cover-ups in the Netherlands, but usually about things the country is embarrassed about, not just a minister. Like civilian deaths in Iraq.

              However, the point I was making about the training is that that is how I think your proposal would be interpreted because many of the others aspects are already so normal there. Part of the education trajectory already.

              You want far more than just a habit of measuring and reflection, Nick. You are proposing a whole culture of how to work. The great advantage of being able to say “I want you all to adopt the Danish model of working” is that you can then point to many institutions and habits in Denmark, and you have 10 million people who can tell others about how it’s done. Even if what you are sketching is an idyllic notion of the Danes, that still gives you a natural place to point to for inspiration on how it could all fit together. The problem with calling it the Gruen Method is that you are then the only source of inspiration about how to do it.

              What you sketch does sound very egalitarian in situ science to me. The Scandi -science package? The Riks-model? The Viking system? Or just “in situ science”? Sounds poncy enough, no?

              • Nicholas Gruen says:

                Thanks Paul,

                I’m sure the Danes and those who need structural separation least can be very useful in trying to operationalise what I’m talking about. But I don’t need to call it anything as eccentric as the Gruen Method. It’s the Western Method. It’s the method that’s already there in our structural separation between knowing (including verifying) and doing in the governance of our structural institutions agency by agency. It’s there in the separation of functions in private sector corporations where, for instance, accounts is structurally separate from engineering or marketing, auditing is separate from accounting (and everything else) but is then replicated in ‘internal audit’ practices which, one imagines is more collaborative than external audit.

                While it’s certainly true that ideas about the architecture of organisations – the scaffolding on which one is trying to culture some new way of doing things – don’t get far without the culture (noun) that one is trying to culture (verb), it’s very antithetical to the way I think (and to the way we think in the West and within Western management) to simply point to culture and say ‘do that’. That’s way too new age for me. And I think it’s no way to actually learn from and then institutionalise the lessons from another culture. Anyway, I didn’t learn those lessons from Northern European culture, I learned them by paying close attention to what was in front of my eyes – which again offers a much firmer cultural foundation for passing it on to others in my culture than importing it from some exotic source.

                But I could still do with another name – though it will be very hard to find one that does what Evaluator General does which is to impute seriousness and authority.

                PS: if you want to reply, I’ll look for your reply at the margin – rather than following the margin to the right and off the screen ;)

  3. Nicholas Gruen says:

    Btw, I also highly recommend this video.

    I normally hate videos and if I do need to watch them, I listen to them while going to sleep. You can do that here, though the daggy charts say something.

    Anyway, this brought me back decades when I watched it. I don’t know if you’ll like it. You may think it’s pretty schlocky and sloganistic, but for me it just reeks an integrity that is very rare in such presentations. And we know Toyota have done something remarkable because their commercial performance and just their basic productivity and quality demonstrates it.

    When I first encountered the Toyota production system in 1983, the two highest quality built cars on the Australian market – those with the lowest errors when bought new were the Mercedes Benz top of the line 350SLE (I think it was called) and the Toyota Corolla. Mercedes achieved its performance by having more dedicated inspectors than any other line. Toyota did it by having none. That’s surely worthy of some curiosity. Anyway, economists were unphased. After all, is it anything more than a rightward shift of the supply curve – you know the way the abolition of slavery was.

  4. Daniel Work says:

    Hi Nicholas
    I just watched the whole video from this post, and I came away thinking the theory and proposed practices could benefit from what is going on in the DevOps\Agile software communities. Specifically the work of:
    Prof David Snowden (Cynefin), problem solving in complex adaptive systems (like your child protection scenario).
    Simon Wardley (Wardley Mapping), takes the value chain mapping from lean to the next level by integrating principals from evolutionary biology.

  5. Nicholas Gruen says:

    Thanks for this Daniel,

    I agree those tools are useful and commented on Dave Snowden’s Cynefin framework briefly here, but they’re tools. I think of what I’m proposing as institution building, as it would be if the Treasury was in charge of collecting statistics on which its own performance would be judged, and I proposed the structural separation we have now – into ABS and Treasury. In that context, the Evaluator General might well adopt such methods in its practice, but the tools they use are a separate issue from the institutional structure of the institutions using them but don’t substitute for, the institution building I am proposing.

    Don’t you think?

  6. Nicholas Gruen says:

    I post this particular successful exchange with someone initially critical of the idea of ‘self-accountability’. It is in the context of an internet discussion group discussing Michael Polanyi’s philosophy of science.

    Hi Nicholas,
    I watched your video on the topic of the Evaluator General – now I see what you mean. It is a bloody brilliant idea! So it involves, as you say, a ‘knowing’ system (involving people who engage in evaluation, statistics, causality factors etc.) and a ‘doing’ system involving those engaged in the practice of the social program itself. Different reporting requirements, as you say as well (also a sound plan). Legitimately I am baffled this is not currently operating across government.

    On Sun, 31 Jan 2021 at 01:52, Nicholas Gruen wrote:
    Thanks for your thoughts xxx,

    I know that Adam Smith is associated with free market economics, but I was citing Smith’s more basic ideas about how society works – best outlined in his earlier book The Theory of Moral Sentiments. If the discussion is not to be argument by word association, I think you really do need to respond to what I was referring to, not some other part of Smith’s ideas, or what you associate them with.

    On ‘self-accountability’, I agree that if this was to mean that individuals in government agencies or the agencies themselves just get to hold themselves accountable, that wouldn’t be any good. But then that can’t possibly be what I mean because that kind of accountability to oneself is just a parody of accountability. It would be a bit like saying that someone could be democratically elected Prime Minister even if he cast everyone’s votes for them.

    It would also be odd if I proposed an Evaluator General and then said that in fact it just involved everyone holding themselves accountable. So the ideas of self-accountability and accountability to an Evaluator General are subtly tied together and it’s necessary for you to try to understand what that relation is to understand what I’m saying – though to be fair to you, the essay was addressed to an audience that has been exposed to my writing on this for some years :)

    In any event the basic idea is a little like Polanyi’s regarding science (though for the avoidance of misunderstanding, it’s not exactly the same.) That is, ideally the scientist is, in the first instance, accountable to themselves and without this science becomes degraded because it cannot be effectively governed in a corporate manner. But the scientist’s self-accountability to scientific standards is in fact constantly checked against the scientist’s accountability to science – as other scientists challenge, test and debate their ideas and results.

    Though some of this was not spelled out in the particular essay you read. I’ve elaborated my idea for an Evaluator General for several years now and with supporting critiques of traditional notions of accountability – for instance elaborated in this essay). My central claims are:

    * That accountability to another degrades the degree to which the person accountable can learn (or be scientific) because the imposition of accountability from outside imposes intellectual constructs that will be at odds with “internal necessities” of learning from the field.
    *That traditional bureaucracy is full of this kind of problem and that where learning in the field is necessary to success, it imposes a huge cost – analogous to the cost that Polanyi would have thought Soviet methods of managing science would impose on true scientific productivity. For instance he has some quite amusing passages in Science, Faith and Society such as this one
    *If each scientist set to work every morning [to do] the best bit of safe charlatanry which would just help him into a good post, there would soon exist no effective standards by which such deception could be detected.

    There are plenty of bureaucrats doing that – trust me – and an increasing number of scientists given how much science is being managed in a similar way.

    That given the situation, and the political impossibility and general inadvisability of simply saying to individuals and government agencies they can do what they like, we may be able to develop mechanisms to build accountability throughout the organisation which are built in the first instance on the self-accountability of those in the field. This does not mean that individuals and teams have the last say. It means that they have the first say and that, so long as that first say retains its integrity against various parallel but independent endeavours (e.g. the Evaluator General), their self-accountability is what is also accepted as the mechanism by which they’re held accountable more widely. This is essentially how Toyota quadrupled labour productivity in car manufacturing. This lecture is probably the best elaboration of what I mean, though there are plenty of others if you are curious.


    From: on behalf of xxx
    Date: Sunday, 31 January 2021 at 12:21 am
    Subject: Re: An essay on Science, objectivity and public policy

    Hello Nicholas,

    I found this essay thought-provoking and enjoyable to read. For what it is worth, I will throw in my own thoughts and reflections on some of the themes and issues examined. Within the spirit of respectfully disagreeing with some of your points to ‘test your ideas’ (and develop my own understanding in the process), I present the following arguments and observations.

    A core argument you raise is that what is holding back social programs, or undermining the capacity of such programs to succeed, is due to the reality that, in your words ‘…bureaucracy is built on a parody of self-accountability which is accountability to others.’ Thus, for social programs to operate effectively, they should be run by bureaucrats who are accountable to themselves, similar to how scientists are accountable to themselves within their field. I disagree with this argument, as I will explore shortly.

    Whilst your quote from Adam Smith is an admirable sentiment, it did make me think about his most well-known (and implemented) economic ideals: the aim of allowing markets to operate with an ‘invisible hand’, adjusting themselves (not by any, or little, intervention by government). The history of modern economics has demonstrated this is a deeply flawed concept. I am adding this point in here because much of the spirit of social programs invariably relates to modern economics (the basis underlying the justification, implementation and evaluation of a programs success). In relation to this broad theme of economics and how this relates to the operation of social programs, I make the following series of arguments.

    The reason that markets do not function necessarily better by themselves, with little to no intervention from governments, is because generally speaking, everyday citizens do not actually know or understand what is in their best interests. This may appear obnoxious coming from me – but I will quickly add that I am not in the possession of the knowledge of what they should need or want. I am just stating people don’t actually know what they want for meeting their basic needs or requirements – or more specifically, they don’t know how those needs/requirements should be best achieved. If you were to ask people whether they want better employment prospects, improvements to public health, better education opportunities and so forth, they will nod their heads in agreement. But if you then stated that they need to pay more taxes to achieve these goals, they will generally shake their heads. There is a deep misunderstanding between what people want and how that is achieved (and the price they are willing to pay for it). Governments are all too aware of this social reality and they play into it (yes I am a bit cynical…). Therefore, governments will pursue various programs with the understanding that many of these initiatives may appear on the face of it to be achieving the aims of the ‘public good’. Some initiatives clearly are designed for the public good. Yet many are not, despite their alleged purpose. This is not a radical observation – many people would agree with this. What is important to consider in relation to this political reality is that this feature of politics is genuinely a core characteristic of politics. That is, the management of the perception that government policies are always aimed towards achieving the ‘public good’. The world is far more complex now than it was hundreds of years ago – therefore the citizen is required to trust elected officials to pursue the best possible policies for achieving the public good. Governments take advantage of this situation and are the principal beneficiaries of it. The irony to this situation is that if governments were made accountable to themselves (the invisible hand playing out here), this would potentially (or even inevitably) lead to a worse situation. I have not read your essay on the proposal for an Evaluator General (which, as you state, would operationalise the principle of making people in their field more accountable), yet I anticipate this may be going in the wrong direction. I do not understand how it could be a good idea to structurally separate knowing from doing. One does because they know. Polanyi’s concept of ‘indwelling’ comes to mind here: of the idea that as the individual ‘dwells’ in their practice, they come to embody the principles and standards of that field. Thus, the do because they know (the standards and principles of their field). To complicate matters, the policy-maker who is involved in the process of generating and implementing social programs, does so for the benefit (in theory) of the public. Therefore, the public (not the social-program-director-or-agency) must be the ones who hold up some level of accountability. Yet because the public generally do not know how such programs are meeting their needs, they are often not in the position to exercise accountability. And round and round we go. I do not have the recipe for success – but I have reservations for the deeply flawed system we are currently in. Yet in short, making government agencies more accountable to themselves, as a system, makes me raise an eyebrow in concern. The very basis of the Separation of Powers (the Executive, Judiciary and Legislature) came from the understanding that if power is concentrated within a single body/agency, they will inevitably abuse that power. That was perhaps the key lesson of Napoleon’s reign to the world in terms of governance – specifically of something to avoid in the future, by implementing the Separation of Powers.

    To my understanding, if things were to be improved with respect to the success of social programs (and the agencies running them), those involved within such programs would need to have more ‘skin in the game’. If the policies fail, then this would have more immediate results towards those running such initiatives. Relatedly, citizens would need to be better educated regarding the complexities of social policies (based on empirical studies, social scientific findings etc.) and be informed more honestly about such policies – perhaps by some kind of hybrid Evaluator General. I think the key issue around reforming the system has less to do with making agencies more accountable to themselves. The essential change lies within improving the understanding of the citizens regarding the basis of such initiatives: of what works, of what does not, and why that is the case. Raise the understanding and there will be a similar raising of expectations as to the viability and usefulness of social programs – based on knowledge derived (and clearly and honestly communicated) from experts in their respective fields.

    For anyone who is interested, feel free to add or subtract to these thoughts. Or disregard them entirely and respond to the original post by Nicholas.

  7. Nicholas Gruen says:

    Note to self: Sergey and Larry from Google made some interesting observations in their initial paper on search.

    See Appendix A: Advertising and Mixed Motivesabout the way in which advertising gets in the way of good search

  8. Nicholas Gruen says:

    Science is the prime example of what all freedom, contrary to popular conceptions of it, must contain; and that is discipline. The discipline of relating itself, whatever the cost – in human pride, vanity, or dignity – to reality

    Paraphrase of the philosopher John Macmurray’s views

Leave a Reply

Your email address will not be published. Required fields are marked *