Why ‘how to’ guides on innovation are of limited use: An ‘untheory’ of innovation

‘Nor is wisdom only concerned with universals: to be wise, one must also be familiar with the particular, since wisdom has to do with action, and the sphere of action is constituted by particulars’.


Looking for a graphic for this blogpost, it was amazing how many of the pictures were of besuited men looking anything but innovative. Not a woman or unbesuited person for miles around. But I digress …

Thinking about how to write a fairly substantial review of knowledge and innovation in the urban water industry, I listed all the things that need to go well for innovation to thrive. What began as a kind of memo to self turned into a kind of unmanifesto, which is to say an explanation for why theorising about innovation can’t be taken very far (or, to be more circumspect, perhaps it can, but I’ve never found it very useful.) In fact there’s lots of writing about innovation – far too much – but most of it – including the best of it – is incredibly light on theory and is in fact storytelling. This is a complement to it, not a criticism given the lack of usefulness of theory.

The thing is, for innovation to thrive, so many things have to work well together that that is the secret of innovation. And this challenge of ‘alignment’ – of purposes, of people of populations (I meant systems, but it didn’t start with ‘P’ and I’m trying to learn to be more arbitrarily and automatically alliterative) – will be particular to particular projects. There’s little of a general nature that can be said about them. Anyway, what began as a memo to self is now an important part of the way I think about this stuff. A theory (or untheory) of the non-theorisable. The things that must be finessed for innovation – doing things in new and better ways – to thrive in an industry or wider system are many and varied. Just listing them gives an indication of the difficulty of the task owing to its complexity and many faceted nature.

From creation to implementation
  • Innovation must be initiated either by those with a problem to solve or an opportunity to seize or elsewhere within the system. Where this is not spontaneous, it may require careful cultivation and the application of financial and other resources.
  • Once innovations are generated, they must become known to those who can use them beneficially including in some situations, those who are unaware of any specific problems, but who nevertheless can use such innovations to advantage.
  • Those who become aware of new knowledge must understand how to implement it to advantage.
Governance of innovation
  • A specific innovation may solve problems for some levels of a system but it may involve new routines, higher costs, inconvenience or disruption elsewhere. There will often (generally?) be no well-accepted means of determining priorities between the various considerations arising. So decisions must be made.  
  • These kinds of trade-offs will often be required not just within an organisation but between different organisations. Thus for instance if the system one is considering is the urban water system, the issues to be handled might involve the industry supplying it with plant and equipment, or even entirely different systems such as the health system.
  • The outcomes of innovation are often uncertain and assessments must be made as to whether the risk of bearing unexpected costs should be taken, whether internal or external to the organisation.
  • For innovation to thrive, incentives must be conducive to it. They can include monetary rewards but will often involve richer cultural cues and rewards. Yet incentives – for instance bonuses – can obstruct innovation, for instance, by undermining cooperation. Patents can reward innovators, but will often frustrate the spread of ideas and can involve large legal costs and inflexibilities. Career advancement within organisations can reward innovators, but can also exacerbate organisational ‘groupthink’.
  • The more new knowledge depends on abstract principles or science or technology, the more difficult it is for those at the ‘coalface’ to govern its creation. Not only may those at the coalface and those with expertise not understand each other’s’ perspectives well, but they will often inhabit different institutional worlds with different incentives which will not necessarily align. Researchers will often be motivated by furthering their careers and the incentives for doing so will often not align well with the development of their research for practical, industrial application. Often the different domains don’t just communicate with each other well, they harbour cultures that are actively hostile to one another. Thus for instance, economists’ perspectives often come with a fully articulated interpretation of how those who disagree with them are fools or knaves, soft headed and/or self-interested.
  • Where a system is in search of new approaches, local experimentation can contain the cost of failure whilst retaining the option to generalise success to other parts of the system. This, and the facilitation of different approaches, is one justification for more decentralised action, though this consideration should be balanced against the possible costs of duplication and fragmentation of expertise and effort.
  • Innovation is often nothing more nor less than the joint resolution of different perspectives. This puts collaboration at the heart of much innovation. Yet the melding of different perspectives and modes of operating is often difficult and its difficulty rises as the number of collaborators grows, increasing the scope for free riding and intensifying other obstacles to effective coordination.
  • Skills are important for innovative capacity, yet developing and retaining them is not straightforward. It is a difficult activity to govern within organisations, and also involves coordination between businesses and outside institutions such as training institutions, unions, accreditation and occupational regulation. There are also potential ‘free rider’ problems as firms invest in employee skills but employees can choose to take those skills elsewhere.

Disseminating and building on innovation 

  • Once knowledge has been created, it is ‘non-rival’, potentially freely available to use again and again.
  • Because new knowledge accumulates and improves existing knowledge, access to existing knowledge and the ability to build on it is a growing resource.
  • Though knowledge is a potential public good, those in a position to fund such public good – such as governments – may be poorly placed to generate the new knowledge for numerous reasons including the lack of alignment between the political incentives governments face and the technical demands of the innovation, and governments’ distance from the ‘coalface’ of the industry’s problems.
  • The better information flows throughout the industry, the better those in it will be placed to determine their innovation priorities. In a well-functioning market, prices help those at every level in the system to prioritise actions. Where there are impediments to well-functioning markets, good information systems may nevertheless foster benchmark competition, enabling those within and without the industry to more effectively learn from and apply others’ experience.

Finessing any of these points can be challenging, some of them profoundly so. Given that innovation will typically run the gamut of many or all of them, it is no surprise that innovating well is not easy, nor that there is no one accepted method to make innovation thrive. There are numerous takeouts from this – a few of which are sketched beneath the following headings.

On the merits

Probably most important is the idea that decisions must be made on the merits. As Kahneman has recently put it, “organisations … are factories for making decisions”. And decisions are hard and subject to any number of pathologies within an organisation. Of course this sounds so commonplace that it seems hardly worth saying. Don’t management structures exist to ensure that decisions within an organisation are made on the merits? Well no. Organisations might say they want to make decisions on the merits. They might repeat it, or words to that effect endlessly in meetings, on corporate collateral and at strategy retreats. But they also tell their employees that they want them to do what their bosses tell them. So what do you do if your boss makes a decision you think is wrong? What if you know it’s wrong. What if he’s just a jerk?

The fact is, there are all sorts of pathologies that prevent decisions being taken on their merits. Making decisions on the merits is actually extremely hard – both in one’s life and in organisations – and there are any number of pathologies alive in our own psychologies and even more so in the psychodynamics of organisations that prevent decision making on the merits.

There are hierarchies, there’s groupthink, there’s second guessing hierarchies, there’s trying to keep people happy and consensus at any cost, there’s ‘not invented here’, there’s ‘not in my backyard’ there’s excessive risk aversion (though it’s usually aversion to a certain kind of risk dictated by another pathology – process hugging) and other manifestations of status quo bias, there’s adulation of the wealthy and powerful over the less so and on and on. If such tendencies are even fairly mild, then in a decision involving numerous parts of an organisation or aspects of its work as outlined in the points above, there are any number of ways for a decision going off the rails and in fact not being made on the merits.

This helps explain why flattening management structures has been an important objective of firms seeking to make themselves more innovative – it makes informal contact and problem solving between different parts of and people in an organisation easier, and can minimise the possibility of such mutual adjustment falling victim to the formal rules and pathologies of hierarchical decision making and decision making between silos in organisations.    

The ecology of business innovation: Competition, collaboration and the knowledge commons

One hallmark of advances in innovation is some new way of finessing the many tasks/principles outlined above. This occurred for instance with the advent of the ‘Toyota Production System’ and in the emergent innovation ecology of ‘Web 2.0’. Breakthroughs to new, more innovative business ecologies typically involve some new configuration of the mix between centralisation and decentralisation and competition and collaboration within an industry or business ecology. Innovation requires some felicitous coming together of the many tasks/principles set out above. People and institutions must be empowered to do new things.

They require incentives to innovate which are likely to direct resources and esteem in their direction. But at the same time, so long as it takes place, there is then a premium on spreading that innovation wherever it can do good. Thus, for instance, the innovation ecology that was incubated within Toyota’s production system from the 1950s to the 1970s involved not just the provision of incentives to suppliers and employees to innovate – where those incentives had been suppressed by adversarial relations previously – but also the encouragement of a knowledge commons amongst all of Toyota’s suppliers to meet their collective needs in sharing any and all innovation and knowhow they could.

High roads and low roads: intrinsic and extrinsic motivation

Observing the list of preconditions for innovation to thrive above, many of those preconditions involve successful new connections being made between ideas, systems, people, parts of organisations, organisations themselves and whole systems. There are important consequences from the fact that these connections are new.

Firstly, the desirability of new connections being made and the difficulty of predicting what they will be before the event places a premium on openness. Secondly, once connections are made, new arrangements may be necessary to facilitate the innovation. Negotiation between the different parties involved may be required to fund the innovation (where it may be necessary to negotiate some share of the benefits to different parties) or to accommodate the innovation, which generates value in some part of a system but imposes costs on other parts of the system.

Thus, for instance, new sensors on infrastructure might reduce maintenance and capital costs but may involve additional costs elsewhere in the system, for instance, in re-skilling operational staff and informing customers. These decisions typically need to be made on their merits with mutual accommodation between all concerned. Because the innovation is – by definition – new, there is a sense in which such arrangements will have to be improvised, at least initially.

In all these areas, trust and high standards of conduct – high morale between the parties – is at a premium because lack of trust or suspicion will obstruct progress in making the relevant connections and conceiving and negotiating whatever new arrangements are necessary to realise the innovation. Managerialist wheezes – performance management down the hierarchy, bonuses and so on – may be of assistance, but they can often get in the way because of difficulties of attribution – who was really responsible for the innovation when the whole system had to accede to it? Those problems of attribution may create more friction than they’re worth.

And in all this, intrinsic motivation is at a premium. It’s a secret weapon! I say secret because economists know very little about it – economists formal and quantitative tools can be of some use here but they’re unlikely to be of the essence. Likewise, the science of managing for intrinsic motivation is either at an early stage, or perhaps more likely, it’s not much of a science – it’s an art, a craft etc. In all this though intrinsic motivation is just as much a potential engine of action as extrinsic motivation – the better understood kind of self-interest around which so much economics is built. Intrinsic motivation however is harder to understand from the outside – which is hardly surprising what with it being intrinsic and all!

This entry was posted in Economics and public policy, Innovation. Bookmark the permalink.
Notify of

Newest Most Voted
Inline Feedbacks
View all comments
Larry Langman
Larry Langman
9 years ago

Dear Nicholas
Why I am on the site today – I listened to your contribution to RN’s Ockham’s Razor this morning. I must admit I needed to go to the transcript to get a sense of what your were on about……and I am wading through that on another screen, and thought to check on this site to see if you had made comment on that contribution.

Instead I have found this discussion….I’m interested in the subject and will keep on reading

However I’m at the first para or two and I note that you had problem with the word “population” when what you wanted to refer to was system. I think the word “process” might have been appropriately used here……and it has a certain cadence……”of people of process”…………

Back to the article

Larry Langman
Larry Langman
9 years ago

Dear Nicholas

So now I have finished the article…..and can get back to the Ockham’s Razor piece.

I liked this piece though I thought that where ever the word ‘knowledge” was used that it should have been written with a capital “K”. Knowledge in this article’s scheme of things seems to have a particularly enabling quality to it. Knowledge with a capital “K” would be that knowledge that pertains to the domain of the system in which the focus of the innovation sought exists. Knowledge with a lower case “k” would be that more general sense of knowledge that might come in from “left field”. Identifying that knowledge is whole other discussion when it comes to the question of innovation.

Now being someone interested in Knowledge Management for about thirty years I felt a little more might have been said about the notion of a “knowledge commons” and its contribution to innovation. Might I refer readers of this particular article of your’s to the excellent discussion by Hess and Ostrom
( http://mitpress.mit.edu/sites/default/files/titles/content/9780262083577_sch_0001.pdf )

Now back to that Ockham Razor piece……I want to look at the information density of that piece….it came across as particularly dense and difficult to follow so I’ve got to look at the text…….thank goodness RN still does, selectively, transcripts !!

Matt B
Matt B
9 years ago

In skimming through various submissions to a current Senate Inquiry on Australia’s Innovation System – there was little that came close to this level of sophisticated abstract analysis – most were institutional and corporate rent seeking type submissions.

You should at the very least send a link of this through to the Committee.

I’d trust you to lead the design of the system ahead of any others.


[…] previously written about what I call irreducibility at least three times before. As I wrote here, “What I’m arguing for is a recognition of […]


[…] readers may recall an earlier post which I christened an ‘untheory’ of innovation. It argued that there’s not much use in ‘theories’ of innovation if they’re […]


[…] clearinghouse of successful and less successful experience in an attempt to seed a national knowledge commons on these things could be helpful. Governments could also negotiate with the providers of software […]