Abstract:
Financial incentives for teachers to increase student performance is an increasingly popular education policy around the world. This paper describes a school-based randomized trial in over two-hundred New York City public schools designed to better understand the impact of teacher incentives on student achievement. I find no evidence that teacher incentives increase student performance, attendance, or graduation, nor do I find any evidence that the incentives change student or teacher behavior. If anything, teacher incentives may decrease student achievement, especially in larger schools. The paper concludes with a speculative discussion of theories that may explain these stark results.
Not just that case — there’s a review by ACER from a few years back that looked at many studies (I think it was commissioned by the Aus government), and that’s the general finding across studies. It’s also worthwhile looking at the opportunity cost of these things, which is almost never considered, and is almost inevitably big.
To me, the fact that performance pay and many other standard “economic solutions” don’t work is quite interesting, since it basically shows you that schools arn’t businesses, which some people seem to forget. The other thing that’s interesting is that despite them not working, they gets proposed time and time again, obviously by people that don’t know how to do a literature search (or perhaps just for political gain). No doubt the same people will be telling us how well vouchers or something like that will work next (but they won’t have any effect either).
I actually prefer to think of incentive pay as a command-economy tactic. It requires targets, quotas etc and ties carrots and sticks to them. Given the historical experience of such schemes it’s not surprising that they don’t work.
On incentive pay generally, it is very far from a standard “economic solution”. There is a huge body of studies about when and where it works. It does for some jobs, but never works where the value of a contribution is genuinely hard to measure. And it is actively counterproductve where that value depends on “intrinsic motivation” – ie altruistic dedication.
This has been well known by organisational economists for decades – indeed it is utterly conventional wisdom in that field. Its just that there are vested interests that like to pretend otherwise.
The main reason I’ve had for thinking that performance pay may be justified in an Australian setting – is that while I accept the arguments above, it seems to me that performance pay provides a means to target payment on good teachers, and so to retain teachers who are competent on the miserable wages that we’re prepared to pay teachers.
I’ve always thought it’s a kind of ‘second best’ argument. The best thing would be to pay all teachers a lot more, but since we’re not prepared to do that targeted bonuses may offer a better option than not.
However if that were right one would expect the American data to demonstrate it, because I would have thought that if that argument held here, it would hold there. Two other attempts to save the hypothesis would be to argue that
1) they did it wrong in the test places – that it was too mechanical an approach, that one should give bonuses based on broad criteria, not just the achievement of student scores – though I’m not enamoured of such subjective measures for giving bonuses and imagine they’re a nightmare in the public service.
2) my hypothesis is about the extensive margin of teacher exit and retention rather than the intensive margin of greater teacher effort. If so the effects one expects might emerge over a longer time horizon as more teachers with high student achievement stick at teaching rather than wandering off into PR and other jobs. But if this were the case one would expect to see mounting evidence over time, rather than no evidence in the short term.
So it looks like another hypothesis might have to bite the dust.
DD is absolutely right, there is a body of evidence on where performance related pay works and where it does not. At their current levels of pay and conditions, who in Australia would be a teacher unless they had some other underlying motivation?
The word incentives though is a little misleading. I take it from the excerpt quoted by Nicholas that the article relates to financial incentives, but there are other incentives that are important to people in positions like teaching: eg peer recognition, public acknowledgement and thanks, opportunities for increased learning and development, opportunities for more challenging tasks. I am not aware of any studies that have tried to measure these types of incentives, unfortunately – maybe because they are not as easy to quantify. I’d suspect intuitively that these kinds of incentives might make a difference to teacher performance and student outcomes.
There is however no question in all the research that the quality of teachers is one of the strongest predictors of student outcomes. So measures to improve teacher quality, other than incentive payments, ought to be an educational priority. At the risk of provoking howls of outrage, one of the most significant ways to improve average teacher quality would be for the education unions to allow education systems to remove teachers who were not performing. I don’t mean ‘average to poor’ performers, where measuring may be tricky, but the much smaller number of really very bad teachers, those who everyone including their own colleagues know about and who drag down average performance markedly. But that kind of measure still seems a long way off.
Nicholas
The problem is how do you distinguish “good teachers” from their peers ie what are the criteria you are going to use and how do you measure performance. There is also an issue of who makes the assessment.
Yep. It’s certainly a problem.
You’re right Stephen. On behalf of right thinking Troppodillians everywhere I protest at the notion that the very worst teachers should not be teaching. After all, they have had their police check and have passed the test of not being paedophiles with flying colours.
After all, why do we have schools? To employ teachers. Simple as that. You can’t make an omelette without breaking eggs.
Shocked and horrified,
Port Melbourne.
“The best thing would be to pay all teachers a lot more, but since we’re not prepared to do that targeted bonuses may offer a better option than not.”
An alternative way of targeting (at least in high schools) would be to look at particular areas where there are simply not enough teachers and where people agree about this, e.g., maths and physics, and offer, say, 20K more to teachers with degrees in those areas. At least this acknowledges that there is a problem and hopefully raises the status of teachers in those areas at least. It also means that you don’t waste oodles of time and money trying to work out performance measuring systems, which is exceptionally hard to do (I think essentially impossible to do well) and always creates resentment in people who rightly or wrongly get an evaluation different to their perception.
I don’t think the abstract gives the full picture. The Discussion section of the paper reveals more.
The author says the most likely explanation for the incentive scheme not working is that “along with all other American pilot initiatives thus far, (it) is too complex and provides teachers with too little agency”.
He also says that “although schools had discretion over how to distribute the incentives to teachers if they met their performance targets, an overwhelming majority of them chose to pay teachers equally”.
This project doesn’t seem to throw a lot of light on the question of whether or not a simple scheme with incentives targeted to individual teachers would work.
Nevertheless it certainly highlights the cultural challenges with the idea of individual incentives.
“There is however no question in all the research that the quality of teachers is one of the strongest predictors of student outcomes”
There are lots of questions about this actually. People just assume this because they haven’t bothered to actually check the real data. Try doing a google search and have a look at the amount of variance it accounts for in most studies. It isn’t very much. Now do a search on SES and see how much that accounts for.
Fair point Conrad, but SES isn’t exactly easy to manipulate – teacher quality is at least relatively speaking).
The easiest thing to manipulate is the curriculum, and this is where all the “free” gains are. It’s also much easier to change this than to change, for example, societal attitudes to teachers, which would allow teachers to be paid more. This is no doubt one reason the UK now uses it’s National Literacy and Numeracy Strategy (see here. Basically, you think of really good ways to teach kids (at least for the reading component which I know about, the UK got all its experts in — not just pure education people), prepare everything including daily lessons plans, plans for kids that are not doing as well, etc., and that way kids get benefits from being taught well in often non-obvious ways even if their teacher is not the most inspiring person on Earth.
Conrad, of course you are correct; the brevity of comments gets in the way of precision in explanation. Background makes a huge difference, not only SES but also some of the attributes associated with SES. For example, the ease with which children learn to read is directly correlated with the number of books their parents possess.
However, having taken SES into account, teacher quality does make a significant difference to the trajectory taken in learning by a child. A useful Australian oriented review of the literature is on the NSW DET site at https://www.det.nsw.edu.au/proflearn/areas/qt/research.htm (apologies, where I am today does not allow me to include a hyperlink). There is also heaps of material in the US on the same topic, googling with the search terms “teacher quality and student outcomes” is the easiest way to get to the latter sources.
Conrad
What on odd to thing say. Firstly, just as they can all dance, so too even savages interact via incentives. Secondly, incentives operate at every level and in ever nook and cranny of modern life. Where there is social interaction, you will find incentives: where incentives exist, you will find social interaction.
Do you seriously think that teachers do not already operate in an environment full of incentives, let alone students?
We turn to ‘incentives’ for insight into ANY social circumstance; hardly just in whatever you think “businesses” are.
Thanks for setting us straight on the savages Peter.
Sometimes we need that kind of clean, crisp insight.
The current situation is an example of a suboptimal pooling equilibrium, where all are paid the same, because there is no way of cheaply identifying any distinct layers of teachers in quality/performance, as the superor separating equilibrium would demand. Except in the classic pooling situation, the pool has too many high quality candidates to make it worth paying the investment that would signal one is superior to the pool, such as via education. In the Australian teacher situation, the system is designed to disallow any signals of superior teachers, and that is how the lock-step pay status quo can be justified.
To this extent, we also have a classic principal agent problem. The mechanism ultimately rewards shirking, laziness, and punishes any perceived signalling by teachers, let alone any attempts to reduce the asymmetries of information; a reduction that would lead to increased outcomes overall, even if it meant getting rid of the lower third of teachers, or so.
You can thank Mr. Darcy, Nicholas. If you recall, it was his excuse to Mr. Bingley for declining to dance at the Netherfield Ball.
The other profound implication is that lowly pooling equilibrium is too low to encourage many graduates to consider teaching. A separating equilibrium would dramatically increase incentives for graduates to go into teaching.
PP may well have a point about the savages. I’m sure it would focus the minds of mediocre teachers wonderfully if a failed performance assessment led to their being cooked and eaten. I’ll bet they didn’t trial that sort of performance pay in those NY public schools, the decadent woozies.
Why do so many people find this such an insurmountable conundrum, when the rest of our society and economy deals with it every day, without blinking?
PP – What was Darcy’s excuse?
Peter,
What do you think of all that empirical literature which says that there are a whole host of professions in which performance pay doesn’t improve outcomes – should there perhaps be a bit more blinking? At least in those professions?
I think Mr. Bingley said something like, ‘come on, Darcy, all civilized men must dance with the ladies’ to which Darcy replied something like, ‘My dear, Bingley, every savage can dance’.
Nicholas
Well all of those studies, like any study in anything, can only focus our generalities and intuition. The comments I made above about pooling and separation comes from a LOT of studies. And like all economic analysis, the classic case provides us with a beacon, or an anchor, to guide our particular study. But I doubt you could argue successfully, “the research shows that the NSW school system has nothing to gain by rewarding superior teachers”.
No, it wasn’t. He’s usually pretty good at cutting & pasting the insights of others, but Greenfield’s clearly been eating the paste again.
Well no, but that begs the question of how you decide who they are.
The public service has something to gain by rewarding superior public servants. But deciding who they are is the tricky part and can bugger up any good you do in trying.
This insight comes from EVEN MORE studies!
the reason we cant increase teacher pay is because we now have far more teachers than before, many of whom teach very few pupils (remedial teaching, rural communities, teacher-trainers) and all whom are tied to the same basic pay schedule. What you really want is to cheat on the existing stock of teachers (dont increase their pay because they are not going anywhere) and pay the new ones more so that better ones come in. There is no way you can do that within a public system.
The NY experiment was interesting because it ‘asked schools’ to hand out a pot of money only certain schools got. Then, the schools decided on the incentive system, and the majority of these schools decided to hand the incentives out not based on performance but on some form of seniority (despite instructions to the contrary).
Nicholas
I don’t know of any parent, student/former-student, or teacher who is unable to distinguish who the good teachers are, or who are shy about identifying those superior teachers. The gap between this empirical reality, and the quality of what emerges in the academic economics journals, tells us more about the academics than it does about the parents, students, and teachers.
The chances are slim of finding insight within academic papers published by the overwhelmingly autistic practitioners of academic economics. Sure, economics is always improving, and you approach any social analysis ill-equipped if you are ignorant of economics. But the sheer complexity of what it is trying to pin down means there will be inevitable huge gaps of knowledge, insight, data, tools, blah.
It is a great guide to a government auctioning broadcasting spectrum, or a board of directors on whether on not to take on the only widget maker in town, and a whole stack of other stuff, but so far, it seems that the less autistic practitioners are having too much fun starting/fighting/fuelling fires elsewhere.
Hmm …
I was in a hospital emergency room the other week and found myself seated near a group of economics students. It was O week and it seems that one of them set himself on fire while drinking overproof rum from a funnel.
In any case, two of the less inebriated baby economists were talking about how to make hospitals work properly. Apparently doctors are idiots and have no idea about anything. The baby-ecs plan involved cashing out the money doctors wasted on various treatments and using it to make incentive payments to ICU patients who managed not to die.
From what I could hear, the plan involved making large payments to patients who exited early and reducing the payments the longer the patient stayed in the hospital. One of them suggested running a randomised control trial. They seemed to get very excited about that.
After they started chanting “Economists versus doctors! Economists versus doctors!” and swigging rum from a bottle, the security guards kicked them out.
Having taught in a range of NSW high schools for a long time, i have found that most ineffective teachers do not cope with the rigours of the class room, students drive them out with their behaviour. And teachers can be sacked – it is a myth that there are lots of useless teachers hanging around.
Margaret, I agree that it’s a myth that there are lots of useless teachers hanging around, but there are plenty of time servers and some useless teachers. Surely you agree it’s pretty bad that they can’t be easily sacked so that better teachers can replace them?
Gee, Margaret, I do like teachers, I’m related to quite a few, but I also have kids and little siblings so I’ve been interacting with schools (not all in Australia) for a couple of decades now and I’m pretty damn sure there are a few teachers hanging around who really should be sacked.
Putting some sort of incentive selection into the hands of the parents and/or students might be a step in the right direction, but although students recognise the likable teachers, that doesn’t necessarily mean they are learning more from those teachers.
If the real problem is lack of visibility into the process, then it makes more sense to put effort into improving the metric long before you start throwing payments around. However, teachers on the whole are resistant to attempts to apply any sort of metric to what they do. The various “school league table” publications were hated just on the basis that the public should not be entitled to know.
I fully agree that test scores are poor metrics, but they are better than no metric at all, and if teachers are genuinely interested in helping themselves, and helping the students and their industry as a whole, they should be the ones proposing workable performance metrics, and they should be the ones pushing to get that information out to the public as much as possible. As pointed out above, the only mechanism teachers ever come up with on their own is one of seniority — which just entrenches the existing power structure. A culture of secrecy guarantees that all attempts at improvement hit a dead end… and right now that seems to be the way that teachers (or at least their unions) want it to stay. Mind you, this is not by any means unique to teachers.
Sounds open to gaming unless you also pay healthy people for they days they don’t go to hospital… in which case what you essentially have is cash welfare payments plus private hospitals (both of which have already been invented I believe).
But medicine is another place where visibility into the process is poor. If you are going for an operation, how do you get statistics on percentage success rate of this particular surgeon, or this particular hospital? Have you ever been asked to fill a customer satisfaction survey at any stage, anywhere in the medical system? What about regular hygiene checks by an independent outside agency, do they get done? Can I see the report?
“I fully agree that test scores are poor metrics, but they are better than no metric at all”
Many people would disagree with that, since, apart from annoying people, poor metrics can obviously change behavior in weird and often negative ways (not that I need to point that out to you), and that is certainly true of education. They also cost money to collect.
So get rid of any university entrance exams then?
Nicholas,
Your objective seems to be to find a way to pay teachers less overall (or, at least, no more) whilst attracting more good teachers in and keeping them.
It sounds like you need not performance discrimination but price discrimination: ie pay individual teachers no more than it takes to recruit them and keep them.
Good teachers will leave only if they have better opportunities elsewhere. So, perhaps pay should be based on marketability outside of teaching. For example, a teacher with a law degree should be paid more, because they have the option of leaving to become a lawyer.
Meanwhile the “time servers” should be paid less. Not necessarily because they are poor teachers, but because they have nowhere else to go.
Yes, it’s my (reluctant) objective because Australian governments (like most governments) seem compelled (by their constituents) to act as monopsonists in pretty much any market they enter – except the market for public servants (funny that!)
So yes what you suggest makes some sense and is in fact occurring to some extent. I think there are some bonus funds floating around to recruit teachers with specific attributes – like maths teachers, teachers who’ll teach in low SES areas etc.
“So get rid of any university entrance exams then?”
I think universities could certainly do worse than have their own exams that might have far higher validity than year 12 results in many situations. It’s also worthwhile noting that for some courses, where there once were other entrance hurdles (e.g., interviews), these have been scrapped in some areas (medicine) because they have no validity.
I also think that the situation is different — many people are happy to admit that universities are just trying to ration resources such that a certain group of people get all of them and the other gets none. They don’t have the option of giving everyone an equal share, unlike teaching.
I think you fell into a bit of a trap there, conrad.
Students tests are a reasonable measure of student ability. The issue at hand, though, is whether they’re a reasonable measure of teacher ability.
BTW, here’s link to a paper mentioned by Kevin Drum a few days ago, which, if true, would render all of this teacher performance pay nonsense moot.
(You need to first look at Kevin’s graph, otherwise the paper might not seem to be relevent).
Tel, SJ: Students Know Good Teaching When They Get It, Survey Finds
Critically, they weren’t asked whether the teachers were ‘good teachers’…