Teacher incentives don’t improve student achievement – at least in this case . . .

Abstract:
Financial incentives for teachers to increase student performance is an increasingly popular education policy around the world.  This paper describes a school-based randomized trial in over two-hundred New York City public schools designed to better understand the impact of teacher incentives on student achievement.  I find no evidence that teacher incentives increase student performance, attendance, or graduation, nor do I find any evidence that the incentives change student or teacher behavior.  If anything, teacher incentives may decrease student achievement, especially in larger schools.  The paper concludes with a speculative discussion of theories that may explain these stark results.

This entry was posted in Economics and public policy, Education. Bookmark the permalink.
Subscribe
Notify of
guest
43 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
conrad
conrad
10 years ago

Not just that case — there’s a review by ACER from a few years back that looked at many studies (I think it was commissioned by the Aus government), and that’s the general finding across studies. It’s also worthwhile looking at the opportunity cost of these things, which is almost never considered, and is almost inevitably big.

To me, the fact that performance pay and many other standard “economic solutions” don’t work is quite interesting, since it basically shows you that schools arn’t businesses, which some people seem to forget. The other thing that’s interesting is that despite them not working, they gets proposed time and time again, obviously by people that don’t know how to do a literature search (or perhaps just for political gain). No doubt the same people will be telling us how well vouchers or something like that will work next (but they won’t have any effect either).

Jacques Chester
Jacques Chester
10 years ago

I actually prefer to think of incentive pay as a command-economy tactic. It requires targets, quotas etc and ties carrots and sticks to them. Given the historical experience of such schemes it’s not surprising that they don’t work.

derrida derider
derrida derider
10 years ago

On incentive pay generally, it is very far from a standard “economic solution”. There is a huge body of studies about when and where it works. It does for some jobs, but never works where the value of a contribution is genuinely hard to measure. And it is actively counterproductve where that value depends on “intrinsic motivation” – ie altruistic dedication.

This has been well known by organisational economists for decades – indeed it is utterly conventional wisdom in that field. Its just that there are vested interests that like to pretend otherwise.

stephen
stephen
10 years ago

DD is absolutely right, there is a body of evidence on where performance related pay works and where it does not. At their current levels of pay and conditions, who in Australia would be a teacher unless they had some other underlying motivation?

The word incentives though is a little misleading. I take it from the excerpt quoted by Nicholas that the article relates to financial incentives, but there are other incentives that are important to people in positions like teaching: eg peer recognition, public acknowledgement and thanks, opportunities for increased learning and development, opportunities for more challenging tasks. I am not aware of any studies that have tried to measure these types of incentives, unfortunately – maybe because they are not as easy to quantify. I’d suspect intuitively that these kinds of incentives might make a difference to teacher performance and student outcomes.

There is however no question in all the research that the quality of teachers is one of the strongest predictors of student outcomes. So measures to improve teacher quality, other than incentive payments, ought to be an educational priority. At the risk of provoking howls of outrage, one of the most significant ways to improve average teacher quality would be for the education unions to allow education systems to remove teachers who were not performing. I don’t mean ‘average to poor’ performers, where measuring may be tricky, but the much smaller number of really very bad teachers, those who everyone including their own colleagues know about and who drag down average performance markedly. But that kind of measure still seems a long way off.

rossco
rossco
10 years ago

Nicholas

The problem is how do you distinguish “good teachers” from their peers ie what are the criteria you are going to use and how do you measure performance. There is also an issue of who makes the assessment.

conrad
conrad
10 years ago

“The best thing would be to pay all teachers a lot more, but since we’re not prepared to do that targeted bonuses may offer a better option than not.”

An alternative way of targeting (at least in high schools) would be to look at particular areas where there are simply not enough teachers and where people agree about this, e.g., maths and physics, and offer, say, 20K more to teachers with degrees in those areas. At least this acknowledges that there is a problem and hopefully raises the status of teachers in those areas at least. It also means that you don’t waste oodles of time and money trying to work out performance measuring systems, which is exceptionally hard to do (I think essentially impossible to do well) and always creates resentment in people who rightly or wrongly get an evaluation different to their perception.

Alan Davies
Alan Davies
10 years ago

I don’t think the abstract gives the full picture. The Discussion section of the paper reveals more.

The author says the most likely explanation for the incentive scheme not working is that “along with all other American pilot initiatives thus far, (it) is too complex and provides teachers with too little agency”.

He also says that “although schools had discretion over how to distribute the incentives to teachers if they met their performance targets, an overwhelming majority of them chose to pay teachers equally”.

This project doesn’t seem to throw a lot of light on the question of whether or not a simple scheme with incentives targeted to individual teachers would work.

Nevertheless it certainly highlights the cultural challenges with the idea of individual incentives.

conrad
conrad
10 years ago

“There is however no question in all the research that the quality of teachers is one of the strongest predictors of student outcomes”

There are lots of questions about this actually. People just assume this because they haven’t bothered to actually check the real data. Try doing a google search and have a look at the amount of variance it accounts for in most studies. It isn’t very much. Now do a search on SES and see how much that accounts for.

conrad
conrad
10 years ago

The easiest thing to manipulate is the curriculum, and this is where all the “free” gains are. It’s also much easier to change this than to change, for example, societal attitudes to teachers, which would allow teachers to be paid more. This is no doubt one reason the UK now uses it’s National Literacy and Numeracy Strategy (see here. Basically, you think of really good ways to teach kids (at least for the reading component which I know about, the UK got all its experts in — not just pure education people), prepare everything including daily lessons plans, plans for kids that are not doing as well, etc., and that way kids get benefits from being taught well in often non-obvious ways even if their teacher is not the most inspiring person on Earth.

stephen
stephen
10 years ago

Conrad, of course you are correct; the brevity of comments gets in the way of precision in explanation. Background makes a huge difference, not only SES but also some of the attributes associated with SES. For example, the ease with which children learn to read is directly correlated with the number of books their parents possess.

However, having taken SES into account, teacher quality does make a significant difference to the trajectory taken in learning by a child. A useful Australian oriented review of the literature is on the NSW DET site at https://www.det.nsw.edu.au/proflearn/areas/qt/research.htm (apologies, where I am today does not allow me to include a hyperlink). There is also heaps of material in the US on the same topic, googling with the search terms “teacher quality and student outcomes” is the easiest way to get to the latter sources.

Peter Patton
Peter Patton
10 years ago

Conrad

To me, the fact that performance pay and many other standard “economic solutions” don’t work is quite interesting, since it basically shows you that schools arn’t businesses, which some people seem to forget.

What on odd to thing say. Firstly, just as they can all dance, so too even savages interact via incentives. Secondly, incentives operate at every level and in ever nook and cranny of modern life. Where there is social interaction, you will find incentives: where incentives exist, you will find social interaction.

Do you seriously think that teachers do not already operate in an environment full of incentives, let alone students?

We turn to ‘incentives’ for insight into ANY social circumstance; hardly just in whatever you think “businesses” are.

Peter Patton
Peter Patton
10 years ago

The current situation is an example of a suboptimal pooling equilibrium, where all are paid the same, because there is no way of cheaply identifying any distinct layers of teachers in quality/performance, as the superor separating equilibrium would demand. Except in the classic pooling situation, the pool has too many high quality candidates to make it worth paying the investment that would signal one is superior to the pool, such as via education. In the Australian teacher situation, the system is designed to disallow any signals of superior teachers, and that is how the lock-step pay status quo can be justified.

To this extent, we also have a classic principal agent problem. The mechanism ultimately rewards shirking, laziness, and punishes any perceived signalling by teachers, let alone any attempts to reduce the asymmetries of information; a reduction that would lead to increased outcomes overall, even if it meant getting rid of the lower third of teachers, or so.

Peter Patton
Peter Patton
10 years ago

You can thank Mr. Darcy, Nicholas. If you recall, it was his excuse to Mr. Bingley for declining to dance at the Netherfield Ball.

Peter Patton
Peter Patton
10 years ago

The other profound implication is that lowly pooling equilibrium is too low to encourage many graduates to consider teaching. A separating equilibrium would dramatically increase incentives for graduates to go into teaching.

Ken Parish
Admin
Ken Parish(@ken-parish)
10 years ago

PP may well have a point about the savages. I’m sure it would focus the minds of mediocre teachers wonderfully if a failed performance assessment led to their being cooked and eaten. I’ll bet they didn’t trial that sort of performance pay in those NY public schools, the decadent woozies.

Peter Patton
Peter Patton
10 years ago

The problem is how do you distinguish “good teachers” from their peers ie what are the criteria you are going to use and how do you measure performance. There is also an issue of who makes the assessment.

Why do so many people find this such an insurmountable conundrum, when the rest of our society and economy deals with it every day, without blinking?

Peter Patton
Peter Patton
10 years ago

I think Mr. Bingley said something like, ‘come on, Darcy, all civilized men must dance with the ladies’ to which Darcy replied something like, ‘My dear, Bingley, every savage can dance’.

Peter Patton
Peter Patton
10 years ago

Nicholas

Well all of those studies, like any study in anything, can only focus our generalities and intuition. The comments I made above about pooling and separation comes from a LOT of studies. And like all economic analysis, the classic case provides us with a beacon, or an anchor, to guide our particular study. But I doubt you could argue successfully, “the research shows that the NSW school system has nothing to gain by rewarding superior teachers”.

Fyodor
10 years ago

If you recall, it was his excuse to Mr. Bingley for declining to dance at the Netherfield Ball.

No, it wasn’t. He’s usually pretty good at cutting & pasting the insights of others, but Greenfield’s clearly been eating the paste again.

Paul Frijters
Paul Frijters
10 years ago

the reason we cant increase teacher pay is because we now have far more teachers than before, many of whom teach very few pupils (remedial teaching, rural communities, teacher-trainers) and all whom are tied to the same basic pay schedule. What you really want is to cheat on the existing stock of teachers (dont increase their pay because they are not going anywhere) and pay the new ones more so that better ones come in. There is no way you can do that within a public system.

The NY experiment was interesting because it ‘asked schools’ to hand out a pot of money only certain schools got. Then, the schools decided on the incentive system, and the majority of these schools decided to hand the incentives out not based on performance but on some form of seniority (despite instructions to the contrary).

Peter Patton
Peter Patton
10 years ago

Nicholas

I don’t know of any parent, student/former-student, or teacher who is unable to distinguish who the good teachers are, or who are shy about identifying those superior teachers. The gap between this empirical reality, and the quality of what emerges in the academic economics journals, tells us more about the academics than it does about the parents, students, and teachers.

The chances are slim of finding insight within academic papers published by the overwhelmingly autistic practitioners of academic economics. Sure, economics is always improving, and you approach any social analysis ill-equipped if you are ignorant of economics. But the sheer complexity of what it is trying to pin down means there will be inevitable huge gaps of knowledge, insight, data, tools, blah.

It is a great guide to a government auctioning broadcasting spectrum, or a board of directors on whether on not to take on the only widget maker in town, and a whole stack of other stuff, but so far, it seems that the less autistic practitioners are having too much fun starting/fighting/fuelling fires elsewhere.

Dr Troppo
Dr Troppo
10 years ago

Hmm …

I was in a hospital emergency room the other week and found myself seated near a group of economics students. It was O week and it seems that one of them set himself on fire while drinking overproof rum from a funnel.

In any case, two of the less inebriated baby economists were talking about how to make hospitals work properly. Apparently doctors are idiots and have no idea about anything. The baby-ecs plan involved cashing out the money doctors wasted on various treatments and using it to make incentive payments to ICU patients who managed not to die.

From what I could hear, the plan involved making large payments to patients who exited early and reducing the payments the longer the patient stayed in the hospital. One of them suggested running a randomised control trial. They seemed to get very excited about that.

After they started chanting “Economists versus doctors! Economists versus doctors!” and swigging rum from a bottle, the security guards kicked them out.

margaret
margaret
10 years ago

Having taught in a range of NSW high schools for a long time, i have found that most ineffective teachers do not cope with the rigours of the class room, students drive them out with their behaviour. And teachers can be sacked – it is a myth that there are lots of useless teachers hanging around.

Patrick
Patrick
10 years ago

Gee, Margaret, I do like teachers, I’m related to quite a few, but I also have kids and little siblings so I’ve been interacting with schools (not all in Australia) for a couple of decades now and I’m pretty damn sure there are a few teachers hanging around who really should be sacked.

Tel
Tel
10 years ago

I don’t know of any parent, student/former-student, or teacher who is unable to distinguish who the good teachers are, or who are shy about identifying those superior teachers.

Putting some sort of incentive selection into the hands of the parents and/or students might be a step in the right direction, but although students recognise the likable teachers, that doesn’t necessarily mean they are learning more from those teachers.

If the real problem is lack of visibility into the process, then it makes more sense to put effort into improving the metric long before you start throwing payments around. However, teachers on the whole are resistant to attempts to apply any sort of metric to what they do. The various “school league table” publications were hated just on the basis that the public should not be entitled to know.

I fully agree that test scores are poor metrics, but they are better than no metric at all, and if teachers are genuinely interested in helping themselves, and helping the students and their industry as a whole, they should be the ones proposing workable performance metrics, and they should be the ones pushing to get that information out to the public as much as possible. As pointed out above, the only mechanism teachers ever come up with on their own is one of seniority — which just entrenches the existing power structure. A culture of secrecy guarantees that all attempts at improvement hit a dead end… and right now that seems to be the way that teachers (or at least their unions) want it to stay. Mind you, this is not by any means unique to teachers.

From what I could hear, the plan involved making large payments to patients who exited early and reducing the payments the longer the patient stayed in the hospital.

Sounds open to gaming unless you also pay healthy people for they days they don’t go to hospital… in which case what you essentially have is cash welfare payments plus private hospitals (both of which have already been invented I believe).

But medicine is another place where visibility into the process is poor. If you are going for an operation, how do you get statistics on percentage success rate of this particular surgeon, or this particular hospital? Have you ever been asked to fill a customer satisfaction survey at any stage, anywhere in the medical system? What about regular hygiene checks by an independent outside agency, do they get done? Can I see the report?

conrad
conrad
10 years ago

“I fully agree that test scores are poor metrics, but they are better than no metric at all”

Many people would disagree with that, since, apart from annoying people, poor metrics can obviously change behavior in weird and often negative ways (not that I need to point that out to you), and that is certainly true of education. They also cost money to collect.

Tel
Tel
10 years ago

So get rid of any university entrance exams then?

Incurious and Unread (aka Dave)
Incurious and Unread (aka Dave)
10 years ago

Nicholas,

Your objective seems to be to find a way to pay teachers less overall (or, at least, no more) whilst attracting more good teachers in and keeping them.

It sounds like you need not performance discrimination but price discrimination: ie pay individual teachers no more than it takes to recruit them and keep them.

Good teachers will leave only if they have better opportunities elsewhere. So, perhaps pay should be based on marketability outside of teaching. For example, a teacher with a law degree should be paid more, because they have the option of leaving to become a lawyer.

Meanwhile the “time servers” should be paid less. Not necessarily because they are poor teachers, but because they have nowhere else to go.

conrad
conrad
10 years ago

“So get rid of any university entrance exams then?”

I think universities could certainly do worse than have their own exams that might have far higher validity than year 12 results in many situations. It’s also worthwhile noting that for some courses, where there once were other entrance hurdles (e.g., interviews), these have been scrapped in some areas (medicine) because they have no validity.

I also think that the situation is different — many people are happy to admit that universities are just trying to ration resources such that a certain group of people get all of them and the other gets none. They don’t have the option of giving everyone an equal share, unlike teaching.

SJ
SJ
10 years ago

I think you fell into a bit of a trap there, conrad.

Students tests are a reasonable measure of student ability. The issue at hand, though, is whether they’re a reasonable measure of teacher ability.

SJ
SJ
10 years ago

BTW, here’s link to a paper mentioned by Kevin Drum a few days ago, which, if true, would render all of this teacher performance pay nonsense moot.

(You need to first look at Kevin’s graph, otherwise the paper might not seem to be relevent).

James A
James A
10 years ago

Tel, SJ: Students Know Good Teaching When They Get It, Survey Finds

Teachers whose students described them as skillful at maintaining classroom order, at focusing their instruction and at helping their charges learn from their mistakes are often the same teachers whose students learn the most in the course of a year, as measured by gains on standardized test scores, according to a progress report on the research.

Patrick
Patrick
10 years ago

Critically, they weren’t asked whether the teachers were ‘good teachers’…