Lottery policies – places for transparent arbitrariness

As a summer exercise I’ve been thinking about places where more lotteries might be a good idea. By lotteries, I mean a decision maker selecting an option randomly, albeit perhaps from a selected pool, rather than using flawed criteria. After all, in a complex and uncertain world, even the best available criteria will be flawed and will insert some arbitrariness into the system. At best this just results random bad choices. At worst it will promote adverse behavior and selection and resentment amongst participants. Why not just make arbitrariness transparent and more purely random?

The Peter Principle is one (humourous but insightful) example, often put as “people are promoted to their level of incompetence”. Using the available criteria of people who are competent in their current jobs, selection of those to be promoted often fails to select the bets candidates for duties higher in the hierarchy. The bad selections oly become apparent when someone fails to perform in their new role. Plunchino and Rapisarda, after crunching the concept through game theory, found that random promotion may be the most efficient way of selecting those to promote.

Hierarchies, both in government and in the corporate sector, are rife with terrible selection procedures. The Peter Principle merely choses people on a criteria that doesn’t reflect what is really wanted. Many selection techniques actively choose bad traits. I’ve talked about the way narcissism may be partly responsible for a gender bias in executive positions. Those who are over confident are much more likely to put themselves forward, and to be able to sell themselves to decision makers. Unfortunately the most confident people are over confident. They’ll tend to be risk seeking, over estimate their own judgement and disregard advice, have an inflated sense of their own value (and subsequent entitlements) as well as an ability to deflect blame and subsequently fail to learn from mistakes. In short, you might be choosing the kind of person that takes a company down a risky road to the cost of stockholders, employees and eventually taxpayers – and then considers that they deserve a large bonus. Perhaps random promotion would have resulted in a broader array of neurological types, and more sober and responsible decision making from executives – or at least the kind that aren’t predisposed to exploit the principal-agent problem whenever it arises. It certainly would address the critique of quotas that Nicholas gives here, where a quota for women would merely promote narcissistic women.

Another problem with flawed criteria is that participants will quickly start altering their behavior to target it. If you start rewarding teachers based on test results, you risk them teaching to the test or “juking the stats”. When it comes to selections, this behavior may be very costly and, like persuasive advertising, is an aggregate economic loss. It’s only virtue is affecting the selection, often to the detriment of the quality of the decision, and at the expense of other behavior that might benefit the broader context, whether the organisation or society. In the hierarchical context, this can mean brown nosing and sycophancy at the expense of actual work (and which may be damaging to morale), or behavior that looks like hard work instead of productive work. The long working hours in Japanese firms are often filled with not very much work for the sole reason that selection favours the easily observable (working hours) over the less easily observable (productivity) – this has social costs of course, but it also means a tired and less productive workforce.

Another example includes our system of university admissions. The ranking (TER or UAI or ATAR or whatever the current acronym is) may be broadly a fairly good measure over all of intelligence, work ethic, studiousness etc. The things you want when selecting a university applicant. But at the margin small increments count for a lot. Towards the top rankings, where actual admissions are determined, particularly in popular courses, a few points make all the difference, and those increments become relatively less about the characteristics universities want (and what we might consider virtue) than they are about very costly behavior. Financially costly, like tutors and private schools (particularly those adept at gaming the system[fn1]); but also constant cramming at the expense of extra curricular activity, or of a social life (including every teenager’s right to do dumb things and make romantic mistakes), or of genuine learning or simply of sleep. Selection of subjects that the student is interested in comes second to their perceived contribution to the ranking – despite efforts to make selection neutral by ranking designers. All sacrificed things are valuable things in their own right might even contribute to the qualities universities might look for (and in other countries do look for by far more flawed application procedures than here), but all are sacrificed for a few more points, points that are mere noise when it comes to aptitude but are crucial to selection. Additionally of course, there is adverse selection, by choosing hothoused kids who need constant reinforcement in terms of marks and have narrow experience.

Why not just make the selection random above a certain threshold. A student with 90.00 is probably more adept than one with 80.00, but above that the gradations mean little in terms of what we want in university. Those above this threshold could simply be put in a lottery. That way we still select for quality without as much (we cannot eliminate it) incentive to engage in virtueless but costly activity at the margin. We might also avoid fostering a exam culture, such as those inspired by Confucian or Raj civil service exams.

A similar system may work for research funding. There are enough problems picking winners in research (particulary science), but we also have a great deal of wasted time when researchers write funding proposals and start trying to second guess what funding priorities might change, what “themes” the bureaucracy will pursue. Satisfactory proposals (and this still allows for a focus on certain areas) go into the hat, funded proposals come out and researchers can get on with research. Maybe research journals as well, which already use quite arbitrary techniques, and may lead academics to seek to satisfy the prejudices of editors. Likewise arts funding. Instead of selecting based on the tastes of the Australia Council fiefdom, or a desire to ape what has been done elsewhere by means of expensive cover bands, we fund random applicants who show a sufficient merit. We might even get a film watchable outside the self congratulatory arts stockade, hostile to the barbarians outside who like things like character and plots.

The main problem with all these is that transparent arbitrariness  might cause resentment. But I am not convinced. Here’s Chris Blattman discussing random controlled experiments in aid.

Is it the trial that causes hard feelings, or the fact that we are giving aid to one person and not another? The latter happens as a rule. So long as aid is scarce, there’s always a ‘control group’ not getting the program.

There are two differences with an RCT: getting the program is random, and we’re collecting data on the unlucky ones. Does collecting the data create hard feelings? Not in my experience. What about the lottery aspect? The main reaction I’ve heard from communities about RCTs: “Finally, we know why some people get aid and others don’t, and we all have a fair shot.”

Aid allocation is often ad hoc, sometimes corrupt, and never transparent to the people who don’t get it. In some cases, lotteries among the deserving might actually be an improvement, or at least no worse. That’s when RCTs can be appropriate.

In an uncertain world with flwed criteria, there are always arbitrary decisions. A lottery makes this transparent. The losers may have missed out, but they know that they were always in the hat, and that their time might come again. They won’t be bitter that so and so was chosen because they were a suck up, or because of their colour or looks or height. They won’t resent that other schools gamed the system, or that they had one bad teacher, or that they got a bad question on the day of the exam. They won’t resent that a ministerial change meant that funding themes for research changed or that a rival had daddy’s friends on the funding council.

FIFA might learn that give world cups randomly to any proposal that is acceptable, lest they continue to be considered corrupt.

With flawed criteria, the losers suspect it was merely bad luck that resulted in their falure, but they also suspect that they were never in the hat to begin with. With a lottery, they know not to stop trying, because it was clear that they had as much chance as anyone else. It helps that losers know that they were in the hat.

[fn1] Such as allowing a student a scribe in a arts exam because they had difficult writing, but a scribe who coincidentally was a masters student in fine arts.

About Richard Tsukamasa Green

Richard Tsukamasa Green is an economist. Public employment means he can't post on policy much anymore. Also found at @RHTGreen on twitter.
This entry was posted in Economics and public policy, Uncategorized. Bookmark the permalink.
Subscribe
Notify of
guest
6 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
conrad
conrad
10 years ago

I personally don’t mind the idea of a lottery for many things — however, I think:

a) there are often quite powerful vested interests in many of these things, and, in some areas, its simply too easy to convince people that there is validity in your method, even if there isn’t (indeed, I doubt the average punter even realizes that there is a thing called validity, so you can often just rely on people thinking that there is). This means people want the tests anyway. The grade 3 NAPLAN is a good example of this — the test has awful face validity (i.e., doesn’t even contemplate looking like it measures what it is supposed to), has never been examined for convergent validity (i.e., has never been shown to produce similar scores on tests everyone agrees measure what they are supposed to), but it’s validity is hardly even meaningfully questioned.

b) Many tests of questionable validity might be very weak predictors of things. When does one say a lottery is as good as a test? When the test only explains 10%, 5%, 1%, or .1% of the variance? Should the negative effects, like those that you be mentioned, be included in this?

c) For reasons which I don’t quite understand (it’s an interesting question in itself), people seem to think tests of no validity are fairer than lotteries for some but not all things.

Nicholas Gruen
Admin
Nicholas Gruen(@nicholas-gruen)
10 years ago

Thanks – I think I agree. A couple of points.

1. One thing distancing the 99.95% kid from the 98% kid may not be any of the things you mentioned. At least in maths and science, you can’t do the best in the state without learning to remove mistakes. Now removing mistakes is by no means a worthless thing. But it shouldn’t be the main criterion by which one separates the very goods from the very best. I’d like to see such distinctions made, if they can be based on something more substantial.

2. Randomisation is an important thing. Lateral Economics once participated in an ‘independent’ assessment of whether a certain grants scheme was worth the money the government spent on it. Hired by the agency that doled out the money, you’ll never guess what we found? Infact we were honest enough to say that one couldn’t really tell though we thought the answer was yes. Why couldn’t we tell? Because there was no way of determining whether the firms that got the grants would have done the good things they were now doing even if they’d not got the grants. There’s basically no way of testing this – or no satisfactory way. But one could do it if there was random selection. You just compare the population randomly given the grant with the population that wasn’t and Bob’s your uncle.

Since these grants were supposed to go only to firms who wouldn’t do the desired thing unless they got the grants (something that was untestable once you’d given them the money, it occurred to me that there was an incentive compatible way to do this. One would randomise the grants, get applicants to say what they would and wouldn’t do with and without the grants and then if they didn’t get the grant and went ahead doing what they said they needed the grant for, you’d fine them some amount to make it worthwhile for them to tell the truth in the first place. It wouldn’t really work politically or even administratively, but it works at the thought experiment level, and it depends on randomisation. We did end up suggesting that at the margin they should engage in randomised approvals, so that they gave themselves something with which to measure the success of their program.

No prizes for guessing whether they have done this.

Why are we so resistant to such ideas. I think the same mind muscles that make us prey to what Nicholas Nasim Taleb calls “the narrative fallacy”. Just as we must have explanations for what is, in fact, the random, so we actively dislike randomness in a normative space, and want to cover it with our narrative – our normative narrative where the person we tell ourselves was most deserving got the lolly.

trackback

[…] of reverse discrimination and came home to find something else I’d said about it linked to by Richard Green. To introduce the issue, here was my […]

conrad
conrad
10 years ago

“But it shouldn’t be the main criterion by which one separates the very goods from the very best. I’d like to see such distinctions made, if they can be based on something more substantial.”

At least for things like maths and university entry, I imagine this would be very easy — you could just set an extra test for the top echelon that would test more complicated things (e.g., proofs) that arn’t on the normal exams because so few people would get them. The obvious reason this isn’t done is because for all of the courses that require almost perfect marks (e.g., medicine), this type of knowledge of maths presumably makes no difference (alternatively, as you note, being obsessive could), and thus we come back to the problem of people wanting any form of marker, no matter how little validity. Off hand, I can’t think of any courses where mind boggling knowledge of something is needed to get in (although I can imagine cases where someone might be super at something and not so good at something else, and the current system doesn’t recognize that).

“Randomisation is an important thing.”

I think economists are obsessed with randomization (although perhaps that’s from reading Andrew Leigh’s old blog). For many things, it just doesn’t make sense (e.g., probably most types of general science grants — since each project has different outcomes that are often almost impossible to compare). Even for things whether it could be used (e.g., evaluating an educational program), the cost in terms of extra time and money is often vastly more than non-randomized designs. This means that for the time and effort of running a randomized study, you could have run a couple of non-randomized ones, and in areas like education, you are often likely to learn far more running a couple of little tightly controlled experiments with nice designs than big randomized experiments. Of course it depends on the area — if the government is going to roll out some massive change (e.g., a national curriculum), it makes sense to really go the whole hog, but if you’re interested in, say, whether some tool helps kids learn some aspects of maths better, its a waste of time and money.

john walker
john walker
10 years ago

“… we fund random applicants who show a sufficient merit…”

With the possible exception of art forms that really involve public performance “Merit” is a tricky slope. The boards system started on the idea of expert peers choosing and promoting meritorious “advanced” art ; art that was by inverse definition not supported by the market I.E it had to be something that had little or no public support and was hard to see – it has rolled onward ever since.

Some extracts from a SMH editorial published in 1975 about the then crisis in the OZCO.

“…Still public questioning of the Australia Council’s lack of a coherent policy in overdue. So much is plain from Dr Battersby’s waffle to the IAC. “since the grants system was introduced” she said, “there has been an enormous increase in creative activity”. No doubt. But artistic quality? Perhaps it is too early to say. But, granted that answers to questions about ends are bound to be value-judgments and arguable, it is simply not good enough to pretend that generalities about “activity” meet the need for a critical economic rationale in examining the cost-effectiveness of the means being employed. No such rationale has been offered or, apparently considered. ……….

….The justification for Government subsidies used to be summed up by “the pursuit of excellence” a phrase now damned as “elitist”. At least the philosophy behind it clarified objectives and indicated means and priorities…..”

“It put the emphasis on the context in which the artist works (eg. the performing arts, uncommercial institutions) rather than the artist, in whom self-reliance was believed to be a virtue. So far as individual artists were concerned, its criterion was attested achievement rather than aspirations. No doubt it had practical limitations – partly because in Australia until 1973 it was under-funded.

“The current confusion about about ends and means is, ironically, a result of vastly increased funds. “The pursuit of excellence” is submerged by other considerations. There is the artist’s prosperity. (“If you want to have certain creative people in our community, then you have to support them”, says Dr Battersby. Really? How did all who are our culture, from Lawson and Roberts to Hope, White and Drysdale survive?) There is a vague egalitarian hankering for cultural democracy (“taking the arts to the people”). There is the veneration of youth and “experiment” (ie. Untested talents and works).”

“The result: the subordination of “excellence” to “activity”.”