School Lunch Quality and Academic Performance, by Michael L. Anderson, Justin Gallagher, Elizabeth Ramirez Ritchie
Improving the nutritional content of public school meals is a topic
of intense policy interest. A main motivation is the health of
school children, and, in particular, the rising childhood obesity
rate. Medical and nutrition literature has long argued that a
healthy diet can have a second important impact: improved cognitive
function. In this paper, we test whether offering healthier lunches
affects student achievement as measured by test scores. Our sample
includes all California (CA) public schools over a five-year period.
We estimate difference-in-difference style regressions using
variation that takes advantage of frequent lunch vendor contract
turnover. Students at schools that contract with a healthy school
lunch vendor score higher on CA state achievement tests, with larger
test score increases for students who are eligible for reduced price
or free school lunches. We do not find any evidence that healthier
school lunches lead to a decrease in obesity rates.
Here’s an ungated version of the paper.
OK, I’ll bite. This is the sort of paper that I don’t generally take seriously. Small effect; complicated environment; several assumptions needed, notably:
“The identifying assumption is that, after controlling for time-invariant
school-by-grade factors, common state factors, and the vector of time-varying,
school-level characteristics, a school’s decision to contract with an outside
vendor for school lunch provision is uncorrelated with other school-specific,
time-varying factors that affect student test performance.”
One red flag is the conclusion that healthier lunches improve grades while at the same time not reducing obesity. That seems a little unlikely, and I’m struggling to see what the causal mechanism is for the grade improvement if this finding is true.
I’m always suspicious of the presence of confounding variables, but this paper really does seem to be a prime candidate. Would changes in school management lead to both healthier meal contracting and grade improvements? Probably. Do the robustness checks control for this? Not well. They try to control for changes in expenditure and student-teacher ratios “as proxies for whether there were additional school policy changes that occurred at the same time as the lunch vendor decisions”. But changes in expenditure and student-teacher ratios have surprisingly little influence on school outcomes compared to a range of other factors, particularly related to management and especially hiring.
Nick, is there a reason you find this particularly convincing?
Note: All my statistics knowledge comes from reading and from running experiments in a business environment. I’ve never taken a full university-level statistics course. Feel very free to point out what I’ve missed, misunderstood or just messed up here.
Thanks David,
This kind of thing has happened once or twice before where people have assumed I’m recommending the paper. Sometimes I do and sometimes I’ve looked at the papers and like them. But all you can be sure of when you see one of these abstracts is that I’ve seen the abstract and think it’s an interesting result/claim. As I posted it I presumed that they might have made the stupid assumption you document them having made. After all, they’re academics right. They’re not interested in getting the right answer, just in publishing papers.
In that sense I’ve outsourced the checking up to anyone who might want to do it, but apologies if you felt misled.
My only excuse is that after my shock resignation, I’ve only remained on at Troppo in an imaginary capacity.
No worries, Nick. I imagine you’ll continue to do a good job.
Imagination is what it’s all about here at Club Pony. In fact I was a silent (imaginary) co-author of Imagine Australia along with Andrew Leigh and other Australians from Harvard which led to a number of further developments.
I didn’t read the paper in-depth, but I moved to the US in 2008, just in time to witness the transition to ‘healthy’ school lunches. The vast majority of students at my school just went off campus for lunch. Around half of schools in California are open campus (students can leave for lunch) and car ownership is high. Any study that doesn’t account for this is probably of limited value.
This is the “complex environment” problem that makes me worry about so many social science papers …