Another scandalously flawed discipline . . . medical research

As readers of this blog will know I regard the state of the economics profession as a scandal, and have for years. It’s only occasionally when it really matters, as no matter how good the discipline was it is mostly condemned to ignorance – the world is too complex to understand. But as people like Delong and Krugman have been pointing out simply bizarre arguments – like that US unemployment is mainly structural – (it structurally went from ~5 to ~ 10% following a financial crisis and it’s structural!  Sure it is. And the Great Depression was a spontaneous holiday – sadly that’s not a joke. Economics is a profession where people much cleverer than me think things that are absurd.)  This is just one example, pretty much every odd week sane economists fight off idiotic propositions championed by highly credentialled economists. Like the ‘bond vigilantes’ who were lurking ready to strike at any moment during a prolonged and deep recession. Like that low interest rates present a danger of deflation not presented by higher interest rates – yes folks this view was put by Minnesota Fed President Narayana Kocherlakota.

Anyway, I knew most of the pathology documented in this critique of medical research and publication, but it’s somehow shocking nevertheless.  Of course it’s not really news to anyone but the most naive that the drug industry corrupts both the conduct and the reporting of drug research. But it’s amazing how much of the problem is driven by something much more mundane. Publication bias – the fact that publications publish results, not non-results and science is mostly made up of non-results – tests for possible correlations that turned out not to be there. Publication bias is a bad bad thing, but then of course it gets made much worse because academics need publications and go hunting them. Anyway the article is about Professor John Ioannidis – which in Greek I would have thought would be something like Ioanis Ioannidis:

In poring over medical journals, he was struck by how many findings of all types were refuted by later findings. Of course, medical-science “never minds” are hardly secret. And they sometimes make headlines, as when in recent years large studies or growing consensuses of researchers concluded that mammograms, colonoscopies, and PSA tests are far less useful cancer-detection tools than we had been told; or when widely prescribed antidepressants such as Prozac, Zoloft, and Paxil were revealed to be no more effective than a placebo for most cases of depression; or when we learned that staying out of the sun entirely can actually increase cancer risks; or when we were told that the advice to drink lots of water during intense exercise was potentially fatal; or when, last April, we were informed that taking fish oil, exercising, and doing puzzles doesn’t really help fend off Alzheimer’s disease, as long claimed. Peer-reviewed studies have come to opposite conclusions on whether using cell phones can cause brain cancer, whether sleeping more than eight hours a night is healthful or dangerous, whether taking aspirin every day is more likely to save your life or cut it short, and whether routine angioplasty works better than pills to unclog heart arteries.

But beyond the headlines, Ioannidis was shocked at the range and reach of the reversals he was seeing in everyday medical research. “Randomized controlled trials,” which compare how one group responds to a treatment against how an identical group fares without the treatment, had long been considered nearly unshakable evidence, but they, too, ended up being wrong some of the time. . . . This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” . . . Perhaps only a minority of researchers were succumbing to this bias, but their distorted findings were having an outsize effect on published research. To get funding and tenured positions, and often merely to stay afloat, researchers have to get their work published in well-regarded journals, where rejection rates can climb above 90 percent.  . . .

In the late 1990s, Ioannidis set up a base at the University of Ioannina. He pulled together his team, which remains largely intact today, and started chipping away at the problem in a series of papers that pointed out specific ways certain studies were getting misleading results. Other meta-researchers were also starting to spotlight disturbingly high rates of error in the medical literature. But Ioannidis wanted to get the big picture across, and to do so with solid data, clear reasoning, and good statistical analysis. . . .  In 2005, he unleashed two papers that challenged the foundations of medical research.

He chose to publish one paper, fittingly, in the online journal PLoS Medicine, which is committed to running any methodologically sound article without regard to how “interesting” the results may be. In the paper, Ioannidis laid out a detailed mathematical proof that, assuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time. Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right. . . .  “You can question some of the details of John’s calculations, but it’s hard to argue that the essential ideas aren’t absolutely correct,” says Doug Altman, an Oxford University researcher who directs the Centre for Statistics in Medicine.

Still, Ioannidis anticipated that the community might shrug off his findings: sure, a lot of dubious research makes it into journals, but we researchers and physicians know to ignore it and focus on the good stuff, so what’s the big deal? The other paper headed off that claim. He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association. . . .

Ioannidis points out that obviously questionable findings cram the pages of top medical journals, not to mention the morning headlines. Consider, he says, the endless stream of results from nutritional studies in which researchers follow thousands of people for some number of years, tracking what they eat and what supplements they take, and how their health changes over the course of the study. “Then the researchers start asking, ‘What did vitamin E do? What did vitamin C or D or A do? What changed with calorie intake, or protein or fat intake? What happened to cholesterol levels? Who got what type of cancer?’” he says. “They run everything through the mill, one at a time, and they start finding associations, and eventually conclude that vitamin X lowers the risk of cancer Y, or this food helps with the risk of that disease.” In a single week this fall, Google’s news page offered these headlines: “More Omega-3 Fats Didn’t Aid Heart Patients”; “Fruits, Vegetables Cut Cancer Risk for Smokers”; “Soy May Ease Sleep Problems in Older Women”; and dozens of similar stories. . . .

Most journal editors don’t even claim to protect against the problems that plague these studies. University and government research overseers rarely step in to directly enforce research quality, and when they do, the science community goes ballistic over the outside interference. The ultimate protection against research error and bias is supposed to come from the way scientists constantly retest each other’s results—except they don’t. Only the most prominent findings are likely to be put to the test, because there’s likely to be publication payoff in firming up the proof, or contradicting it.

But even for medicine’s most influential studies, the evidence sometimes remains surprisingly narrow. Of those 45 super-cited studies that Ioannidis focused on, 11 had never been retested. Perhaps worse, Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correct more often than as flawed—in one case for at least 12 years after the results were discredited.

An extremely weird part of the story is where the Prof is asked a question. “If I did a study and the results showed that in fact there wasn’t really much bias in research, would I be willing to publish it?” he asks. “That would create a real psychological conflict for me.” So he’s just as biased as all the scientists he blows the whistle on it seems.

All of which leads me to my conclusion which is that it really is a disgrace that there isn’t some more concerted effort to try to develop protocols to avoid some of the worst of this. Deirdre McCloskey published work many years ago showing how economists’ articles almost invariably reported the statistical significance of their findings without reporting the economic significance (the former is a matter of judgement whilst economic is always what really matters). This created quite a stir, and she did the same survey a decade later. I expected it would have got quite a lot better. It hadn’t – from memory it had got worse, but it may have got (marginally) better.

It is harder in economics because it’s so ideologically loaded, technically dense and ultimately difficult to ever really prove anything of real significance. So even if McCloskey’s critique had been taken to heart – as it should have been – it wouldn’t make a huge difference to our economic understanding, which will never be able to get too far from the informed largely atheoretical commonsense of the chief economists of banks flavoured by some really basic big theories of Smith, Ricardo, Keynes etc.

But medical science? Well it really can aspire to be a lot more useful than that. It’s a biological science and we know a lot about the basic problems. So would it be so hard for those in positions of power in these disciplines and the gatekeepers of the top journals to try to develop protocols to reduce the effects of these problems. I would have thought it is the kind of thing that could be influenced by the leadership of a few acknowledged leaders in their fields – Nobel Prize winners getting together and making a fuss. And if that doesn’t work, whilst I can accept that governments should not be interfering in professional ethics in any direct way regarding individual research, could they not help such a group to form and start promoting their message. We have vast bureaucratic effort from both universities and governments dedicated to measuring scientists’ position in the pecking order research quality frameworks and the variants of the same kind of thing elsewhere, something that may or may not make sense in ranking scientists but certainly makes many of these kinds of problems worse. Might we not aspire to a quality framework which might actively try to work against these well known pathologies of science as it’s practised today and do a better job of generating good science? It’s hard to imagine a more important public good.