As an aside, the term ‘data mining’ is not entirely negative. There is a large industry of software and practitioners who use data mining in a statistically respectable manner. The trick is to have a very large dataset, do your ‘mining’ in one half of the dataset, then test the estimated model in the other half.

Something that annoys me is proponents’ of econometrics unwillingness to adopt a more visceral and honest approach to research.

To admit that the crafting of variables is probably THE MOST important part of the process, and that everything else (the relationships between them) can flow from there.

Sadly, variables are restricted to those which have data available.

NB I ask Andrew Leigh about this via blog comments (I deign to use private correspondence in a principled attempt to make these discussions public, but I guess people of status have got there because they care about such things, and discussions with the likes of me may undermine that status) but received no response.

Of course if it’s sampling without replacement and the population size is small, this is perfectly rational.

Economists aren’t the only ones who data mine – in fact, they’re far from the worst (marketing people IME are the worst). It’s widespread wherever statistics are used and a risk in all frequentist approaches. So maybe the best remedy is to be more Bayesian.

But ultimately we have to understand that there are no shortcuts to truth – definitely establishing whether A causes B is often far harder than it seems.

As a non-economist I’ve always been puzzled by the rituals of the discipline (how to get published, win peer respect etc). Each discipline seems to have its own.

Psychologists like experiments. But they tend to run a lot of them with American college students and under conditions that make it difficult to generalise the results outside the laboratory.

For a while it was fashionable in psychology to practice a crude kind of operationalism where it was forbidden to appeal to any unobservable process (eg thoughts, feelings or neural processes).

Anthropologists like ethnography and are very impressed when a researcher immerses themselves for years in an interesting foreign culture (an economics department perhaps?).

Economists love regressions seem to hover like seagulls around organisations with large data sets.

If an economist was researching unemployment and you offered to let them spend a week watching people in a welfare to work office do their job, they’d probably think you were trying to contaminate the research process (or waste their valuable time).

The odd thing is, if there are any good theories in social science, they’d have to apply across these disciplinary boundaries.

Sorry if this off topic.

I think the real test would be for someone else to re-run a pile of other people’s experiments from the same journal (which would be possible in some areas but not others). That way you could see to what extent results really are getting exaggerated by everybody trying to find the magical .05 value. Unfortunately, in many areas, this is an impossible strategy, and failures to replicate almost never get published. This is why we end up with a massive proliferation of little effects that don’t really matter.