Science Marches On

The 2012 IgNobel Awards are out, and I trust you will join me in a rousing round of applause for the winners.

I particularly commend to your attention this study, in which the investigators used functional magnetic resonance imaging (fMRI) to show that a dead salmon can determine the emotional experiences of humans depicted in photographs.

The point is that there is currently an avalanche of these studies, mostly funded by you through the National Institutes of Health, in which the investigators claim that they can detect which regions of the brain are active when carrying out certain cognitive tasks, reacting to particular stimuli, or having certain internal experiences. Here is a reasonably accessible explanation of how they have very often gotten it wrong. I'll try to fit it into a pistachio shell for you and just say that the method divides the brain into thousands of "voxels" -- small regions equivalent to 3-dimensional pixels -- and obviously, some of the same ones will happen to light up on the fMRI image of several subjects exposed to the same stimulus, purely by coincidence. There may be a very low p value for these correlations, which the investigators interpret to mean they are not coincidental after all, but that logic is spurious because they fail to account for having made thousands of comparisons.

This misuse of statistics is actually surprisingly common and manages to get past peer review in other fields as well. It just drives me nuts that it continues. Yeah yeah, the spurious findings eventually get forgotten because they don't lead anywhere. Once in a while somebody might even try to replicate them and fail, but that doesn't happen often because it is very difficult to get a study published that only replicates, or refutes, a previous finding. Editors are much more interested in original stuff, and they also don't like to be embarrassed by findings that call into question previous peer review and editorial judgment. (Viz my earlier post about the experiences of Russell Lyons.) We really need to shake up this culture.