Map of life expectancy at birth from Global Education Project.

Monday, June 20, 2011

Well, nobody can accuse me of chasing traffic

Not if I refer you to a highly technical essay that will make your head hurt. Russell Lyons of Indiana University is annoyed -- so annoyed in fact, that he transgresses the boundaries normally expected of academic writing to trash both some highly respected and widely touted investigators, the world's leading medical journals, and peer review in general -- the latter two categories of trashee coming in for extra trashing in part because he couldn't get this particular essay published in high impact journals. So he's uppity, shrill, rude and obnoxious.

He's also probably right. In order to save you from a headache, I'll just explain this very simply. You may remember that the popular media a few years back got all excited by research based on data from the Framingham Heart Study which found that fatness is contagious. Basically, if your friends get fat, you are likely to get fat too. These investigators -- principally N.A. Christakis and J.H. Fowler -- followed up with similar findings for quitting smoking, being happy, and being lonely. These findings were all published in very high impact, prestigious journals.

Lyons calls bullshit. Without going into all the arcana, his main points are:

1) Their claims of statistical significance for many findings are bogus. In fact, the confidence intervals overlap, by a lot, but they conjure of the confidence intervals away by pretending it is really the number zero. (I kid you not.)

2) The logic by which they conclude that obesity is contagious, rather than the alternative explanation that obese people are more likely to name other obese people as friends, is bo-o-o-o-gus.

3) The statistical methods they use to try to control for potentially confounding variables do no such thing.

But the main point of this rant -- which is what it really is in spite of the venue -- is that a lot of investigators who use statistics don't really understand what they are doing, and neither do the peer reviewers who decide where they get their stuff published. This is true, I think. I have written before about the almost universal misinterpretation of the p value, and that is indeed part of what is going on with these social network analyses. It is a major reason why, as John Ioannidis has found, most published findings are false.

Now get a grip. This doesn't mean that global warming is a hoax or HIV doesn't cause AIDS. We're talking about specific, new findings right where the mining machine of science cuts into the rock face of ignorance. After a while we get the basic story right. But physicians and yes, scientists; reporters; and average people are just too quick to accept the latest weird and unlikely discovery as an important scientific breakthrough, whereas most of them are just noise. And as Lyons suggests, scientific journals should be a lot more interested in replication and criticism of prior findings, rather than relegating them to the lowest impact venues.

No comments: