Evidence

A couple of questions from commenters have inspired me to produce a bit of a primer on the kinds of evidence we use in health research. It seems to me that a lot of the political controversy over health policy and official recommendations is fueled by limited understanding of the use of evidence to make causal inferences. As I set out on this project I really have no idea how long it will take. We'll just see what happens.

Categories of evidence are commonly organized in a hierarchy, with some kinds said to be stronger or more compelling than others. I don't exactly look at it that way. Different kinds of evidence are useful in different situations and lead us to different kinds of conclusions. Just as important, how we regard any kind of evidence must depend on the prior plausibility of what it seems to show. If there is already very strong evidence for or against some conclusion, then any new evidence that contradicts what already seems highly probable is less compelling, and we should be much more diligent in looking for flaws in the study or concluding that chance alone is responsible for the observations.

This is different from confirmation bias, which is a feature of human cognition in which we tend to ignore or explain away evidence that contradicts what we already believe. We should certainly take surprising findings seriously, but we should subject them to careful scrutiny. If they do compel us to re-examine our prior beliefs, we should certainly do so. I will address these issues a bit more formally as we go along.

Alright. As most people know, conventionally anecdote is presented as the least convincing kind of evidence. An famous example of anecdotal evidence is the putative association between vaccination and autism. Parents see their child receive a shot, and some time not long after they start to see developmental regression and symptoms of autism. Unfortunately, some parents have an unshakeable conviction after this experience that the vaccine caused the autism. As I'm sure most readers already know, this is a common fallacy called post hoc ergo property hoc, because if you say it in Latin you must be really smart. I think if we started naming it in English we would still be just as smart, so I'll call it after this, therefore because of it.

While overwhelming, really incontrovertible evidence of much greater force has shown that vaccination does not cause autism, it is wrong to say that anecdotal evidence is worthless in general or should be ignored. A famous example is the efficacy of parachutes. Since we know that falling from 10,000 feet is invariably fatal, seeing a person do so with the aid of a parachute and land unharmed, even once, is extremely compelling evidence for the efficacy of parachutes. Not only has something obviously extraordinary happened, we can immediately see how it apparently works: by air resistance. The scenario makes perfect sense: it has high prior plausibility. Seeing it happen once wouldn't necessarily make me confident enough to strap on a parachute and jump out of an airplane; but it would make me believe that if the equipment were sufficiently reliable, and weather and other conditions similar to those pertaining to the observation, the parachute would be highly likely to work.

To return to the case of autism, suppose no child had ever been vaccinated and no child had ever been autistic - or at least that autism was extremely rare. If the first child ever to be vaccinated subsequently became autistic, we would properly have a very high index of suspicion that the events might be associated. If we could think of a highly plausible biological mechanism, our suspicion would be even stronger. Neither of these requirements holds in the real case, however.

Moving along, although it is often said that the plural of anecdote is not data, this is certainly false. Data is a synonym for information, and the more anecdotes, the more information. In fact the plural of anecdote can be organized formally into a kind of study called a within subjects design. I'll talk about that in the next installment.