Reproducibility

Here is an important essay by Jeffrey Flier, former dean of Harvard Medical School. It's a PDF, and it's long, substantial and complicated, so click the link only if you're prepared to make the effort.

Flier approaches the subject with trepidation, as do I, because we both fear that honest discussion of the problem of reproducibility in science can play into the hands of science deniers, who are running rampant and doing great harm to humanity and the rest of the planet. It turns out that systematic attempts to replicate results of published research in the biomedical sciences find that only around 25% are successful, with considerable variation. This means that other scientists followed the methods described in the original paper -- perhaps consulting the authors to get more detail than was published -- and usually failed to get the same results. This is a problem that has been much discussed lately, and I addressed it here a few years ago referring specifically to the work of John Ioannidis (who has since gone off the rails, it happens).

A naive conclusion might be that accepted scientific knowledge is more likely to be wrong than right so why should we believe scientists about anything? But that would be extremely naive. There are literally millions of scientific publications every year, most of which disappear into the slag heap and go unnoticed. It doesn't really matter if they're reproducible. Now, that is a problem in itself. Faculty evaluation, tenure and promotion is largely based on quantity of publication, not quality, and that incentive drives all this publication of work that may not be of the highest quality.

Another problem is that work that does get noticed is likely to present novel or surprising results. I don't want to get too deeply into the philosophy of science here, but the basic problem is that researchers conventionally use frequentist statistics, that is contextless rues of probability and conclude that a finding is correct if it could only happen by random chance 5% of the time or less. But, the likelihood that a finding is correct depends on how likely it was before you did the experiment, not just on the experiment itself. Scientists do many experiments, but they're only likely to write up and publish the ones with "significant" findings -- but if the finding was unlikely before they started, it's still a lot less likely than 95% even after the experiment appears to confirm it. (This is called Bayesian reasoning, after the discoverer of a quantitative theorem about this.)

There are other reasons -- unconscious bias on the part of investigators, unmeasured confounding factors in the experiment, inconsistent reagents and cell lines -- Flier explores these in depth. But . . . 

If researchers go on to rely on these irreproducible findings as the basis for further investigation and experiments, they will soon enough find out that the program of research isn't working and they aren't getting anywhere. Sometimes that takes too long and a lot of time and money are wasted, but it has to happen eventually. Propositions are generally accepted as scientific knowledge only when they make consistent predictions and serve as the basis for additional development of knowledge. All that chaff buried in the mountain doesn't really matter, except for the waste it represents. So when the relevant scientific community tells you, consensually and with confidence that ivermectin is useless against Coviid-19, and anthropogenic climate change is real, you can believe it.