Map of life expectancy at birth from Global Education Project.

Tuesday, August 30, 2022

Reproducibility

Here is an important essay by Jeffrey Flier, former dean of Harvard Medical School. It's a PDF, and it's long, substantial and complicated, so click the link only if you're prepared to make the effort.


Flier approaches the subject with trepidation, as do I, because we both fear that honest discussion of the problem of reproducibility in science can play into the hands of science deniers, who are running rampant and doing great harm to humanity and the rest of the planet. It turns out that systematic attempts to replicate results of published research in the biomedical sciences find that only around 25% are successful, with considerable variation. This means that other scientists followed the methods described in the original paper -- perhaps consulting the authors to get more detail than was published -- and usually failed to get the same results. This is a problem that has been much discussed lately, and I addressed it here a few years ago referring specifically to the work of John Ioannidis (who has since gone off the rails, it happens).


A naive conclusion might be that accepted scientific knowledge is more likely to be wrong than right so why should we believe scientists about anything? But that would be extremely naive. There are literally millions of scientific publications every year, most of which disappear into the slag heap and go unnoticed. It doesn't really matter if they're reproducible. Now, that is a problem in itself. Faculty evaluation, tenure and promotion is largely based on quantity of publication, not quality, and that incentive drives all this publication of work that may not be of the highest quality.


Another problem is that work that does get noticed is likely to present novel or surprising results. I don't want to get too deeply into the philosophy of science here, but the basic problem is that researchers conventionally use frequentist statistics, that is contextless rues of probability and conclude that a finding is correct if it could only happen by random chance 5% of the time or less. But, the likelihood that a finding is correct depends on how likely it was before you did the experiment, not just on the experiment itself. Scientists do many experiments, but they're only likely to write up and publish the ones with "significant" findings -- but if the finding was unlikely before they started, it's still a lot less likely than 95% even after the experiment appears to confirm it. (This is called Bayesian reasoning, after the discoverer of a quantitative theorem about this.)


There are other reasons -- unconscious bias on the part of investigators, unmeasured confounding factors in the experiment, inconsistent reagents and cell lines -- Flier explores these in depth. But . . . 


If researchers go on to rely on these irreproducible findings as the basis for further investigation and experiments, they will soon enough find out that the program of research isn't working and they aren't getting anywhere. Sometimes that takes too long and a lot of time and money are wasted, but it has to happen eventually. Propositions are generally accepted as scientific knowledge only when they make consistent predictions and serve as the basis for additional development of knowledge. All that chaff buried in the mountain doesn't really matter, except for the waste it represents. So when the relevant scientific community tells you, consensually and with confidence that ivermectin is useless against Coviid-19, and anthropogenic climate change is real, you can believe it.

3 comments:

Don Quixote said...

I guess the problem here of proliferating numbers naysayers around this issue is the same as it is for the problem of our supreme court assholes: They say and do whatever they want, throwing out all precedent. In science, this approach includes repeating lies over and over. It's what Shitler does; it's what Hitler did; it's what Republicans do. They lie, over and over, to the point that they can't hear anything else but their lies and reality -- facts -- mean nothing to them.

So your well-thought-out summation of Flier's writing would mean nothing to people determined to deny truth.

Sugarloaf said...

You experts can look after yourselves, I am sure, when it comes to the replication crisis. The layperson needs to rely on an intermediary, like Cochrane, which certainly clarifies the state of knowledge on ivermectin. For “the public understanding of science” maybe more organizations like Cochrane are needed, and for fields other than medical science. Then the effect of the replication crisis might be reduced, at least for the general population.

Cervantes said...

Sug, that sounds like a reasonable idea in principle, but the Cochrane reviews are very lengthy and pretty technical -- they're aimed at clinicians more than lay people. There are resources, by NIH and the Mayo Clinic and others, aimed at lay people. I think the problem is that people don't know what sources are trustworthy, and the quacks are very good at marketing. Viz. the career of Dr. Oz.