Map of life expectancy at birth from Global Education Project.

Saturday, February 13, 2021

A primer on the evaluation of therapeutics

I'm actually going to talk here mostly about how we decide that medications are effective, but the basic principles apply to other kinds of interventions such as surgery, physical therapy, or devices. However, those do present some additional difficulties, which I won't go into right now. 


Conventionally, we speak of a hierarchy of evidence, from least to most persuasive. The weakest form of evidence is anecdote. Dr. Fell says he gave eye of newt to several of his patients with crotch rot and they got better. (This is essentially what started the hydroxychloroquine madness.) 

 

There are many reasons why doctors all over the world don't rush out and start giving eye of newt to all their patients. Dr. Fell may simply be a nut. But even if he's sincere, with most diseases, presumably including crotch rot, and including Covid-19, most people normally get better. So Dr. Fell's observation doesn't show that eye of newt actually helped. He may be unconsciously motivated to see more or faster improvement than expected, because he wants his treatment to be useful. And there may be adverse effects that he overlooked or haven't emerged yet. He may also have, perhaps unconsciously, given the nostrum to patients who would be most expected to improve. There are more reasons to doubt this but I'll stop there.


The next level of evidence is still considered too weak to draw any conclusions about the safety and effectiveness of a medication. This is called a retrospective cohort study. It usually isn't feasible because it would require that some substantial number of physicians are prescribing a drug for an unapproved indication, but this is what happened fairly early in the Covid-19 pandemic during the HCQ craze. In this form of study, researchers find a group of people who have had the disease and they look back to see which of them were prescribed the drug, then look for differences in their outcomes.


You probably don't need a Ph.D. to see why this sort of evidence is usually not very persuasive. The people who were prescribed the drug might be different from those who weren't for many reasons. In the case of Covid-19 specifically, doctors who prescribed HCQ might have been in practices with more affluent and lower risk patients. Given that it is known to present a risk for cardiac complications, they likely would have chosen not to prescribe it to people with risk factors such as heart disease or obesity, which also happen to be risk factors for worse outcomes of Covid-19. You can try to adjust for some of these potential "confounders" in your statistical analysis, but you can't adjust for ones you hadn't thought of or that you can't measure.  In the case of a specific study which has been brought to my attention, only a small number of patients were actually prescribed HCQ, which means the sample size was small, and as it happens the effect was only marginally significant. It is possible to massage your regression until you get a confidence interval that doesn't cross zero, and unfortunately that sort of post hoc analysis is all too easy to do in this situation.


Furthermore, there is the problem of publication bias. One of these studies might get a significant result just by chance, while others do not. But the negative studies won't be published. The particular study in question was published in an open access journal that evidently has low standards, and it has been superseded by later, higher quality studies that find no effect, or even a possible risk of harm.


These studies are called randomized controlled trials, and they are the only way to get FDA approval. The problem of publication bias has also been ameliorated for these studies because they have to be registered in advance. That means the data will be available regardless of whether they are published. In these studies, a group of patients with the disease -- probably matched in some way such as disease severity and stage -- are randomly assigned to receive the drug, or a "placebo" that is physically indistinguishable. Neither the patient, the treating physician, or the assessor who judges outcomes knows who did and did not get the real drug. Then the patients are followed for a time to see how they do with respect to the disease, and also to look for adverse effects. If a large sample is randomly assigned, chances are the intervention and control arms will have similar characteristics, but statistical techniques can be used to adjust for any major differences that happen by chance. You don't have to worry about prescriber bias because the intervention was assigned at random.


In the case of HCQ, after the anecdotes and retrospective studies that in some cases indicated the treatment might be promising, investigators undertook large, high quality randomized controlled trials at every stage -- for prophylaxis, for treatment of moderate disease, and for treatment of hospitalized patients. They were all, uniformly negative. It doesn't work for Covid-19 under any circumstances, and it may be harmful. The editors of the New England Journal of Medicine, The Lancet, and other leading medical journals have all come to his conclusion, as has the FDA. You don't know more than they do, and you don't know more than I do about this subject either.


Let me conclude by saying that unfortunately, this is an all too common pattern. People hype therapies based on weak evidence, the corporate media pick up on the hype, and later, more rigorous studies find the snake oil doesn't work. We see it again and again. What we don't see is an ignoramus occupying the office of president of the United States joining in the hype. That's because usually, they aren't ignorant idiots and they know better.

Addendum: Everything I have written here is absolutely, irrefutably true and you will learn this if you take a basic course in medical school or a school of public health. It is not to be confused with what doctors actually do, because if a drug is approved for any use, they are legally allowed to prescribe it however they want, and sometimes doctors prescribe drugs "off label." That doesn't mean the drugs actually work, it just means some doctors think they do, or might. The FDA can also issue what's called an Emergency Use Authorization, in cases of a public health crisis, which they in fact did in the case of HCQ. Then they withdrew it when good evidence came in. Doctors are trying other therapies for Covid-19 now, but just because they are trying some doesn't mean we know they work. Most of them will turn out not to, if the usual pattern follows.

No comments: