Clinical trial: More epistemology

 A side note: I'm well aware of Project 2025, and actually it has been extensively covered by the corporate media. It might interest people to know that the Republican nominee for president "rushed to distance himself from the Republicans’ highly controversial Project 2025 Friday, calling parts of it “ridiculous and abysmal.”The ex-president used his Truth Social platform to disavow the platform, drawn up by the Heritage Foundation, which offers a 900-page preview of how the most powerful think-tank in the conservative movement wants him to govern."Actually, however, he would govern as a tool and do everything they want him to.

I've written about the expense, complexity and other challenges of conducting rigorous clinical trials. What can we do in situations where a trial may be unfeasible, or unethical, or can't give us good information about how a treatment would work in the real world? One possibility is what are called pragmatic trials. Where different treatments are available, or people may or may not choose to accept or fill a prescription, we could just look at what happens in reality. For example, some physicians may happen to favor one course over another. This can happen because they have different specialties -- surgeons will go for surgery, radiologists for radiation, oncologists for chemotherapy. (Really.) And believe it or not there are regional variations in what doctors are likely to prescribe, in other words essentially different cultures of medical practice. Of course, individual physicians also have their idiosyncrasies.

 

However, it is difficult to do this kind of study, both logistically and analytically. You do need to collect a lot of accurate data about the subjects, which means you already have to intervene in the clinical relationship and process. You don't need people to give informed consent for an intervention, but you probably do need to collect information from them at multiple time points, which means you need to be able to follow them and you probably need to pay them for their time. There may be differences between people who you manage to keep track of and people who are "lost to followup," of which there will likely be quite a few. 

 

Then you have the problem of "confounding by indication." Often the people who get the treatment will actually appear to do worse, but that may be because the physician is more likely to prescribe it to people who seem to need it more. The people who are sicker in the first place are the ones who get it. And obviously there could be other differences between people who do and to not get the treatment having to do with physician biases, personal characteristics affecting the choices patients make, and more. 

There are statistical methods to try to correct for these problems. For example, propensity score matching uses regression to find the predictors of who does and does not get the treatment from the total population, and then matches people who have similar scores but different treatments, to compare their outcomes. Marginal structural models are more sophisticated, and don't require one-to-one matching but rather use information from everyone who is within a short distance of the mean score. But these assume we have all the relevant information, and we probably don't. 

In the end, these sorts of studies or analyses vary a lot in how compelling they are. To really get a credible idea about treatments, it's necessary to combine information from many different studies in what's called a meta-analysis, which may include both rigorously structured trials and some of these less formal pragmatic analyses. Getting a solid idea about what works often takes a long time, and treatments may be in fact be in use for a long time before people figure out that they don't really work after all -- but then it turns out that it's really difficult to convince the medical profession or the FDA of that and get them off the market or at least stop using them inappropriately.

But for readers of this blog, the lesson I want you take away is not to send me a link to a single study purporting to show that horse paste cures Covid-19. Yes, there were a couple of studies claiming that it does. But it doesn't. Unfortunately there are other treatments that don't work that don't get the scrutiny that results in definitive rejection, and others that doctors keep doing because they get paid for it. I'll have more to say about what we might do about this in due course.