Clinical Trials, Part Two: The ideal

Last time I sketched out, very briefly, some of the ways we can fool ourselves about medical interventions -- treatments. It should be a sobering thought that for 2,000 years physicians -- including people like Benjamin Rush, a Philadelphia physician who had immense prestige and signed the Declaration of Independence  -- believed in the ancient Greek Galen's humoral theory of disease, which is complete bullshit, and treated people in ways that only helped to kill them. 

The randomized controlled trial is intended to eliminate possible sources of bias or faulty inference. Although the Scottish physician James Lind famously conducted a trial of citrus fruit to treat scurvy in 1747 (and he didn't actually draw the correct conclusions from it, contrary to legend), RCTs didn't become commonplace in medical research until the 1950s, and exacting standards for their conduct were not developed until the 1990s. Now, however, in order to get published in a prestigious journal your trial has to meet the standards. In principle, the same is true for approval of a new drug or a new indication from the FDA, although in fact FDA standards can often be a bit lax. We'll get to that later.

So I'll give you a list, generated from my own head, but I believe pretty accurate, of what you need to do to meet the conventional standards of rigor. Today we'll start with:

State your hypothesis in advance: Exactly what is the indication for your treatment, including the diagnostic criteria for the condition you propose to treat, the characteristics of the population you will study (e.g. age, sex, comorbid conditions or lack thereof), reasons for exclusion, how you will try to achieve diversity within the parameters you have set. What is the outcome you are hypothesizing, e.g. cure, amelioration of symptoms, extension of life; and how exactly will these be defined and measured.

This is very important because of the rules of inference. I'm won't get into the very deep weeds of current ideas and controversies in biostatistics, but whether you're doing Gaussian or Bayesian statistics, this is absolutely essential. Since it's still the norm in RCTs to report p values and confidence intervals, without explicit consideration of prior probabilities, I'll briefly explain in qualitative terms what these mean. 

Purely by chance, the people who get the intervention -- the treatment being tested -- might turn out to do better than the people in the control group. (We'll get to the control group, including the placebo concept, later. For now we're just talking about the math.) The difference between group A, the treatment group, and group B, the control group, in the proportion who improve, or the average amount of improvement, might be caused by the intervention, or it might just be a coincidence. (It could also be that group A and group B weren't really the same in the first place, but we'll ignore that for now.)

In a random sample drawn from the population, p is the probability that you would see the difference you actually observed between Group A and Group B if there is no real difference between them in the total population. In an experiment such as an RCT, it's the probability you would see the observed difference if the treatment actually has no effect.

The smaller p is, the more likely the difference actually exists. E.g., if p=.05 (i.e. 5%), in principle (not necessarily in reality) you would see this difference only 5% of the time, if it isn't real. (That's assuming you don't have other reasons for believing it is or isn't real, which is where we get into Bayesian inference, which we aren't doing right now.)

p depends on 3 quantities:

The size of the difference between the groups. The bigger it is, the smaller p will be.

The size of the groups (n). The bigger the groups, the smaller p will be.

How “spread out” the values of B are, and of other variables in the 2 groups that might matter.

Now I'm going to give you a couple of screen shots from my lecture about this, and go away until tomorrow.