Here's some boring wonkery . . .

. . . to give you some relief from all the April foolishness.

Xin Sun and a cast of thousands reviewed 469 clinical trials reported in both high impact and lower impact journals. Okay, here's the wonkish part. It's actually kind of important and profound, in ways that go well beyond the current context. So I'm using this as an occasion to talk about a big issue.

It is conventional in clinical science to consider results to be valid when they have an associated "p value" of .05 or less, the jargon for which is "statistical significance." The term is misleading, in both directions. Findings with p values <.05 aren't necessarily "significant" in the sense that they actually matter; and "statistical insignificance" doesn't mean that an association doesn't exist, or is trivial, it just means you haven't established it to the satisfaction of an arbitrary convention. Most people misunderstand what a p value really means. It's usually interpreted as the probability that your result is not real, but arose just by chance. If the p value is less than .05, then you would say that there's a 95% chance your result is real. But that isn't what a p value means. It is the fallacy of the transposed conditional. The p value is actually the probability of getting your results given the hypothesis, in this case the null hypothesis that there is no real association. But, obviously if you make a lot of comparisons you'll see a lot of low p values that are spurious. The true probability that the drug is effective depends not only on the results of your study, but on the probability that the drug is effective based on what you knew before you did the study. If it is extremely unlikely -- say because there is no biological plausibility -- then a p value <.05 doesn't make it likely after all. It's probably just a coincidence. That's why, in order for p values to make sense, they have to be used to test plausible hypotheses that are specified before you undertake your trial. You always have a lot of information about clinical trial subjects, so even if the results of the trial are negative overall, you can comb through your data and look for "significant" results for some group that you didn't define ahead of time - say, women, or people of a certain age range, or people with a particular severity of disease, whatever you like. Chances are, you'll find some. And guess what -- people publish results of that kind. But they're basically BS. They might constitute hypotheses worth testing, but they aren't findings.

What Sun and the gang found is that when trials are funded by drug manufacturers, that did not have significant results for the pre-specified primary outcomes, they are much more likely to report sub-group analyses than trials that aren't funded by the manufacturer. What this means is that they are trying to find a way to sell their product, even though it apparently doesn't work, by inventing some specific category of person for whom they claim it does work, when in fact they don't have credible evidence for that conclusion.

Whoo. Sorry about that. Anyway, we need to do things differently. Industry funding of clinical trials must be totally isolated from the conduct and reporting of the trials, not just because of this finding but because of a vast record of deception, spin and scandals going back decades. There are probably several plausible ways to organize medical research to bring this about. We could establish a government sponsored institute to which the drug companies would transfer the funds for clinical trials. The institute would then accept competitive proposals from researchers, and award the funds to the most credible investigators, who would carry out the trial with no contact whatsoever from the interested parties.

Doesn't look exactly like the Free Market™ of conservative corporatist fantasy. But it would be a better world.