In 2005, John Ioannidis published a review of 45 highlighted studies in major medical journals. He found that 24% were never replicated, 16% were contradicted by subsequent research, and another 16% were shown to have smaller effect sizes than originally reported. Less than half 44% were truly replicated.
Consider a situation in which 10% of all hypotheses are actually true. Now consider that most studies have a type 1 error rate (the probability of claiming an association when none exists [ie, a false positive]) of 5% and a type 2 error rate (the probability of claiming there is no association when one actually exists [ie, a false negative)] of 20%, which are the standard error rates presumed by most clinical trials. This allows us to create the following 2×2 table.
This would imply that of the 125 studies with a positive finding, only 80/125 or 64% are true. Therefore, one third of statistically significant findings are false positives purely by random chance. That assumes, of course, that there is no bias in the studies, which we will deal with presently.
Bias: Coffee, Cellphones, and Chocolate
Bias occurs when there is no real association between X and Y, but one is manufactured because of the way we conducted our study. Delgado-Rodriguez and Llorca identified 74 types of bias in their glossary of the most common biases, which can be broadly categorized into 2 main types: selection bias and information bias.
One classic example of selection bias occurred in 1981 with a NEJM study showing an association between coffee consumption and pancreatic cancer. The selection bias occurred when the controls were recruited for the study. The control group had a high incidence ofpeptic ulcer disease, and so as not to worsen their symptoms, they drank little coffee. Thus, the association between coffee and cancer was artificially created because the control group was fundamentally different from the general population in terms of their coffee consumption. When the study was repeated with proper controls, no effect was seen.