Genuine Very Large Effects in Trials Rare in MedicineLast Updated: October 23, 2012. Large treatment effects are most likely to be found in small studies, with the effect diminishing with additional trials, according to research published in the Oct. 24 issue of the Journal of the American Medical Association.
TUESDAY, Oct. 23 (HealthDay News) -- Large treatment effects are most likely to be found in small studies, with the effect diminishing with additional trials, according to research published in the Oct. 24 issue of the Journal of the American Medical Association.
Tiago V. Pereira, Ph.D., from the German Hospital Oswaldo Cruz in São Paulo, Brazil, and colleagues analyzed the Cochrane Database of Systematic Reviews (CDSR) and separated all binary-outcome forest plots with comparisons of interventions according to whether the first published trial, a subsequent trial, or no trial had a nominally statistically significant (P < 0.05) very large effect (odds ratio, ≥5).
The researchers found that 8,239 (9.7 percent) of the 85,002 forest plots (from 3,082 reviews) had a significant very large effect in the first published trial; 5,158 (6.1 percent) only after the first published trial; and the majority (71,605; 84.2 percent) had no trials with significant very large effects. Nominally significant, very large effects were typically observed in small trials, with a median of 18 events in first trials and 15 events in subsequent trials. Very large effects were less likely to address mortality and were more likely to address laboratory-defined efficacy. Subsequent published trials were just as likely for first trials with and for those without very large effects. In subsequent meta-analysis, 90 and 98 percent of the very large effects observed in first and subsequently published trials, respectively, became smaller. Only one intervention across the whole CDSR had large beneficial effects on mortality with no major concerns about the quality of evidence.
"Genuine very large effects with extensive support from substantial evidence appear to be rare in medicine and large benefits for mortality are almost entirely nonexistent," the authors write.