One year after a damning review suggested that many published clinical trials contain statistical errors, The
New England Journal of Medicine (NEJM) today is correcting five of the papers fingered and retracting and republishing a sixth, about whether a Mediterranean diet helps prevent heart disease. (Spoiler alert: It still does, according to the new version of the paper.) Despite errors missed until now, in many ways the journal system worked as intended, with
NEJM launching an inquiry within days of the accusations.
The journal's unusual move was prompted by a controversial analysis published in June 2017. Writing in
Anaesthesia, where he is also an editor, anesthesiologist John Carlisle of Torbay Hospital in Torquay, U.K., took a statistical deep dive into 5087 randomized, controlled trials.
With the help of a computer program, Carlisle looked for a specific type of anomaly: nonrandom assignment of volunteers to different treatments, when the trial had claimed the assignments were random. This
can skew a trial's results-for example, if many more elderly people are assigned to a control group while younger ones get an experimental treatment, the new drug may look like it has fewer side effects because the people getting it are healthier.
Across eight journals, Carlisle analyzed how certain features of the volunteers-such as their height, weight, and age-were spread across the treatments tested. If he didn't see certain patterns-if the distribution was too perfect, or too far off-he suspected the assignments were not truly random, whether because of scientific misconduct or honest error.
Roughly 2% of the papers he ran through his program fell into this questionable category.
Comment: See also: