In January, a review paper1 about ways to detect human illnesses by examining the eye appeared in a conference proceedings published by the Institute of Electrical and Electronics Engineers (IEEE) in New York City. But neither its authors nor its editors noticed that 60% of the paper it cited had already been retracted.
The case is one of the most extreme spotted by a giant project to find papers whose results might be in question because they cite retracted or problematic research. The project’s creator, computer scientist Guillaume Cabanac at the University of Toulouse in France, shared his data with Nature’s news team, which analysed it to find the papers that most heavily cite retracted work yet haven’t themselves been withdrawn (see ‘Retracted references’).
“We are not accusing anybody of doing something wrong. We are just observing that in some bibliographies, the references have been retracted or withdrawn, meaning that the paper may be unreliable,” Cabanac says. He calls his tool a Feet of Clay Detector, referring to an analogy, originally from the Bible, about statues or edifices that collapse because of their weak clay foundations.
The IEEE paper is the second-highest on the list assembled by Nature, with 18 of the 30 studies it cites withdrawn. Its authors didn’t respond to requests for comment, but IEEE integrity director Luigi Longobardi says that the publisher didn’t know about the issue until Nature asked, and that it is investigating.
Cabanac, a research-integrity sleuth, has already created software to flag thousands of problematic papers in the literature for issues such as computer-written text or disguised plagiarism. He hopes that his latest detector, which he has been developing over the past two years and describes this week in a Comment article in Nature, will provide another way to stop bad research propagating through the scientific literature — some of it fake work created by ‘papermill’ firms.
Cabanac lists the detector’s findings on his website, but elsewhere online — on the paper-review site PubPeer and on social media — he has explicitly flagged more than 1,700 papers that caught his eye because of their reliance on retracted work. Some authors have thanked Cabanac for alerting them to problems in their references. Others argue that it’s unfair to effectively cast aspersions on their work because of retractions made after publication that, they say, don’t affect their paper.
Retracted references don’t definitively show that a paper is problematic, notes Tamara Welschot, part of the research-integrity team at Springer Nature in Dordrecht, the Netherlands, but they are a useful sign that a paper might benefit from further scrutiny. (Nature’s news team is independent of its publisher, Springer Nature.)
Some researchers argue that retraction of references in a narrative review — which describes the state of research in a field — doesn’t necessarily invalidate the original paper. But when studies assessed by a systematic review or meta-analysis are withdrawn, the results of that review should always be recalculated to keep the literature up to date, says epidemiologist Isabelle Boutron at Paris City University.
