A cancer drug discovered in a humble lichen, and ready for testing in patients, might sound too good to be true. That's because it is. But more than a hundred lower-tier scientific journals accepted a fake, error-ridden cancer study for publication in a spoof organized by Science magazine.
The fake study points to a "Wild West" of pay-to-publish outlets feeding off lower tiers of the scientific enterprise by publishing studies without any appreciable scrutiny, say research ethics experts. (See "Who's Afraid of Peer Review?")
Some 8,250 "open-access" scientific journals worldwide are now listed in a directory supported by publishers. Unlike traditional science journals that charge for subscriptions or fees from those wishing to read their contents, open-access journals make research studies free to the public. In return, study authors pay up-front publishing costs if the paper is accepted for publication.
"From humble and idealistic beginnings a decade ago, open-access scientiﬁc journals have mushroomed into a global industry, driven by author publication fees," says journalist John Bohannon, writing in the Science magazine report of his survey-style spoof of review practices at such journals.
The cover of Science magazine.
Image courtesy Science/AAAS
"The goal was to create a credible but mundane scientiﬁc paper, one with such grave errors that a competent peer reviewer should easily identify it as ﬂawed and unpublishable," Bohannon says. Of 255 open-access journals that said they would review his study, 157 accepted the fake study for publication. "Acceptance was the norm, not the exception," he writes.
Science Spoofs Not New
Spoof studies intended to spotlight problems with individual journals and their review practices have made news before. New York University physicist Alan Sokal spoofed the cultural studies journal Social Text in 1996 with a crackpot physics treatise. And last month, Serbian academics spoofed a Romanian journal with a similarly ludicrous data-processing paper.
But the Bohannon study, which claimed to have discovered a cancer-fighting, lichen-derived drug ready for immediate testing on patients, represents a first systematic test of review practices, or their absence, across many journals at once, says research ethics expert Nicholas Steneck of the University of Michigan in Ann Arbor.
"The public wanted open access to scientific literature, and now they are getting it," Steneck says. "They now need to get over the idea that they can get all that information for free without someone doing the real hard work of reviewing papers."
Cancer Study Faked
The spoof study should have swiftly failed acceptance by "peer" reviewers at the science journals. Peer-reviewed science journals are supposed to publish papers only after a panel of two or three anonymous experts judge its acceptability for publication.
To test these reviewers, Bohannon submitted versions of his study to 304 open-access journals over the course of the year. The name of the cancer, lichen, and drug in each version was essentially picked out of a hat, along with an equally random, made-up name and institution for an author situated in an African capital.
Only 255 journals responded.
The journals tested were ones with relevant medical or biological titles for a cancer study, such as the European Journal of Chemistry or the International Journal of Cancer and Tumor (the latter edited by a "Grace Groovy," according to correspondence with Bohannon). Most appear to be headquartered in India and the United States.
The spoof study had at least three problems:
The study drug killed cancer cells with increasing doses, even though its data didn't show any such effect.
The drug killed cancer cells exposed to medical radiation with increasing effect, even though the study showed the cells weren't exposed to radiation.
The study author concluded the paper by promising to start treating people with the drug immediately, without further safety testing.
"If the scientiﬁc errors aren’t motivation enough to reject the paper, its apparent advocacy of bypassing clinical trials certainly should be," Bohannon writes.
Peer Review Missing
But in many cases, it appears the study wasn't peer-reviewed at all by the journals that responded to the spoof submission. Many of the reviews were just requests to format the study for publication. And of 106 journals that performed any review, 70 percent accepted the study.
"If a bogus paper is able to get through peer review, think about how many legitimate, but deeply flawed, papers must also get through," says Michael Eisen of the University of California, Berkeley, a founder of the Public Library of Science family of open-access journals.
One of those journals, PLOS One, was the only one of the 255 journals that received the spoof that noted its ethical flaws and "meticulously" reviewed the bogus study before rejecting it.
Although Eisen applauds the reviewers at PLOS One, he says of the spoof, "in all honesty, I think it is kind of a general indictment of peer review."
Instead he thinks scientists should move to a process of massive peer review after a study is released, a movement now pursued among physicists who widely upload draft versions of their papers to an online archive prior to journal review and publication.
The University of Michigan's Steneck, however, sees the spoof as exposing an onslaught of shoddy journals and bad studies cluttering the scientific literature. "I don't think it is an indictment of peer review as an idea, but rather shows how hard it is to get right," he says.
Cute, Clever Hoax
"These aren't really science journals pointed out by this very cute and clever hoax; they are more check-cashing operations," says Stanford University study-design expert John Ioannidis. "I don't think that open access is the problem either. I think you would see the same problem with the lower tier of traditional peer-review journals as well."
In fact, the hoax may show that scientists are caught in a double bind when it comes to publishing studies, Ioannidis says. His own research shows that the most prestigious science journals, ones important in hiring decisions for researchers, demand outsize effects from the studies they publish. That may lead to study authors subconsciously biasing their statistics to make a startling discovery worthy of a big-name journal.
Science, which has defended traditional publishing models against open-access efforts, is not without its own blemishes. The journal published a 2004 paper claiming the first cloning of human stem cells that turned out to be faked. And in 2010, it published a study of "arseniclife" microbes that researchers initially believed were using poisonous arsenic in their metabolism.
The paper generated a firestorm of criticism, and was largely refuted in 2012. Later investigation revealed that the paper's original peer reviewers had loved the 2010 paper and largely missed its flaws.
At the same time, a study that is honestly conducted, but doesn't offer headline-making revelations, may end up buried in journals with bad reputations due to the same kinds of shoddy peer-review practices pointed out by the hoax. Then the researchers don't receive grants or jobs offered to others.
The post-publication review advocated by Eisen is no panacea, Ioannidis argues, because researchers aren't rewarded for such reviews and therefore won't do them.
"I don't think there is any one solution," Ioannidis says. "The scientific community has to come up with solutions that rewards good studies. It's an ongoing fight."
Follow Dan Vergano on Twitter.