A tenet of becoming a scientist and earning a doctoral degree (Ph.D) in hard sciences is to be able to develop a critical thinking and skepticism over scientific findings. We learn how to not accept scientific claims as facts “just because someone said it” and learn to fact-check such claims by analyzing the data and see if the data are robust enough to support the claims or if they are simply inaccurate, non-conclusive or worse…..simply fabricated.
Data fabrication, adulteration, plagiarism and manipulation is unfortunately present in science. This is why peer-review is playing an important role in filtering out studies that are robust enough from studies that are murky or questionable enough. That latter is usually what I refer as “junk science”, scientific studies that are not standing to scientific rigor and should not have been reaching the publication stage. The peer-review process is not the most optimal one. If you want an analogy, consider peer-review as the wooden fence lining your backyard: it will not stop a burglar to climb over it but it will stop trespassers and marauders to come too close from your home.
Yesterday, I woke up straight in a middle of a Twitter firestorm about the retraction of a paper. Seeing papers retracted is not uncommon, there is even a website for that called “Retraction Watch” that track studies retracted by scientific journals. But yesterday it was such a bad paper that yesterday’s Dr. Derek Lowe that hold a PhD in Chemistry from Duke University had a fiery blog post about it (the access was denied soon after I read it but seems to be online again this morning) named “Crap, courtesy of a major scientific publisher“.
The problem was not facing a junk scientific paper, there are plenty around nowadays since Open-Access journals started to kick inside the world of scientific publishing and thanks to predatory publishers (I will talk about it later). The problem was the journal that has such junk paper published: Scientific Reports (SciRep, from Nature Publishing Group) (Disclosure: I have co-authored a paper published in Scientific Reports). Scientific Reports is the response of NPG to open-access (OA) journals such as Public Library of Sciences (PLoS). Because it is coming from NPG, everyone is expecting to attain a certain rigor for peer-review (Nature is one of the hardest journal to get your scientific study published). I always joke around that it is so demanding that we are facing “icebergs” papers, studies with five main figures and 50 supplemental figures that are only accessed online.
Using this debacle, I thought it would serve well as a poster child to expose some scientific fraud and provide some tips to distinguish good papers from bad papers.
1. Scientific Publishing 101: Peer-review, open-access, predatory journals and publication fees.
Publications in peer-reviewed journals is the bread and butter of academic researchers. It is as vital for a researcher as a credit report is for anyone living in the US. Two criteria matters in big time decisions such as finding a job or earning tenure in an University: how many papers you have your name affiliated to and which journals. These metrics are very important, especially with the latter driven by the impact factor (IF). The IF is the equivalent of a BBB rating: the higher, the better. Two giants dominates the field: Nature (from NPG, IF ~42) and Sciences (From the American Association of Advancement of Sciences or AAAS, IF~32).
It is so important that the number of papers coming from these two journals conditions the odd of a researcher to get a job in prestigious institutions such as Harvard, MIT, Stanford or UC Berkeley.
Papers are part of a particular cycle that I don’t know if we should call it vicious or virtuous.
1. To publish papers you need data.
2. To obtain data you need research funds.
3. To obtain funds, you need to write winning grants.
4. To have a grant having a chance to get funded you need papers
5. Repeat step 1.
All peer-reviewed journals follow the same procedure: I submit my draft manuscript that I consider solid enough for peer-review to a journal. The editor-in-chief (usually a well seasoned scientist) decides using both an objective and subjective point-of-view what to do with it: the objective one is if the paper fits into the editorial policy (for instance publishing my work on BBB into a plant biology journal is fairly no-sense) and the subjective one is if the paper is “attractive” enough for the editor-in-chief or not. If not, it will toss it fast. If it is, it will proceed and pick 2 reviewers that have more-or-less the adequate expertise. Such reviewers are kept anonymous for most journals with very few exceptions. Reviewers have a moral obligation to keep their review objective and fair. Sometimes they do, sometimes they don’t. You can easily imagine that if reviewer X is a scientist working on the same topic than me, that reviewer feels the risk of being scooped and therefore will work hard to find flaws to get my paper rejected and work hard to scoop me.
At the end, 2 or 3 reviewers will provide their comments and feedback giving the editor-in-chief the decision to accept or reject your paper. Once rejected, you have no other choice to move on to another journal and restart the same game.
The competition is fierce, with only less than 1% of papers submitted to the top 2 journals will end up being published. This also raised a race-arms to publish only papers that are groundbreaking science in big way and usually can shake up an entire field and a fierce competition for getting published. This is what I call the “wow factor”. But thats only a small problem that raised to OA journals and sometimes it can backfire due to scientific misconducts (examples: Two stem cells papers retracted because of data fabrications such as the Hwang paper about the cloning of hESCs from human oocytes published in Science in 2005 and the STAP “pickled stem cells” published in Nature in 2014)
The main problem is that once accepted, this study will suffer from a double-jeopardy in terms of publication fees: the authors have to pay publication fees to get the accepted paper published (usually goes from ~$1000-3000 per study). Once published, you secede the copyrights to the publisher, this publisher will ask anyone wanting to read the paper to pay for its accession (~$50 per study). This second fee hinders how many scientists can read your study, limit access of scientists from developing countries to these studies and also limit the number of studies that will cite your study. Certain public health agencies like the National Institute of Health (NIH) responded to such issue by asking any studies funded with $$$ from NIH grant to be available free 12 months after publication through their “Pubmed Central” portal.
OA journals were born from these concerns. The OA publication follows the same protocol than regular published journals except for two aspect: they will accept any papers based on the robustness of the data rather than the novelty or “wow factor”. If your paper is not as exciting and breaking ground as higher journals but it solid and can provide the field with small but solid information, it will get accepted.
once published such studies are made open-access. Anyone can read them freely. This is because once accepted, the journal recover the costs by asking higher publication fees (~$2500-$3500) from the author of the study.
This is an interesting alternative publication method, however it also opened a new wild wild West in academic publishing. Like any good Western movies, you have wandering snake oil sellers and in academic publishing these snake oil sellers are represented by predatory journals and publishers. These publishers found some easy preys to feed on: academic scientists with studies that are so poorly designed or just simply fraud and could not pass the peer-review filters. As long as you give them money in form of publishing fees, they will publish your paper through an expedited review. This lead in recent years in the appearance of “junk papers” that are little or no scientific merit and yet get the right to get cited. This lead to a hall of shame through the Beall’s list of predatory publishers providing a database of journals and publishers with suspicious or demonstrated predatory practices. There is even one publisher found with a mailing address pointing….to a suburban house. How serious this can be? This is what feed most of the pseudoscience outside. Anti-GMO, anti-vaccines, chiropractic, naturopaths and homeopaths are all relying heavily on such “junk science” to provide a scientific rationale to their claims.
2. What was about this paper that made such firestorm and retraction by Scientific Reports?
The paper in question is titled “Novel piperazine core compound induces death in human liver cancer cells: possible pharmacological properties” by Samie and colleagues from the University of Malaya, Kuala Lumpur, Malaysia and published in SciRep last April. As today, the paper was not available through SciRep yesterday and seems back online today. I guess the academic firestorm put the server into severe stress.
I will go step by step and explained in comments what is wrong with this paper (see figures below).
After reviewing the paper, you can notice how many flaws and blatant data manipulation was mined inside this paper. Peer-review cannot be a fool-proof system, as some very elaborated data fabrication may go unnoticed even by the most seasoned reviewer. I am not surprised either to see such junk study to made it through publication, if it was coming from a predatory journal. But seeing such paper coming from Scientific Reports being unnoticed although a fairly reasonable turnover (it was received in October 1 2015, accepted March 23 2016 suggesting at least one round of review and the submission of a revised form) is disturbing. Scientific Reports editorial has to consider what went wrong and investigate the review history of this paper but also whether reviewers assigned to review this study displayed the expertise needed and the objectivity to do it.
At that time, I would not be reviewer 1 or 2 (even 3) that reviewed this junk paper. Garbage in, garbage out.