[Junk Sciences] About that scientific paper retracted from Scientific Reports yesterday and the limits of peer-review

A tenet of becoming a scientist and earning a doctoral degree (Ph.D) in hard sciences is to be able to develop a critical thinking and skepticism over scientific findings. We learn how to not accept scientific claims as facts “just because someone said it” and learn to fact-check such claims by analyzing the data and see if the data are robust enough to support the claims or if they are simply inaccurate, non-conclusive or worse…..simply fabricated.
Data fabrication, adulteration, plagiarism and manipulation is unfortunately present in science. This is why peer-review is playing an important role in filtering out studies that are robust enough from studies that are murky or questionable enough. That latter is usually what I refer as “junk science”, scientific studies that are not standing to scientific rigor and should not have been reaching the publication stage. The peer-review process is not the most optimal one. If you want an analogy, consider peer-review as the wooden fence lining your backyard: it will not stop a burglar to climb over it but it will stop trespassers and marauders to come too close from your home.
Yesterday, I woke up straight in a middle of a Twitter firestorm about the retraction of a paper. Seeing papers retracted is not uncommon, there is even a website for that called “Retraction Watch” that track studies retracted by scientific journals. But yesterday it was such a bad paper that yesterday’s Dr. Derek Lowe that hold a PhD in Chemistry from Duke University had a fiery blog post about it (the access was denied soon after I read it but seems to be online again this morning) named “Crap, courtesy of a major scientific publisher“.
The problem was not facing a junk scientific paper, there are plenty around nowadays since Open-Access journals started to kick inside the world of scientific publishing and thanks to predatory publishers (I will talk about it later). The problem was the journal that has such junk paper published: Scientific Reports (SciRep, from Nature Publishing Group) (Disclosure: I have co-authored a paper published in Scientific Reports). Scientific Reports is the response of NPG to open-access (OA) journals such as Public Library of Sciences (PLoS). Because it is coming from NPG, everyone is expecting to attain a certain rigor for peer-review (Nature is one of the hardest journal to get your scientific study published). I always joke around that it is so demanding that we are facing “icebergs” papers, studies with five main figures and 50 supplemental figures that are only accessed online.
Using this debacle, I thought it would serve well as a poster child to expose some scientific fraud and provide some tips to distinguish good papers from bad papers.

1. Scientific Publishing 101: Peer-review, open-access, predatory journals and publication fees.

Publications in peer-reviewed journals is the bread and butter of academic researchers. It is as vital for a researcher as a credit report is for anyone living in the US. Two criteria matters in big time decisions such as finding a job or earning tenure in an University: how many papers you have your name affiliated to and which journals. These metrics are very important, especially with the latter driven by the impact factor (IF). The IF is the equivalent of a BBB rating: the higher, the better. Two giants dominates the field: Nature (from NPG, IF ~42) and Sciences (From the American Association of Advancement of Sciences or AAAS, IF~32).
It is so important that the number of papers coming from these two journals conditions the odd of a researcher to get a job in prestigious institutions such as Harvard, MIT, Stanford or UC Berkeley.
Papers are part of a particular cycle that I don’t know if we should call it vicious or virtuous.
1. To publish papers you need data.
2. To obtain data you need research funds.
3. To obtain funds, you need to write winning grants.
4. To have a grant having a chance to get funded you need papers
5. Repeat step 1.
All peer-reviewed journals follow the same procedure: I submit my draft manuscript that I consider solid enough for peer-review to a journal. The editor-in-chief (usually a well seasoned scientist) decides using both an objective and subjective point-of-view what to do with it: the objective one is if the paper fits into the editorial policy (for instance publishing my work on BBB into a plant biology journal is fairly no-sense) and the subjective one is if the paper is “attractive” enough for the editor-in-chief or not. If not, it will toss it fast. If it is, it will proceed and pick 2 reviewers that have more-or-less the adequate expertise. Such reviewers are kept anonymous for most journals with very few exceptions. Reviewers have a moral obligation to keep their review objective and fair. Sometimes they do, sometimes they don’t. You can easily imagine that if reviewer X is a scientist working on the same topic than me, that reviewer feels the risk of being scooped and therefore will work hard to find flaws to get my paper rejected and work hard to scoop me.
At the end, 2 or 3 reviewers will provide their comments and feedback giving the editor-in-chief the decision to accept or reject your paper. Once rejected, you have no other choice to move on to another journal and restart the same game.
The competition is fierce, with only less than 1% of papers submitted to the top 2 journals will end up being published. This also raised a race-arms to publish only papers that are groundbreaking science in big way and usually can shake up an entire field and a fierce competition for getting published. This is what I call the “wow factor”. But thats only a small problem that raised to OA journals and sometimes it can backfire due to scientific misconducts (examples: Two stem cells papers retracted because of data fabrications such as the Hwang paper about the cloning of hESCs from human oocytes published in Science in 2005 and the STAP “pickled stem cells” published in Nature in 2014)
The main problem is that once accepted, this study will suffer from a double-jeopardy in terms of publication fees: the authors have to pay publication fees to get the accepted paper published (usually goes from ~$1000-3000 per study). Once published,  you secede the copyrights to the publisher, this publisher will ask anyone wanting to read the paper to pay for its accession (~$50 per study). This second fee hinders how many scientists can read your study, limit access of scientists from developing countries to these studies and also limit the number of studies that will cite your study. Certain public health agencies like the National Institute of Health (NIH) responded to such issue by asking any studies funded with $$$ from NIH grant to be available free 12 months after publication through their “Pubmed Central” portal.

OA journals were born from these concerns. The OA publication follows the same protocol than regular published journals except for two aspect: they will accept any papers based on the robustness of the data rather than the novelty or “wow factor”. If your paper is not as exciting and breaking ground as higher journals but it solid and can provide the field with small but solid information, it will get accepted.
once published such studies are made open-access. Anyone can read them freely. This is because once accepted, the journal recover the costs by asking higher publication fees (~$2500-$3500) from the author of the study.
This is an interesting alternative publication method, however it also opened a new wild wild West in academic publishing. Like any good Western movies, you have wandering snake oil sellers and in academic publishing these snake oil sellers are represented by predatory journals and publishers. These publishers found some easy preys to feed on: academic scientists with studies that are so poorly designed or just simply fraud and could not pass the peer-review filters. As long as you give them money in form of publishing fees, they will publish your paper through an expedited review. This lead in recent years in the appearance of “junk papers” that are little or no scientific merit and yet get the right to get cited. This lead to a hall of shame through the Beall’s list of predatory publishers providing a database of journals and publishers with suspicious or demonstrated predatory practices. There is even one publisher found with a mailing address pointing….to a suburban house. How serious this can be? This is what feed most of the pseudoscience outside. Anti-GMO, anti-vaccines, chiropractic, naturopaths and homeopaths are all relying heavily on such “junk science” to provide a scientific rationale to their claims.

2. What was about this paper that made such firestorm and retraction by Scientific Reports?
The paper in question is titled “Novel piperazine core compound induces death in human liver cancer cells: possible pharmacological properties” by Samie and colleagues from the University of Malaya, Kuala Lumpur, Malaysia and published in SciRep last April. As today, the paper was not available through SciRep yesterday and seems back online today. I guess the academic firestorm put the server into severe stress.
I will go step by step and explained in comments what is wrong with this paper (see figures below).

Slide1

Slide2

Slide3

Slide4

Slide5

3. Conclusions

After reviewing the paper, you can notice how many flaws and blatant data manipulation was mined inside this paper. Peer-review cannot be a fool-proof system, as some very elaborated data fabrication may go unnoticed even by the most seasoned reviewer. I am not surprised either to see such junk study to made it through publication, if it was coming from a predatory journal. But seeing such paper coming from Scientific Reports being unnoticed although a fairly reasonable turnover (it was received in October 1 2015, accepted March 23 2016 suggesting at least one round of review and the submission of a revised form) is disturbing. Scientific Reports editorial has to consider what went wrong and investigate the review history of this paper but also whether reviewers assigned to review this study displayed the expertise needed and the objectivity to do it.
At that time, I would not be reviewer 1 or 2 (even 3) that reviewed this junk paper. Garbage in, garbage out.

Advertisements

The Supreme Court rules out iPSCs cell patent

If you are in the stem cell field, you know that one of the problem actually mining the field like a minefield is the intellectual properties and patents. Especially here in the US, in which both human embryonic (hESCs) and induced pluripotent stem cells (iPSCs) are subject to patents and IP through the Wisconsin Alumni Research Foundation (WARF).

That have created some stirs with a series of court decisions to define if WARF have the rights to patent these stem cells and hold exclusive aspect of it. This of course ended up at the Supreme Court to get the final word on it.

Well it seems the Supreme Court get its final word to end up its imbroglio, according to U-T San Diego and reported by the Genetics Policy Institute:http://www.utsandiego.com/news/2015/feb/24/supreme-court-rejects-warf-patent-case/

And the final word was a “Yeah!” for the embryonic stem cells (in other words WARF keep their exclusive patents and licensing on them) but a “Nay!” for the iPSCs, in other means anyone could derive the iPSCs without having to request a license to do so from WARF.

Now let’s see how things deconvolute and how the fallout of this news will shake the stem cell field but I think an old post from Pr. Paul Knoepfler worth to be updated in the light of the newest development, here is the original link to the article: Putting the IP in iPS cells: patent war looming? | Knoepfler Lab Stem Cell Blog.

[Sciences] The National Academies call for a postdoc overhaul

postdoc_pay

It is not because I am now a faculty member that I do not care about the fate of my former postdoc fellows. Instead, I try to bring awareness and mentoring to my grads and postdoc in my institute about their professional careers and to take their career as importantly as their bench work.

The National Academies just published a report on the postdoc situation. And it is really a call for action or maybe a recall for action, as the previous one was published in…..2000 (that is a century in terms of scientific research). It simply refocus on one obvious thing that many postdoctoral associations and raw numbers have been saying: as the academic career is now the alternative career after a PhD, so do the postdoc should be. I have and many others have experienced the postdoc limbo in which you try to bridge the gap between your position terminated (as the grant ends up) and securing a new (and more stable position).

I agree that postdocs should have a five-year term and for international postdocs like I was it was somehow already the case. Many international postdocs are here on a H1-B worker status, that gives you 5 years, with possibility to an extra year (6 years). But at the end of this period, you will have to leave the United States for one year before re-applying for an H1-B. It is not easy if you are single or married, but if you have a family with kids in the public school system, it becomes a nightmare. Therefore international postdocs have to think now twice. If you are thinking for applying for a green card, the sponsored option that is available with a faculty position is vanishing quickly (as faculty positions are becoming more and more scarce) and industry jobs are simply a no-no (as other private companies, they have to face a yearly quota of H1-B visas and the queue is extremely long). But then, how to deal then with the status of Research Assistant Professor (RAPs)? This kind of non-tenure faculty position is kind of a purgatory for senior postdoc and looks a lot like a sort of adjunct professor position (more fitted to scientist). You lead a research project, write grants but your are on soft money. That means if you run dry on grants, you are out.

I also agree that postdoc mentors have to be hold accountable for their postdoc career. I strongly advice and recommend my students, in addition to getting results, to also think in long term their goals and expectations, even if it means they may not pursue a scientific (or academic) career after the PhD. I just feel it is criminal to hire a graduate student or postdoc to have the research done and dispose them as a piece of old furniture once the grants runs out or when the clock rings. So far, little have been done to hold PIs accountable for such mentoring. Thank God, I have a AAA mentoring during my graduate and postdoc mentoring but other fellows were less lucky than me.

Now it is coming the problematic point. Salary! Right now the NRSA postdoc salary is fixed $42’000. But here is the trick, you don’t have yet the fringes and benefits associated. Thats roughly 33% of the gross salary. So if I have a $75’000/year startup package and decide to hire a postdoc, I will likely have to carve out my grant $56’000 just for the postdoc and thats leave me with $19’000 to pay for supplies and research equipment. The problem that you encounter is when you are a young PI, you have to hire a postdoc to help to run the lab (at a graduate student is never ready to take a project by day 1) to generate preliminary data to apply for early careers (R03 NIH) because nobody will give you money if you cannot show you are capable to have the research proposed up and running. This is also where you can assess if you have negotiated well your offer letter (and nobody teaches you how to negotiate). Once negotiated, it is almost impossible to discuss the terms. Now if as they request $50’000, thats brings a whooping $66’000 of your startup gone for postdoc salary. Good luck to keep your lab running on $9’000 for a whole fiscal year! That means we gonna have to rethink the current funding mechanism (grants for trainings, grant for consumables) and also rethink about if these fringes and benefits should be now included in the indirect costs. But yes, that means this solution is not viable for young investigators as it would simply kill us in the egg.

This is a critical time. Budget has been shrinking, the “ol’ glory days” of postdocs are gone and we have to find a viable alternative to have our research lab innovation wheel keep spinning. It is time that faculty, postdocs, funding agencies and institutions sit together and discuss on a consensus that provide a quality training, a cost-effective and a viable environment for young academics.

You can judge by yourself:

Report Urges Significant Reforms to Improve the Training and Salary of Postdoctoral Researchers 

WASHINGTON — A new report from the National Academy of Sciences, National Academy of Engineering, and Institute of Medicine urges significant changes to improve the postdoctoral training system in the United States. The postdoctoral experience should be refocused to have training and mentoring at its center, the report says. In addition, the salaries of postdoctoral researchers should be increased to reflect more accurately the value of their training and contribution to research.

The report also recommends that graduate students avoid viewing postdoctoral positions as the default next step, given that growth in the number of postdoctoral researchers far exceeds growth in the number of tenure-track jobs to which many of these researchers aspire. Instead, with the assistance of their institutions, graduate students should consider a broad range of scientific career paths, said the committee that wrote the report.

“The demand for junior research workers has boomed in recent decades, but the number of research faculty positions into which the junior researchers might hope to move has not kept pace,” said committee chair Gregory Petsko, Arthur J. Mahon Professor of Neurology and Neuroscience at Weill Cornell Medical College. “The result is a system that has created expectations for academic career advancement that in many – perhaps most — cases cannot be met.”

Concern about the postdoctoral training system has been gnawing at the research community for decades, the report observes. The National Academies produced a report in 2000 that called for reforms to the system.  Some progress has been made since then, the new report notes. For example, many universities have created offices of postdoctoral affairs to provide better services to postdoctoral researchers, and postdoctoral researchers created the National Postdoctoral Association to provide a forum and unified voice for themselves.

Other aspects of postdoctoral training have seen little change. There is no convincing evidence that most postdoctoral researchers are receiving adequate mentoring, and little evidence that universities and mentors are providing adequate information about other types of careers. Salaries, always relatively low, have failed to even keep pace with inflation.

Meanwhile, the percentage of Ph.D.s who pursue postdoctoral training has been growing steadily and broadening from the biomedical and physical sciences to engineering and the social sciences. In the United States, an estimated 60,000 to 100,000 postdoctoral researchers now work in various research fields.

Although the data are not definitive, the average length of time researchers spend in postdoctoral positions seems to be increasing, the report says. Sources of funding have changed as well. The number of postdoctoral fellowships and trainee positions — which provide postdoctoral researchers relative autonomy and recognition — has remained nearly constant for decades, while the number of postdoctoral researchers hired as part of research grants or supported by nonfederal sources has grown dramatically.

Postdoctoral research positions are intended to give young scientists advanced research training with a fixed term of appointment. In reality, the practice of employing postdocs as long-term researchers, with little mentoring and little hope of moving into a career that requires advanced research training, unfortunately seems to be becoming more common, said the committee.

To address problems in the postdoctoral training system, the committee developed recommendations for best practices covering five areas — period of service, title and role, career development, compensation and benefits, and mentoring – along with a sixth recommendation on data collection.

Period of service. Postdoctoral appointments for a researcher should total no more than five years in duration, barring extraordinary circumstances (e.g. family leave, illness). This maximum term should include cumulative postdoctoral research experience. Host institutions should maintain a record of how long a postdoctoral researcher remains in a position and provide that information to funding agencies as part of grant proposals. And to facilitate tracking of postdoctoral researchers, funding agencies could assign each postdoctoral researcher an identifier and keep record of the total length of time any given individual is holding such a position.

Title and role.  In many instances, positions currently occupied by postdoctoral researchers are more appropriately filled by permanent staff scientists (e.g., technicians, research assistant professors, staff scientists, laboratory managers). The title of “postdoctoral researcher” should be applied only to those people who are receiving significant advanced training in research. When the appointment period is completed, the postdoctoral researchers should move on to a permanent position elsewhere or be transitioned internally to a staff position with a different and appropriate designation and salary. Funding agencies should have a consistent designation for “postdoctoral researchers” and require evidence that advanced research training is a component of the postdoctoral experience.  Host institutions should create or identify professional positions for individuals who are conducting research but not receiving training, and they should receive appropriate remuneration, benefits, and privileges.

Career development. Host institutions and mentors should, beginning at the first year of graduate school, make students aware of the wide variety of career paths available for Ph.D. recipients, and explain that postdoctoral positions are intended only for those seeking advanced research training. The postdoctoral position should not be viewed by graduate students or principal investigators as the default step after the completion of doctoral training.

“Training for the Ph.D. degree is an ideal preparation for many different careers, and this recommendation is not meant to suggest that the number of graduate students in the physical, life and social sciences is too great,” said Petsko. “It is vital, however, that information about the full range of such career opportunities be available to all graduate students, and that the institutional culture not imply that careers outside a traditional academic track are in any sense inferior options.”

Compensation and benefits of employment. Current postdoctoral salaries are low, the report says. The study committee considered five different approaches for determining an appropriate minimum salary, and all of them suggest an amount of $50,000 or more. In addition, data reveal that the starting salary for NIH’s National Research Service Award (NRSA) postdoctoral award – currently set at $42,000 for 2014 — has become the de facto standard for many disciplines and at many universities. The NIH should raise the NRSA postdoctoral starting salary to $50,000 (2014 dollars) and adjust it annually for inflation. Postdoctoral salaries should be appropriately higher where regional cost of living, disciplinary norms, and institutional or sector salary scales dictate higher salaries. (Two committee members did not support the recommendation for a prescriptive salary standard; see footnote on p. 6 of report.)

To implement this recommendation, federal agencies should require host institutions to provide documentation of the salary a postdoctoral researcher will receive with all grant proposals. Professional societies should collect data on salaries for all positions and make these publicly available.

Mentoring. The postdoctoral experience should have training and mentoring at its center, the report stresses. Host institutions should create provisions that encourage postdoctoral scholars to seek advice, either formally or informally, from multiple advisers, in addition to their immediate supervisor. Host institutions and funding agencies should take responsibility for ensuring the quality of mentoring through training programs for the mentors and evaluation of their performance. Funding agencies should identify better ways of evaluating or rewarding mentoring as an essential component of research. Professional societies are in an ideal position to provide additional mentors to supplement those at host institutions.

Data collection. Current data on the postdoctoral population, in terms of demographics, career aspirations, and career outcomes are neither adequate nor timely. Every institution that employs postdoctoral researchers should collect data on the number of currently employed postdoctoral researchers and where they go after completing their research training, and make this information publicly available. The National Science Foundation should serve as the primary curator for establishing and updating a database system that tracks postdoctoral researchers, including non-academic and foreign-trained postdoctoral researchers.

The study was sponsored by the Presidents’ Committee of the National Academy of Sciences, National Academy of Engineering, and Institute of Medicine; the Alfred P. Sloan Foundation; and the Burroughs Wellcome Fund. The National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council make up the National Academies.  They are private, independent nonprofit institutions that provide science, technology, and health policy advice under a congressional charter granted to NAS in 1863.  The National Research Council is the principal operating arm of the National Academy of Sciences and the National Academy of Engineering.  For more information, visit http://national-academies.org.  A committee roster follows.

Contacts:

Sara Frueh, Media Relations Officer

Christina Anderson, Media Relations Assistant

Office of News and Public Information

202-334-2138; e-mail news@nas.edu

http://national-academies.com/newsroom

Twitter: @NAS_news and @NASciences

RSS feed: http://www.nationalacademies.org/rss/index.html

Flickr: http://www.flickr.com/photos/nationalacademyofsciences/sets

Pre-publication copies of The Postdoctoral Experience Revisited are available from the National Academies Press on the Internet at http://www.nap.edu or by calling 202-334-3313 or 1-800-624-6242. Reporters may obtain a copy from the Office of News and Public Information (contacts listed above).

NATIONAL ACADEMY OF SCIENCES

NATIONAL ACADEMY OF ENGINEERING

INSTITUTE OF MEDICINE

Committee on Science, Engineering, and Public Policy

Committee to Review the State of Postdoctoral Experience in Scientists and Engineers

Gregory A. Petsko1,2 (chair)

Arthur J. Mahon Professor of Neurology and Neuroscience, and

Director

Appel Alzheimer’s Disease Research Institute

Weill Cornell Medical College; and

Gyula and Katica Tauber Professor of Biochemistry and Chemistry Emeritus

Brandeis University

New York City

Sibby Anderson-Thompkins

Director of Postdoctoral Affairs

Office of the Vice Chancellor for Research

University of North Carolina

Chapel Hill

  1. Russell Bernard2

Professor Emeritus

Department of Anthropology

University of Florida

Gainesville

Carol Greider1,2

Daniel Nathans Professor and Director

Department of Molecular Biology and Genetics

Johns Hopkins University School of Medicine

Baltimore

James Plummer3

Frederick Emmons Terman Dean

School of Engineering, and

John M. Fluke Professor of Electrical Engineering

Stanford University

Stanford, Calif.

  1. Albert Reece1

Vice President for Medical Affair, ands

Bowers Distinguished Professor and Dean

School of Medicine

University of Maryland

Baltimore

Nancy Schwartz

Professor of Pediatrics and Biochemistry, and

Dean for Postdoctoral Affairs

University of Chicago

Chicago

Paula Stephan

Professor of Economics

Andrew Young School for Policy Studies

Georgia State University

Atlanta

Lorraine Tracey

Medical Science Liaison

Teva Pharmaceuticals

Washington, D.C.

Michael Turner2

Bruce V. and Diana M. Rauner Distinguished Service Professor, and

Director

Kavli Institute for Cosmological Physics

University of Chicago

Chicago

Allison Woodall

Deputy General Counsel

Labor, Employment, and Benefits Group

University of California System

Oakland

Joan Woodard

Former Executive Vice President and Deputy Director

Sandia National Laboratories (retired)

Washington, D.C.

STAFF

Kevin Finneran

Study Director

_________________________________________

1Member, Institute of Medicine

2Member, National Academy of Sciences

3Member, National Academy of Engineering