What's the worst thing I could possibly see in the methods section of a research paper besides an absent catalog number for an antibody?
A sentence including the phrase "as described previously in reference X."
Have you ever looked at the methods sections of a published research article only to find that the details of a technique you are interested in are tucked away in a cited paper? Being a good researcher, you rummage through that referenced paper only to find that the information you want is again cited in a previous paper. You search again. Another citation. The cycle continues until finally, you find that golden paper. But wait, you have to pay for it!
How can scientists learn new protocols, validate reagents, and most importantly reproduce experiments if methods are not correctly described? Where is the rigor that scientists are known for? Is the scientific community really holding each other accountable, especially in preclinical research?
This referencing issue stems from a bigger problem: The reproducibility crisis. Compared to industry settings where quality control and validation are strictly enforced, the same rigor is not always present in academia. This leads to irreproducible research in which a scientists' own work and work from their peers cannot be reproduced. A Nature survey revealed that 90% of scientists realize that reproducibility is a problem, however, only 33% have recently established procedures for reproducibility in their labs. In addition, over 70% of scientists reported that they are were unable to reproduce another scientist’s research while over 50% could not reproduce their own experiments.
This crisis can be attributed to:
Despite the alarming scale of the reproducibility crisis and the statistics affirming the need for independent validation of research studies, a simple method to aid reproducibility is to increase transparency of the methodologies and materials used in scientific work. Considering that reading published literature is the core way researchers keep up to date in their field, responsibility also falls on journal editorial teams that do not have explicit standards for describing experimental design.
Several journals have already taken the initiative to address the reproducibility crisis. Along with the National Institutes of Health (NIH), Nature Publishing Group and Science convened a 2014 joint workshop to support transparency and reproducibility in science publishing. Editors representing 30 preclinical research journals agreed upon and endorsed a list of principles and guidelines for reporting preclinical research.
Some of these criteria include:
Other organizations, especially public funding agencies also have created guidelines requiring compliance in rigor and reproducibility. The Tri-Agency Responsible Conduct of Research Framework developed by the Canadian Institutes of Health Research, the Natural Sciences and Engineering Research Council of Canada, and the Social Sciences and Humanities Research Council of Canada in 2016 require that all researchers who accept or apply for agency funds meet research integrity standards including keeping strict and accurate records of methods and data and accurately referencing work.
The NIH is also requiring that applications for many research and institutional grants to include plans for addressing reproducibility and transparency including authentication of key biological and chemical resources.
The Federation of American Societies for Experimental Biology (FASEB), encompassing 31 scientific societies, also came out with a report on Enhancing Research Reproducibility advocating that stakeholders such as professional societies, reagent vendors, funding agencies, and journals develop a collective standard on describing reagents, laboratory tools, and protocols to increase transparency in science.
While proposed policies and recommendations for universal standards put forth by funding agencies and scientific societies are important, it is up to each scientist to uphold responsible conduct to aid reproducibility. This is also important for scientists whose work is published in journals that do not enforce strict reporting guidelines.
So, what can you do?
With these standards in place, any scientist can tackle the reproducibility crisis in research. Hopefully, there will come a day where we will be able to find all the details needed in a published paper to thoroughly reproduce an experiment and we no longer will see the words "as described previously."