What's the worst thing I could possibly see in the methods section of a research paper besides an absent catalog number for an antibody?
A sentence including the phrase "as described previously in reference X."
Have you ever looked at the methods sections of a published research article only to find that the details of a technique you are interested in are tucked away in a cited paper? Being a good researcher, you rummage through that referenced paper only to find that the information you want is again cited in a previous paper. You search again. Another citation. The cycle continues until finally, you find that golden paper. But wait, you have to pay for it!
How can scientists learn new protocols, validate reagents, and most importantly reproduce experiments if methods are not correctly described? Where is the rigor that scientists are known for? Is the scientific community really holding each other accountable, especially in preclinical research?
The Bigger Problem with Referencing Is Reproducibility
This referencing issue stems from a bigger problem: The reproducibility crisis. Compared to industry settings where quality control and validation are strictly enforced, the same rigor is not always present in academia. This leads to irreproducible research in which a scientists' own work and work from their peers cannot be reproduced. A Nature survey revealed that 90% of scientists realize that reproducibility is a problem, however, only 33% have recently established procedures for reproducibility in their labs. In addition, over 70% of scientists reported that they are were unable to reproduce another scientist’s research while over 50% could not reproduce their own experiments.
This crisis can be attributed to:
- Pressures put on academics to quickly publish in high impact journals
- Lack of openness in reporting research to maintain competitive edge in field
- Poor training of experimental design for new researchers and students
- Shortage of time and resources to reproduce and troubleshoot work
- Space limits on experimental design and methods in journals
One Simple Solution: Increase Transparency
Despite the alarming scale of the reproducibility crisis and the statistics affirming the need for independent validation of research studies, a simple method to aid reproducibility is to increase transparency of the methodologies and materials used in scientific work. Considering that reading published literature is the core way researchers keep up to date in their field, responsibility also falls on journal editorial teams that do not have explicit standards for describing experimental design.
Several journals have already taken the initiative to address the reproducibility crisis. Along with the National Institutes of Health (NIH), Nature Publishing Group and Science convened a 2014 joint workshop to support transparency and reproducibility in science publishing. Editors representing 30 preclinical research journals agreed upon and endorsed a list of principles and guidelines for reporting preclinical research.
Some of these criteria include:
- Adopting a core set of standards for reporting design study based on Landis et al.
- Checking the statistical analysis of submitted work
- Requesting deposition of datasets on which conclusion of papers rely
- Assuming responsibility to consider refutations of published papers
- Establishing practices for description biological materials such as antibodies, cell lines, and animals
Other organizations, especially public funding agencies also have created guidelines requiring compliance in rigor and reproducibility. The Tri-Agency Responsible Conduct of Research Framework developed by the Canadian Institutes of Health Research, the Natural Sciences and Engineering Research Council of Canada, and the Social Sciences and Humanities Research Council of Canada in 2016 require that all researchers who accept or apply for agency funds meet research integrity standards including keeping strict and accurate records of methods and data and accurately referencing work.
The NIH is also requiring that applications for many research and institutional grants to include plans for addressing reproducibility and transparency including authentication of key biological and chemical resources.
The Federation of American Societies for Experimental Biology (FASEB), encompassing 31 scientific societies, also came out with a report on Enhancing Research Reproducibility advocating that stakeholders such as professional societies, reagent vendors, funding agencies, and journals develop a collective standard on describing reagents, laboratory tools, and protocols to increase transparency in science.
Every Scientist Must Take Responsibility
While proposed policies and recommendations for universal standards put forth by funding agencies and scientific societies are important, it is up to each scientist to uphold responsible conduct to aid reproducibility. This is also important for scientists whose work is published in journals that do not enforce strict reporting guidelines.
So, what can you do?
- Keep complete records of your experiments. Comprehensive, accurate and well-organized record-keeping of protocols and results is essential for reporting and reproducibility. Any other scientist should be able to read and reproduce your work based on your lab notebook.
- Describe methods and materials in full. Resolving the reproducibility crisis depends on the collective effort of scientists and journal editors to mandate that full details of experimental design and materials are described in the methods section. Keeping clear records of your experiment protocols and reagents will make this easy and aid transparency.
- Reference accurately. Science advances because we build on the work of our peers. Give credit when credit is due. In addition, while it may be easy to self-cite and write "This assay was performed as described in reference X," you are creating one more barrier to ending the reproducibility crisis, which is not helping the scientific community at large.
- Take reproducibility training. No matter at what stage you are at in your career, periodic reproducibility training is key. As we are making advancements in science every day, there will always be opportunities to improve on experimental design, ensuring scientific rigor and transparency. The NIH launched a series of free reproducibility training modules that anyone can access. Research and academic institutions should start this type of training early and require graduate students to take such workshops to increase research integrity and improve communication within their respective fields.
- Use BenchSci! BenchSci's AI-driven platform was developed with reproducibility in mind. By having access to easily searched for published data, researchers can quickly decide what antibody to purchase for their experiments. Reviews are built into BenchSci's platform to promote independent validations of antibodies that have been tried and tested. So, while scientists search for the best product for their work, they can also contribute to improving reproducibility in research.
With these standards in place, any scientist can tackle the reproducibility crisis in research. Hopefully, there will come a day where we will be able to find all the details needed in a published paper to thoroughly reproduce an experiment and we no longer will see the words "as described previously."
