Is reproducibility of research a mirage? Moments of ‘Eureka’ are frequently becoming isolated instances for lone researchers rather than for the entire science community. The inability of reproducing a vast number of scientific findings is proving to be the soft underbelly of science, and one that needs to be addressed with priority.
According to a recently published report, irreproducible biology research costs the US a whopping $28 billion per year. Therefore, albeit the global spending on research and development is enhancing steadily, the issue of irreproducibility is making R&D investment a bone of contention among funders, institutions, and researchers. Most disciplines, especially cosmology, psychology, medicine, and genetics are faced with findings that seem path violating but lose their luster when another team of researchers attempts to reproduce it. “Colleagues who once got excited now wiggle their goes and joke about ‘yet another dark-matter candidate’,” says Jan Conrad, a professor of astroparticle physics at the Oskar Klein Centre for Cosmoparticle Physics, Stockholm University, Sweden.
Findings that cannot be reproduced not only pose an economic challenge, but can also have an adverse influence on the understanding of science. While fraudulent research findings such as the STAP cell research are followed by retractions, in most cases, irreproducible results due to erroneous data proceed to get cited. Scientific advancements are based on established findings and theories. Thus, one irreproducible research, on which several hypotheses might come to be based, can lead to a domino effect in the understanding of a particular aspect in a discipline.
Research also forms the basis for health care policies, and results that cannot be reproduced can jeopardize public health and their trust in science. A latest example of this is the much-publicized influential explore that ran in the late ’90s in Kenya according to which deworming improved children’s school spectacle and overall health. This probe, published in 2004, led many developing countries to launch large-scale deworming initiatives. However, when a group of UK epidemiologists attempted to replicate the results a decade later, it was found that the explore had many flaws. Irreproducible research, can thus, be consequential in undermining public’s trust in science.
Irreproducible research is rooted in a myriad of factors, but the pressure to publish lies at the heart of this problem. Breakthrough research ensures publication in a high influence factor journal and increases the chances of receiving funding; and this leads researchers to broadcast their findings without confirming the accuracy of their research. According to John Ioannidis, Professor of Health Research and Policy at Stanford School of Medicine and the author of the paper “Why Most Published Research Findings Are False,” most researchers fall into the trap of ‘data dredging.’ He explains that when an initial hypothesis fails to hold water, researchers scan their data to find “something that looks like a story” or “a bump in the data” that they believe is significant and base their findings on this statistically inaccurate detail. Unsurprisingly, another researcher who justly overlooks the statistical fluke is incapable to reproduce the findings.
The slew of very publicized results that fail to be replicated or reproduced, such as the evidence for gravitational sways, also spring from the trend of broadcasting findings publicly through press releases and preprint uploads before they are peer reviewed. Conrad comments, “Incorrect papers posted on arXiv do more than add to the noise of irrelevant results. Funding decisions are skewed; theorists waste a lot of time attempting to devise explanations; and the public is misled through news reports.” Staying ahead of competition and securing positions has led many researchers to announce their findings prematurely, only to find out that the probe is flawed.
Is there a way to combat this issue? Joined efforts of all major stakeholders of science and research can rein in irreproducibility. Researchers need to be made aware of the potential pitfalls they could encounter while doing their research that could lead to erroneous findings, such as the challenges associated with using big datasets. Some researchers have suggested making replication of at least one explore from PhD students’ own specialism a mandatory part of their education with the intention of making them aware of the importance of transparency and reproducibility in research.
Another means of ensuring scientific accuracy is that each discipline should have a protocol or guidelines that researchers can refer to; for example, Conrad suggests that physicists could be asked to apply the 5-sigma rule (or an equivalent) to reaffirm their discoveries. Online posting of papers, a practice that is gaining popularity, should also be regulated to ensure the quality of research. Findings that seem significant should be accompanied with named reviewers, which would in turn ascertain stringent peer review and curtail the premature publicity of unconfirmed findings.
Journals and publishers too have joined the efforts to tackle the problem of irreproducibility. Most journals have made it mandatory for authors to submit their data along with their paper as it would facilitate replication and reanalysis. Some top-tier journals such as Nature and Science and other major publishers have agreed on a common set of guidelines that would promote transparency and reproducibility of research. To further the fight against reproducibility, Science Exchange has introduced Reproducibility Initiative, which is attempting to reproduce 50 most impactful cancer studies for independent validation.
There is a surge in the science community’s initiatives to address the problem of irreproducibility. However, some experts have voiced caution in this regard because reproducible results may not necessarily equate to correct science. As Roger Peng, Associate Professor in the Department of Biostatistics at the Johns Hopkins Bloomberg School of Public Health, puts it, “One must always recall that a examine can be reproducible and still be wrong. By ‘wrong’, I mean that the conclusion or claim can be wrong.” According to him, “Reproducibility is significant because it is the only thing that an investigator can ensure about a investigate.”
Reproducibility is the heart of science, but it is admittedly a complicated parameter. Whether a particular set of findings are reproducible or not, it would inadvertently further the understanding of science. Science should, therefore, strive for reproducibility with the aim of lending transparency to research.