Reproducible research practices are essential to biomedical research because these practices promote trustworthy evidence. Unfortunately, the findings a new study, published in JAMA Oncology, suggests a lack of reproducible research.
The researchers of this study sought to evaluate the reproducible research of meta-analyses in oncology systematic reviews cited by the 49 National Comprehensive Cancer Network (NCCN) guidelines for treating cancer while discerning whether Cochrane reviews or systematic reviews use more reproducible research practices.
In this cross-sectional investigation study, researchers appraised all systematic reviews containing at least one meta-analysis and at least one included randomized clinical trial (RCT) data cited by the NCCN guidelines for cancer treatment. In total, the researchers identified 1,124 potential systematic reviews, and 154 meta-analyses comprising 696 meta-analytic effect size estimates. Subsequently, they scanned references of all NCCN guidelines for potential systematic reviews and analyses.
The primary endpoint of this study was specified as frequency of reproducible research practices, which was defined as effect estimate and measure of precision, a clear list of studies included for each analysis; and with respect to subgroup and sensitivity analyses, it must be clear which studies were included in each group or level. After all ascertained studies were screened, the researchers conducted data analysis between May 6, 2018 and January 28, 2019.
Irreproducibility is a Problem
According to the results of the study, only 64.3% (2,375 of the 3,696 meta-analytic estimates) were reproducible in theory. Among 100 systemic reviews, all meta-analytic estimates were observed as reproducible (64.9%) and in 15 systematic reviews (9.7%), the results showed that no meta-analytic estimates could be reproduced. Overall, the results appeared to show that data were attributed in only 29 meta-analyses, however none specified which data. Moreover, only one meta-analysis contained an online data set link.
The problem of irreproducibility for #cancer meta-analysis studies and systematic reviews: only 64% hold up (and even that is likely an over-estimate)https://t.co/ovhivFuklC @JAMAOnc by @ColeWayant_OK and colleagues
— Eric Topol (@EricTopol) September 5, 2019
“More reproducible research practices are needed in oncology meta-analyses, as suggested by those that are cited by the NCCN,” the study authors wrote in their conclusion. “Reporting meta-analyses in forest plots and requirements for full data sharing are recommended.”
To what extent do clinically relevant oncology systematic reviews cited by @NCCN guidelines use reproducible research practices? Sobering new findings & editorial. Via @JAMAOnc https://t.co/ai9qpl3FaW & https://t.co/DSx30cZIDC pic.twitter.com/ViejaZnYnj
— Muin J. Khoury (@MuinJKhoury) September 5, 2019
Evaluation of Reproducible Research Practices in Oncology Systematic Reviews With Meta-analyses
— Cole Wayant (@ColeWayant_OK) September 5, 2019