Abstract
Introduction As much as 50%-90% of research is estimated to be irreproducible, costing upwards of $28 billion in the United States alone. Reproducible research practices are essential to improving the reproducibility and transparency of biomedical research, such as including pre-registering studies, publishing a protocol, making research data and metadata publicly available, and publishing in open access journals. Here we report an investigation of key reproducible or transparent research practices in the published oncology literature.
Methods We performed a cross-sectional analysis of a random sample of 300 oncology studies published from 2014-2018. We extracted key reproducibility and transparency characteristics in a duplicative fashion by blinded investigators using a pilot tested Google Form.
Results Of the 300 studies randomly sampled, 296 studies were analyzed for study reproducibility characteristics. Of these 296 studies, 194 were contained empirical data that could be analyzed for reproducible and transparent research practices. Raw data was available for 9 studies (4.6%). Approximately 5 studies (2.6%) provided a protocol. Despite our sample including 15 clinical trials and 7 systematic reviews/meta-analyses, only 7 included a pre-registration statement. Less than 25% (65/194) of studies provided an author conflict of interest statement.
Discussion We found that key reproducibility and transparency characteristics were absent from a random sample of published oncology studies. We recommend required pre-registration for all eligible trials and systematic reviews, published protocols for all manuscripts, and deposition of raw data and metadata in public repositories.
Introduction
The ability to reproduce, or replicate, research results is a cornerstone of scientific advancement1,2. Absent efforts to advance the reproducibility of scientific research, advancements in patient care and outcomes may be delayed3,4, in part due to a failure in the translation of evidence to practice5. Evidence may fail translation to practice owing to bias6,7, lack of publication4, or poor reporting8. Thus, it may not be surprising that recent estimates of irreproducible research span a range of 50%-90% of all articles, costing upwards of $28 billion in the United States alone9. Moreover, it may not be surprising that large-scale efforts to replicate (ie, re-enact or reconduct previously published research studies) have failed10, in part due to an inability to navigate published methods. What is lost when scientific research fails to be reproducible carries significant weight; namely, the ability of science to be self-correcting11 and produce trustworthy results12.
It is commonly accepted that certain items are essential to improving the reproducibility of biomedical research. Examples of such items include pre-registering studies, publishing a protocol, making research data and metadata publicly available, and publishing in such a way to allow free access to the final manuscript. Pre-registering a study and publishing a protocol are important to prevent selective publication of studies with “positive” results13 and preventing the reordering of endpoints based on statistical significance14,15. Providing access to one’s raw research data, metadata, and analysis script allows independent researchers to computationally reproduce results, tailor results to specific patient populations, and determine the rigor of statistical analysis16,17. Publishing in open access journals or using preprint servers allows readers across economically diverse countries to access research articles that have implications for clinical practice18. Altogether, reproducible research practices aim to increase the efficiency, usefulness, and rigor of published research5.
Despite a high rate of author endorsement of reproducible practices19,20, some evidence suggests that authors infrequently implement them21. Absent such reproducible research practices, attempts to validate study findings may be thwarted. For example, Bayer and Amgen both attempted to replicate oncology research studies, with each failing to do so22,23. Bayer’s attempt to reproduce prior research studies is especially significant because they attempted to reproduce internal studies. Other non-pharmaceutical entities have attempted to replicate cancer research studies with similar results24. One may hypothesize that improved use and reporting of key reproducible or transparent research practices would improve future efforts to reproduce oncology research studies and build trust in existing evidence. Building on recent, similar analyses25–27, here we report an investigation of key reproducible or transparent research practices in the published oncology literature.
Methods
We performed an observational study using a cross sectional design based on methods developed by Hardwicke et. al.25 with modifications. Our study employed best-practice design in accordance with published guidance, where relevant28,29. Study protocol, raw data, and other pertinent materials are available on the Open Science Framework (https://osf.io/x24n3/). This study did not meet U.S. regulation requirements to be classified as human research, therefore it is exempt from Institutional Review Board approval30.
Journal Selection
We used the National Library of Medicine (NLM) catalog to search for all oncology journals using the subject terms tag Neoplasms[ST]. This search was performed on May, 29, 2019 which identified 344 journals. The inclusion criteria required that journals were both in “English” and “MEDLINE indexed”. We extracted electronic ISSN (or linking if electronic was unavailable)for each journal to use in a PubMed search on May 31, 2019. The total list of publications was then limited to those from January 1, 2014 to December 31, 2018.From search returns, we selected a random sample of 300 publications using Excel’s random number function (https://osf.io/wpev7/).
Data Extraction
We used a pilot-tested Google Form based on the one provided by Hardwicke et. al. 25 with modifications (https://osf.io/3nfa5/). The first modifications were extracting the 5-year impact factor and the date of the most recent impact factor, neither of which were extracted by Hardwicke, et. al. Second, additional study designs were added to include cohort, case series, secondary analyses, chart reviews, and cross-sectional studies. Third, funding options were expanded that allowed for greater specification of university, hospital, public, private/industry, or non-profit sources.
The Google Form contained questions for investigators aimed at identifying whether a study demonstrated the information necessary to be reproducible (Table 1, Supplement 1). Variations in study design changed the data that was extracted from each study. For example, studies with no empirical data (e.g. editorials, commentaries [without reanalysis], simulations, news, reviews, and poems) were unable to examined for reproducibility characteristics. However, for all publications, the following data were extracted: title of study, 5 year impact factor, impact factor of the most recent year, country of corresponding author and publishing journal, type of study participants (eg, human or animal), study design, author conflicts of interest, funding source, and whether the article was open access (Table 2). Studies with empirical data were examined for the following characteristics in addition to those stated above: material and data availability, analysis scripts, linkable protocol, and trial pre-registration statements. Together, the 8 key reproducibility and transparency indicators analyzed were as follows: material availability, raw data availability, analysis scripts, linkable protocol, trial pre-registration statements, author conflict of interest statement, funding source, and open access. Open access was determined using www.openaccessbutton.org. In the event a study could not be found, investigators performed a Google search to see if the study was available elsewhere. Web of Science was used to evaluate whether each examined publication 1) had been replicated in other works and 2) was included in future systematic reviews or meta-analyses.
Prior to data extraction, each investigator underwent a full day of training to increase the interrater reliability of the results between authors. This training consisted of an in-person session that reviewed study design, protocol, and Google Form. Investigators (C.G.W., N.V.) extracted data from 3 sample articles and differences were reconciled following extraction. A recording of this training session is available and listed online for reference (https://osf.io/tf7nw/). One investigator (C.G.W.) extracted data from all 300 publications. Z.J.H. extracted data for 200 publications and N.V. extracted data for 100 publications. C.G.W.’s data were compared to Z.J.H.’s and N.V.’s with discrepancies being resolved via group discussion. All authors were blinded to each other’s results. A final consensus meeting was held by all authors to resolve disagreements. If no agreement could be made, final judgment was made by an additional author (D.T.).
Statistical Analysis
Descriptive statistics were calculated for each category using Microsoft Excel with 95% confidence intervals.
Results
The NLM search identified 344 journals but only 204 fit our inclusion criteria. Our initial search string retrieved 199,420 oncology publications, from which, 300 were randomly sampled. Approximately 296 publications were analyzed for study reproducibility characteristics; 4 studies were not accessible, thus they were excluded from our analysis. Of these 296 publications, 215 contained empirical data and 81 did not. Publications without empirical data were unable to be analyzed for study reproducibility characteristics. Additionally, 21 publications with empirical data were case studies or case series. These case studies and series are unable to be replicated, thus are excluded from the analysis of study characteristics. In total, we were able to extract study reproducibility characteristics for 194 oncology publications (Figure 1).
Study Characteristics
In our sample of oncology publications, the publishing journals had a median 5 year impact factor of 3.445 (IQR 2.27-5.95). The majority (156/296, 52.7%) of journals were located in the United States. Over half (165/296, 55.8%) of published studies were available for free via open access networks. The remaining 131 publications (44.2%) were located behind a paywall — making the studies inaccessible to the public — available only through paid reader access. Approximately 109 publications (36.8%) made no mention of funding source. Public funding (95/296, 32.1%), such as state or government institutions, comprised the next most prevalent source of study funding. Publication authors disclosed no conflict of interest more frequently than conflicts of interest (174/296, 58.8 vs. 57/296, 19.2%); however, 65 publications (22.0%) had no author conflict of interest statement. Human participants were the most common study population in sample (154/269, 52.0%). Citation rates of these 296 publications by systematic reviews and meta-analyses can be found in Table 2.
Reproducibility Characteristics
Only 21 publications (21/194, 10.8%) made their raw data available. Approximately 9 of these publications with available raw data were downloadable by readers, while the rest was available upon request from the corresponding author of the study. A complete description of study materials required to reproduce the study — laboratory instruments, stimuli, computer software — was provided in 6/194 studies (3.2%). Of those publications with available materials, most were only accessible to readers upon request to the corresponding author, rather than being listed in a protocol or methods section. None of the included studies made their analysis scripts accessible, which details the steps the authors used to prepare the data for interpretation. Only 5 (5/194, 2.6%) publications provided a protocol detailing the a priori study design, methods, and analysis plan. Seven publications (7/194, 3.6%) were pre-registered in trial databases, such as ClinicalTrials.gov, prior to commencement of the study, despite there being 15 clinical trials and 7 systematic reviews/meta-analyses included in our analysis. One publication (1/194, 0.05%) claimed to be a replication study; all remaining studies (193/194, 99.5%) claimed to be novel or did not provide a clear statement about being a replication study. A subgroup analysis of the 8 key reproducibility and transparency indicators demonstrated that 29 publications had 0 indicators, 62 publications had 1 indicator, 209 publications had 2 to 5 indicators, and 0 publications had 6 or more.
Discussion
Our cross-sectional investigation of a sample of the published oncology literature found that key reproducibility and transparency practices were lacking or entirely absent. Namely, we found that publications rarely pre-registered their methods, published their full protocol, or deposited raw data and analysis scripts into a publicly-accessible repository. Moreover, conflicts of interest were not discussed approximately 20% of the time and just over half of the included studies were not accessible due to journal paywalls. Given the challenges in understanding the molecular mechanisms that drive cancer, the continuum of research in the field of oncology is slow, laborious and inefficient31. To combat these inherent obstacles, transferring outcomes and information from preclinical to clinical research demands consistency and precision across the continuum. Otherwise, publications downstream in the cancer research continuum may be based on spurious results incapable of independent confirmation due to a lack of access to study data, protocols, or analysis scripts. Science advances more rapidly when people spend less time pursuing false leads32, thus, for patients with cancer and for whom rapid scientific advancement is most significant, it is paramount that scientists, researchers and physicians advocate for an efficient research system that is transparent, reproducible, and free from bias.
Pre-registration of research study methods is a mechanism to improve the reproducibility of published results and prevent bias — either from selective reporting of outcomes or selective publication of a study33. Previously, it has been shown that the selective reporting of study endpoints affects the research portfolio of drugs or diseases15,34,35. For example, Wayant et. al found that 109 RCTs of malignant hematology interventions selectively reported their trial endpoints 118 times, with a significant portion doing so in a manner that highlighted statistically significant findings34. Were trial registries not available, these trials may have never been found to exhibit selective outcome reporting. Now, through trial registries, hematologists and other interested researchers are able to independently assess the robustness of not only study rationale and results, but also study rigor and reporting. The present study indicates that pre-registration of study methods was rare, even among trials and systematic reviews that have available registries. The importance of preregistration across the continuum of cancer research cannot be understated. For example, preclinical animal models serve as the foundation for clinical trials, but have exhibited suboptimal methods36, which may explain why animal study results fail to successfully translate to clinical benefit. In fact, it was recently shown that many phase 3 trials in Oncology are conducted despite no significant phase 2 results37. One possible explanation for why phase 3 trials proceed despite nonsignificant phase 2 results is the strong bioplausibility demonstrated in preclinical studies. If it is true that preclinical studies exhibit poor research methods, it is not unlikely that they are affected by selective outcome reporting bias, just like clinical research studies. Thus, to strengthen oncology research evidence — from foundational, preclinical research to practice-changing trials — we recommend either the creation of relevant study registers or the adherence to existing registration policies. In so doing, one key aspect of research — the accurate reporting of planned study endpoints — could be monitored, detected, and mitigated.
Equally important to self-correcting, rigorous cancer research is the publication of protocols, raw data, and analysis scripts. Protocols include much more information than study outcomes — they may elaborate on statistical analysis plans or decisions fundamental to the critical appraisal of study results38. It is unlikely that anyone would be able to fully appraise a published study without access to a protocol, and far less likely that anyone would be capable of replicating the results independently. In fact, two recent efforts to reproduce preclinical studies revealed extant barriers to independent verification of published findings20,39, including the absence of protocols, data, and analysis scripts. Our present investigation found that only 5 (2.6%) studies published a protocol, 9 (4.6%) fully published their data, and none published their analysis scripts. In the context of the recent failures to reproduce cancer research studies, one may reasonably conclude that our study corroborates the belief that oncology research is not immune to the same shortcomings that contribute to an ever-expanding cohort of irreproducible research findings40. Oncology research, like all biomedical research, is at an inflection point, wherein it may progress toward more transparent, reproducible, efficient research findings. However, in order to do so, the availability of protocols, data, and analysis scripts should be considered fundamental.
In summary, we found that key reproducibility and transparency characteristics were absent from a random sample of published oncology studies. The implication of this finding is a research system that is incapable of rapid self-correction, or a research system that places a stronger emphasis on what is reported rather than what is correct. We recommend three key action items which we believe benefit oncology research and all its stakeholders. First, require pre-registration for eligible trials and systematic reviews, since these study designs have existing registries available, and support the development of registries for preclinical studies. Second, understand that published reports are snapshots of a research study, and require protocols be published. Last, encourage a scientific culture that relies on data that is true and robust, rather than author reports of their data, by requiring the deposition of raw data, meta data, and analysis scripts in public repositories.
This study has several strengths and limitations. First, for strengths, we sampled 300 published oncology articles indexed in PubMed. In doing so we captured a diverse array of research designs in an even more diverse range of journals. As such, all oncology researchers can read our paper and glean useful information and enact changes to improve the reproducibility of new evidence. With respect to our limitations, our study is too broad to make absolute judgments about specific study designs. All signals that suggest irreproducible research practices from our study fall in line with prior data in other areas of medicine25–27, but are nonetheless signals rather than answers. We suggest more narrow investigations of the reproducibility of specific study designs and suggest trials and animal studies be prioritized due to their potential influence (present or future) on patient care. Moreover, we do not suggest that irreproducible research findings are false; however, the trust in the results may be blunted. Further, replicating (ie, reconducting) a study is not necessary in all cases to assess the rigor of the results. If a protocol, statistical analysis plan, and raw data (including metadata) are available, one fundamental pillar of science would be reinforced: self-correction.
Data Availability
Study protocol, raw data, and other pertinent materials are available on the Open Science Framework (https://osf.io/x24n3/).
Footnotes
Conflicts of Interest: None.
Source of Funding: This study was funded through the 2019 Presidential Research Fellowship Mentor – Mentee Program at Oklahoma State University Center for Health Sciences