Abstract
Objectives Prospectively registering study plans in a permanent time-stamped and publicly accessible document is becoming more common across disciplines and aims to reduce risk of bias and make risk of bias transparent. Selective reporting persists, however, when researchers deviate from their registered plans without disclosure. This systematic review aimed to estimate the prevalence of undisclosed discrepancies between prospectively registered study plans and their associated publication. We further aimed to identify the research disciplines where these discrepancies have been observed, whether interventions to reduce discrepancies have been conducted, and gaps in the literature.
Design Systematic review and meta-analyses.
Data sources Scopus and Web of Knowledge, published up to 15 December 2019.
Eligibility criteria Articles that included quantitative data about discrepancies between registrations or study protocols and their associated publications.
Data extraction and synthesis Each included article was independently coded by two reviewers using a coding form designed for this review (osf.io/728ys). We used random-effects meta-analyses to synthesize the results.
Results We reviewed k = 89 articles, which included k = 70 that reported on primary outcome discrepancies from n = 6314 studies and, k = 22 that reported on secondary outcome discrepancies from n = 1436 studies. Meta-analyses indicated that between 29% to 37% (95% confidence interval) of studies contained at least one primary outcome discrepancy and between 50% to 75% (95% confidence interval) contained at least one secondary outcome discrepancy. Almost all articles assessed clinical literature, and there was considerable heterogeneity. We identified only one article that attempted to correct discrepancies.
Conclusions Many articles did not include information on whether discrepancies were disclosed, which version of a registration they compared publications to, and whether the registration was prospective. Thus, our estimates represent discrepancies broadly, rather than our target of undisclosed discrepancies between prospectively registered study plans and their associated publications. Discrepancies are common and reduce the trustworthiness of medical research. Interventions to reduce discrepancies could prove valuable.
Registration osf.io/ktmdg. Protocol amendments are listed in Supplementary Material A.
Competing Interest Statement
The authors have declared no competing interest.
Clinical Trial
osf.io/ktmdg
Clinical Protocols
Funding Statement
Robert Thibault was supported by a general support grant awarded to METRICS from Arnold Ventures and postdoctoral fellowships from the Canadian Institutes of Health Research (CIHR) and the Fonds de recherche du Quebec - Sante (FRQS). Hugo Pedder was supported by funding from the National Institute of Health and Care Excellence's Guidelines Technical Support Unit and Bristol's National Institute for Health and Care Research Technology Assessment Group. Olmo van den Akker was supported by Consolidator Grant (IMPROVE) from the European Research Council (ERC; grant no. 726361). Marcus Munafo, Robert Thibault, Jacqueline Thompson, and Robbie Clark were part of the MRC Integrative Epidemiology Unit (MC_UU_00011/7). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
NA. This is a systematic review.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
Content moved from the supplementary material to the main manuscript. Additional supplementary material provided. Text revised for clarity. The data and analyses remain the same as in version 1.
Data Availability
Data, data dictionaries, analysis script and materials related to this study are publicly available on the Open Science Framework at https://osf.io/5gfty/. The study protocol and materials were registered on 17 January 2020 at https://osf.io/ktmdg. Discrepancies between this manuscript and the registered protocol are outlined in the Supplementary Material A. To facilitate reproducibility, this manuscript was written by interleaving regular prose and analysis code using R Markdown. The relevant files are available in a Code Ocean container (https://doi.org/10.24433/CO.4753181.v1) which recreates the software environment in which the original analyses were performed. This container allows this manuscript to be reproduced from the data and code with a single button press.
