ABSTRACT
Objective To compare results reporting and the presence of spin in COVID-19 study preprints with their finalized journal publications
Design Cross-sectional
Setting International medical literature
Participants Preprints and final journal publications of 67 interventional and observational studies of COVID-19 treatment or prevention from the Cochrane COVID-19 Study Register published between March 1, 2020 and October 30, 2020
Main outcome measures Study characteristics and discrepancies in 1) Results reporting (number of outcomes, outcome descriptor, measure (e.g., PCR test), metric (e.g., mean change from baseline), assessment time point (e.g., 1 week post treatment), data reported (e.g., effect estimate and measures of precision), reported statistical significance of result, type of statistical analysis (e.g., chi-squared test), subgroup analyses (if any), whether outcome was identified as primary or secondary and 2) Spin (reporting practices that distort the interpretation of results so that results are viewed more favorably).
Results Of 67 included studies, 23 (34%) had no discrepancies in results reporting between preprints and journal publications. Fifteen (22%) studies had at least one outcome that was included in the journal publication, but not the preprint; 8 (12%) had at least one outcome that was reported in the preprint only. For outcomes that were reported in both preprints and journals, common discrepancies were differences in numerical values and statistical significance, additional statistical tests and subgroup analyses conducted in journal publications, and longer follow-up times for outcome assessment in journal publications.
At least one instance of spin occurred in both preprints and journals in 23 / 67 (34%) studies, the preprint only in 5 (7%) studies, and the journal publications only in 2 (3%) of studies. Spin was removed between the preprint and journal publication in 5/67 (7%) studies; but added in 1/67 (1%) study.
Conclusions The COVID-19 preprints and their subsequent journal publications were largely similar in reporting of study characteristics, outcomes and spin. All COVID-19 studies published as preprints and journal publications should be critically evaluated for discrepancies and spin.
EQUATOR REPORTING GUIDELINE STROBE
What is already known on this topic
Selective and incomplete reporting of results and spin are threats to the trustworthiness and validity of research.
These reporting practices could be particularly dangerous for users of COVID-19 research as they can inflate the efficacy of interventions and underestimate harms.
Given the high prevalence, visibility, and potentially rapid implementation of COVID-19 research published as preprints, it is important to compare components of results reporting and the presence of spin in COVID-19 studies on treatment or prevention that are published both as preprints and journal publications.
What this study adds
This comparison of 67 COVID-19 preprints related to treatment or prevention and their subsequent journal publications found they were largely similar in reporting of study characteristics, components of results reporting and spin in interpretation.
Even a few important discrepancies could impact decision making.
Competing Interest Statement
no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years. RF is a Cochrane employee and part of the development team for the Cochrane COVID-19 Study Register. No other authors declare any other relationships or activities that could appear to have influenced the submitted work.
Clinical Trial
not applicable
Clinical Protocols
https://osf.io/5ru8w/?view_only=fe509bf54c104354a1e12f011bdff66a
Funding Statement
This study had no funding. Role of the funder: not applicable
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study analyzes publicly available information and is exempt from ethics review.
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
rosa.lawrence{at}cuanschutz.edu
louis.lesile{at}cuanschutz.edu
kellia.chiu{at}sydney.edu.au
sally.mcdonald{at}sydney.edu.au
matthew.page{at}monash.edu
quinn.grundy{at}utoronto.ca
lisa.parker{at}sydney.edu.au
E: sboughton{at}cochrane.org
jamie.kirkham{at}manchester.ac.uk
rfeatherstone{at}cochrane.org
Paper in collection COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv
The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, University of Washington, and Vrije Universiteit Amsterdam.