ABSTRACT
Objectives To examine changes in completeness of reporting and frequency of sharing data, analytic code and other review materials in systematic reviews (SRs) over time; and factors associated with these changes.
Design Cross-sectional meta-research study.
Sample A random sample of 300 SRs with meta-analysis of aggregate data on the effects of a health, social, behavioural or educational intervention, which were indexed in PubMed, Science Citation Index, Social Sciences Citation Index, Scopus and Education Collection in November 2020.
Analysis/Outcomes The extent of complete reporting and frequency of sharing review materials in these reviews were compared with 110 SRs indexed in February 2014. Associations between completeness of reporting and various factors (e.g. self-reported use of reporting guidelines, journal’s data sharing policies) were examined by calculating risk ratios (RR) and 95% confidence intervals (CI).
Results Several items were reported sub-optimally among 300 SRs from 2020, such as a registration record for the review (38%), a full search strategy for at least one database (71%), methods used to assess risk of bias (62%), methods used to prepare data for meta-analysis (34%), and funding source for the review (72%). Only a few items not already reported at a high frequency in 2014 were reported more frequently in 2020. There was no evidence that reviews using a reporting guideline were more completely reported than reviews not using a guideline. Reviews published in 2020 in journals that mandated either data sharing or inclusion of Data Availability Statements were more likely to share their review materials (e.g. data, code files) (18% vs 2%).
Conclusion Incomplete reporting of several recommended items for systematic reviews persists, even in reviews that claim to have followed a reporting guideline. Data sharing policies of journals potentially encourage sharing of review materials.
What is already known on this topic
What is already known on this topic
Complete reporting of methods and results, as well as sharing data and analytic code, enhances transparency and reproducibility of systematic reviews. The extent of complete reporting and sharing of data or analytic code among systematic reviews needs to be comprehensively assessed.
Use of reporting guidelines, which are designed to improve reporting in systematic reviews, is increasing. It is unclear whether this increase has had an impact on reporting of methods and results in systematic reviews.
More journals are adopting open data policies which aim to promote data sharing. The impact of these policies on sharing data and analytic code in systematic reviews is also unclear.
What this study adds
What this study adds
Incomplete reporting of several recommended items in systematic reviews persists. Frequency of sharing review data and analytic code is currently low (7%).
An increase in self-reported use of a reporting guideline was observed between 2014-2020; however, there was no evidence that reviews using a reporting guideline were more completely reported than reviews not using a guideline.
Reviews published in 2020 in journals that mandated either data sharing or inclusion of Data Availability Statements were more likely to share their review materials (e.g. data, code files).
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This research is funded by the Australian Research Council Discovery Early Career Researcher Award (DE200101618), held by MJP; JEM is supported by the National Health and Medical Research Council Career Development Fellowship (APP1143429); DM is supported in part as the University Research Chair, University of Ottawa; NRH is funded by the Alexander von Humboldt Experienced Researcher Fellowship; DGH is supported by the Australian Commonwealth Government Research Training Program Scholarship; RK is supported by the Monash Graduate Scholarship and the Monash International Tuition Scholarship. The funders had no role in the study design, decision to publish, or preparation of the manuscript.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
We have provided greater emphasis on the reporting practices observed in the 2020 sample of systematic reviews and revised our analysis of the impact of data sharing policies on frequencies of data sharing
Data Availability
All datasets and analytic code can be found on the Open Science Framework (DOI: 10.17605/OSF.IO/JSP9T).
