Abstract
Background Mandates and recommendations related to embedding open science practices within the research lifecycle are increasingly common. Few stakeholders, however, are monitoring compliance to their mandates or recommendations. It is necessary to monitor the current state of open science to track changes over time and to identify areas to create interventions to drive improvements.
Monitoring open science practices requires that they are defined and operationalized. Involving the biomedical community, we sought to reach consensus on a core set of open science practices to monitor at biomedical research institutions.
Methods and Findings To establish consensus in a structured and systematic fashion, we conducted a modified 3-round Delphi study. Participants in Round 1 were 80 individuals from 20 biomedical research institutions that exhibit interest in or actively support open science. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In Rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices that could be important and meaningful to monitor in an automated institutional open science dashboard. Participants voted on the inclusion of each item and provided a rationale for their choice. We defined consensus as 80% agreement. Between rounds, participants received aggregated voting scores for each item and anonymized comments from all participants, and were asked to re-vote on items that did not reach consensus. For Round 3, we hosted two half- day virtual meetings with 21 and 17 participants respectively to discuss and vote on all items that had not reached consensus after Round 2. Ultimately, participants reached consensus to include a 19 open science practices.
Conclusions A group of international stakeholders used a modified Delphi process to agree upon open science practices to monitor in a proposed open science dashboard for biomedical institutions. The core set of 19 open science practices identified by participants will form the foundation for institutional dashboards that display compliance with open science practices. They will now be assessed and tested for automatic inclusion in terms of technical feasibility. Using user-centered design, participating institutions will be involved in creating a dashboard prototype, which can then be implemented to monitor rates of open science practices at biomedical institutions. Our methods and approach may also transfer to other research settings–other disciplines could consider using our consensus list as a starting point for agreement upon a discipline-specific set of open science practices to monitor. The findings may also be of broader value to the development of policy, education, and interventions.
Competing Interest Statement
CLA is the editor in chief of JOSPT (Journal of Orthopaedic & Sports Physical Therapy). The other authors declare no conflicts.
Clinical Protocols
Funding Statement
This work was supported by a Wellcome Trust Open Research Fund [223828/Z/21/Z].
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The Ottawa Health Science Network Research Ethics Board gave ethical approval for this work.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
All data produced are available online at https://osf.io/jm8wg/
