ABSTRACT
Background Evaluating seroprevalence study risk of bias (RoB) is crucial for robust infection surveillance, but can be a time-consuming and subjective process. We aimed to develop decision rules for reproducible RoB assessment and an automated tool to implement these decision rules.
Methods We developed the SeroTracker-RoB approach to RoB assessment. To do so, we created objective criteria for items on the Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Prevalence Studies and developed decision rules for RoB based on these items. The criteria and decision rules were based on published guidance for assessing RoB for prevalence studies and expert opinion. Decision rules were validated against the SeroTracker database of seroprevalence studies, which included consensus manual RoB judgements from two independent reviewers. We measured efficiency by calculating paired-samples t-test for time to judge RoB using the automated tool versus manually for 25 randomly selected articles from the SeroTracker database, coverage as the proportion of database studies where the decision rules could evaluate RoB, and reliability by calculating intraclass correlations between automated and manual RoB assessments.
Results We established objective criteria for seven of nine JBI items. We developed a set of decision rules with 61 branches. The SeroTracker-RoB tool was significantly faster than manual assessment with a mean time of 0.80 vs. 2.93 minutes per article (p<0.001), classified 100% (n = 2,070) of studies, and had good reliability compared to manual review (intraclass correlation 0.77, 95% confidence interval 0.74 to 0.80). The SeroTracker-RoB Excel Tool embeds this approach in a simple data extraction sheet for use by other researchers.
Conclusions The SeroTracker-RoB approach was faster than manual assessment, with complete coverage and good reliability compared to two independent human reviewers. This approach and tool enable rapid, transparent, and reproducible evidence synthesis of infection prevalence studies, and may support public health efforts during future outbreaks and pandemics.
What is new?
What is already known: Risk of bias assessments are a core element of evidence synthesis but can be time consuming and subjective. As such, there is a need for validated and transparent tools to automate such assessments, particularly during disease outbreaks and pandemics to inform public health decision making. However, there are currently no automated tools for risk of bias assessment of prevalence studies.
What is new: We developed a reproducible approach to risk of bias assessment for SARS-CoV-2 seroprevalence studies. The automated approach was five times faster than manual human assessment, successfully categorized all 2,070 studies that it was tested on, and had good agreement with manual review. We built a simple Excel tool so that other researchers can use this automated approach.
Potential impact: The SeroTracker-RoB approach and tool enables rapid, transparent, and reproducible risk of bias assessments for SARS-CoV-2 seroprevalence studies, and could be readily adapted for other types of disease prevalence studies. This process may also be applicable to automation of critical appraisal and risk of bias assessment for other types of studies and in other scientific disciplines.
Competing Interest Statement
Rahul K. Arora was previously a Technical Consultant for the Bill and Melinda Gates Foundation Strategic Investment Fund, is a minority shareholder of Alethea Medical, and was a former Senior Policy Advisor at Health Canada. Each of these relationships is entirely unrelated to the present work. DAC reports consulting fees from Sensyne Health, Oxford University Innovation, and BioBeats, each outside the submitted work. MYL has received grants from the Bill and Melinda Gates Foundation, not pertaining to the present study. No other authors have conflicts of interest to report.
Funding Statement
SeroTracker receives funding for SARS-CoV-2 seroprevalence study evidence synthesis from the Public Health Agency of Canada through Canada's COVID-19 Immunity Task Force, the World Health Organization Health Emergencies Programme, the Robert Koch Institute, and the Canadian Medical Association Joule Innovation Fund. No funding source had any role in the design of this study, its execution, analyses, interpretation of the data, or decision to submit results. This manuscript does not necessarily reflect the views of the World Health Organization or any other funder.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
Declaration of interests RKA was previously a Technical Consultant for the Bill and Melinda Gates Foundation Strategic Investment Fund, is a minority shareholder of Alethea Medical, and was a former Senior Policy Advisor at Health Canada. Each of these relationships is entirely unrelated to the present work. DAC reports consulting fees from Sensyne Health, Oxford University Innovation, and BioBeats, each outside the submitted work. MYL has received grants from the Bill and Melinda Gates Foundation, not pertaining to the present study. No other authors have conflicts of interest to report.