RT Journal Article
SR Electronic
T1 What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2021.07.29.21261320
DO 10.1101/2021.07.29.21261320
A1 Austen El-Osta
A1 Iman Webber
A1 Aos Alaa
A1 Emmanouil Bagkeris
A1 Saba Mian
A1 Mansour Sharabiani
A1 Azeem Majeed
YR 2021
UL http://medrxiv.org/content/early/2021/07/31/2021.07.29.21261320.abstract
AB Objective To assess the suitability of primary care vignettes in benchmarking the performance of online symptom checkersDesign Observational study using publicly available, free online symptom checkersParticipants Three symptom checkers (Healthily, Ada and Babylon) that provided consultations in English. 139 standardized patient vignettes were compiled by RCGP. Three independent GPs interpreted the vignettes to arrive at a “Gold Standard” consisting of 3 dispositions and divided into one of three categories of triage urgency: (1) emergency care required, (2) primary care required and (3) self-care.Main outcome measures Six professional non-medical and lay inputters simulated 2774 standardized patient evaluations using 3 online symptom checkers (OSC). We recorded when OSC provided a triage recommendation and whether it correctly recommended the appropriate triage recommendation across three categories of triage urgency (emergency care, primary care or self-care). We collected data on whether the solution appeared within the first 3 dispositions in each of the standards across 2774 standardized patient evaluations.Results When benchmarked against the Gold Standard, Healthily provided an appropriate triage recommendation 61.9% of the time compared to 45.3% and 42.4% of the time for Babylon and Ada respectively. There was poor agreement between OSC consultation outcome and Gold Standard dispositions. When compared to the Gold Standard, Healthily gave an unsafe “under-triage” recommendation 28.6% of the time overall across the three categories compared to 43.3% for Ada and 47.5% for Babylon (P<0.001).Conclusions OSCs recommended ‘very unsafe’ triages only <4% of the time suggesting that the online consultation tools are generally working at a safe level of risk. Primary care vignettes are a helpful tool to support development of OSC, but not ideally suited to benchmark the performance of different OSC. Real-world evidence studies involving general practice are recommended to benchmark the performance of OSC in the community setting.Strengths and limitations of this study139 independently created primary care vignettes covering 18 subcategories of primary care were used to benchmark the performance of three online symptom checkers using 2774 unique patient simulationsA gold standard for each primary care vignette was derived using GP roundtables and single blinded testingWe investigated the extent that different inputters using the same vignette and online symptom checker received differing consultation outcomes and triage recommendationsWe developed an accuracy matrix to objectively monitor online symptom checker consultation outcome and the safety of the triage recommendationLimitations included a different number of inputters to simulate patients across the three online symptom checkers testedCompeting Interest StatementThe authors have declared no competing interest.Funding StatementUnconditional funding for this work was provided by Healthily (Imperial Self- Care 2020/1). The Funder did not have a role in study design or analysis. Austen El-Osta and Azeem Majeed are supported by the National Institute for Health Research (NIHR) Applied Research Collaboration (ARC) North West London. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Imperial College London Research Ethics CommitteeAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesNo additional data are available