PT  - JOURNAL ARTICLE
AU  - Sun, Ju
AU  - Peng, Le
AU  - Li, Taihui
AU  - Adila, Dyah
AU  - Zaiman, Zach
AU  - Melton, Genevieve B.
AU  - Ingraham, Nicholas
AU  - Murray, Eric
AU  - Boley, Daniel
AU  - Switzer, Sean
AU  - Burns, John L.
AU  - Huang, Kun
AU  - Allen, Tadashi
AU  - Steenburg, Scott D.
AU  - Gichoya, Judy Wawira
AU  - Kummerfeld, Erich
AU  - Tignanelli, Christopher
TI  - A Prospective Observational Study to Investigate Performance of a Chest X-ray Artificial Intelligence Diagnostic Support Tool Across 12 U.S. Hospitals
AID  - 10.1101/2021.06.04.21258316
DP  - 2021 Jan 01
TA  - medRxiv
PG  - 2021.06.04.21258316
4099  - http://medrxiv.org/content/early/2021/06/04/2021.06.04.21258316.short
4100  - http://medrxiv.org/content/early/2021/06/04/2021.06.04.21258316.full
AB  - Importance An artificial intelligence (AI)-based model to predict COVID-19 likelihood from chest x-ray (CXR) findings can serve as an important adjunct to accelerate immediate clinical decision making and improve clinical decision making. Despite significant efforts, many limitations and biases exist in previously developed AI diagnostic models for COVID-19. Utilizing a large set of local and international CXR images, we developed an AI model with high performance on temporal and external validation.Objective Investigate real-time performance of an AI-enabled COVID-19 diagnostic support system across a 12-hospital system.Design Prospective observational study.Setting Labeled frontal CXR images (samples of COVID-19 and non-COVID-19) from the M Health Fairview (Minnesota, USA), Valencian Region Medical ImageBank (Spain), MIMIC-CXR, Open-I 2013 Chest X-ray Collection, GitHub COVID-19 Image Data Collection (International), Indiana University (Indiana, USA), and Emory University (Georgia, USA)Participants Internal (training, temporal, and real-time validation): 51,592 CXRs; Public: 27,424 CXRs; External (Indiana University): 10,002 CXRs; External (Emory University): 2002 CXRsMain Outcome and Measure Model performance assessed via receiver operating characteristic (ROC), Precision-Recall curves, and F1 score.Results Patients that were COVID-19 positive had significantly higher COVID-19 Diagnostic Scores (median .1 [IQR: 0.0-0.8] vs median 0.0 [IQR: 0.0-0.1], p < 0.001) than patients that were COVID-19 negative. Pre-implementation the AI-model performed well on temporal validation (AUROC 0.8) and external validation (AUROC 0.76 at Indiana U, AUROC 0.72 at Emory U). The model was noted to have unrealistic performance (AUROC > 0.95) using publicly available databases. Real-time model performance was unchanged over 19 weeks of implementation (AUROC 0.70). On subgroup analysis, the model had improved discrimination for patients with “severe” as compared to “mild or moderate” disease, p < 0.001. Model performance was highest in Asians and lowest in whites and similar between males and females.Conclusions and Relevance AI-based diagnostic tools may serve as an adjunct, but not replacement, for clinical decision support of COVID-19 diagnosis, which largely hinges on exposure history, signs, and symptoms. While AI-based tools have not yet reached full diagnostic potential in COVID-19, they may still offer valuable information to clinicians taken into consideration along with clinical signs and symptoms.Competing Interest Statement1.NIH NHLBI T32HL07741 (NEI) 2.This research was supported by the Agency for Healthcare Research and Quality (AHRQ) and Patient-Centered Outcomes Research Institute (PCORI), grant K12HS026379 (CJT) and the National Institutes of Healths National Center for Advancing Translational Sciences, grants KL2TR002492 (CJT) and UL1TR002494 (EK). 3.NIH NIBIB 75N92020D00018/75N92020F00001 (JWG) 4.This research was supported by the University of Minnesota Office of the Vice President of Research (OVPR) COVID-19 Rapid Response Grants (JS, EK, CJT) Funding Statement1.NIH NHLBI T32HL07741 (NEI) 2.This research was supported by the Agency for Healthcare Research and Quality (AHRQ) and Patient-Centered Outcomes Research Institute (PCORI), grant K12HS026379 (CJT) and the National Institutes of Healths National Center for Advancing Translational Sciences, grants KL2TR002492 (CJT) and UL1TR002494 (EK). 3.NIH NIBIB 75N92020D00018/75N92020F00001 (JWG) 4.This research was supported by the University of Minnesota Office of the Vice President of Research (OVPR) COVID-19 Impact Grant (JS, EK, CJT) 5.This technology was submitted for a U.S. Patent Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This study was approved by the University of Minnesota institutional review board (STUDY 00011158). External validation at IU was deemed exempt by the IU IRB as all secondary data was fully de-identified and remained within IU (STUDY 2010169012). External validation of the model at Emory was approved by the Emory University institutional review board (STUDY00000506). All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesData is not publicly available