Abstract
Introduction The Consortium for Clinical Characterization of COVID-19 by EHR (4CE) includes hundreds of hospitals internationally using a federated computational approach to COVID-19 research using the EHR.
Objective We sought to develop and validate a standard definition of COVID-19 severity from readily accessible EHR data across the Consortium.
Methods We developed an EHR-based severity algorithm and validated it on patient hospitalization data from 12 4CE clinical sites against the outcomes of ICU admission and/or death. We also used a machine learning approach to compare selected predictors of severity to the 4CE algorithm at one site.
Results The 4CE severity algorithm performed with pooled sensitivity of 0.73 and specificity 0.83 for the combined outcome of ICU admission and/or death. The sensitivity of single code categories for acuity were unacceptably inaccurate - varying by up to 0.65 across sites. A multivariate machine learning approach identified codes resulting in mean AUC 0.956 (95% CI: 0.952, 0.959) compared to 0.903 (95% CI: 0.886, 0.921) using expert-derived codes. Billing codes were poor proxies of ICU admission, with 49% precision and recall compared against chart review at one partner institution.
Discussion We developed a proxy measure of severity that proved resilient to coding variability internationally by using a set of 6 code classes. In contrast, machine-learning approaches may tend to overfit hospital-specific orders. Manual chart review revealed discrepancies even in the gold standard outcomes, possibly due to pandemic conditions.
Conclusion We developed an EHR-based algorithm for COVID-19 severity and validated it at 12 international sites.
Competing Interest Statement
RB and AM are shareholders of Biomeris s.r.l. KDM is an advisor to Medal, Inc.
Funding Statement
This work was supported in part by the following grants: NIH National Institute of Neurological Disorders and Stroke R01NS098023; NIH Institute of Child Health and Human Development K12 HD047349; NIH Heart, Lung, and Blood Institute K23-HL148394; NIH Heart, Lung, and Blood Institute L40-HL148910; NIH National Center for Advancing Translational Sciences U01TR002623; NIH National Center for Advancing Translational Sciences UL1-TR001420; NIH National Center for Advancing Translational Sciences UL1-TR001878; NIH National Center for Advancing Translational Sciences UL1TR002240; NIH National Center for Advancing Translational Sciences UL1TR001881; NIH National Center for Advancing Translational Sciences UL1 TR00185705; NIH National Center for Advancing Translational Sciences 5UL1TR001857-05; NIH National Library of Medicine R01 LM012095; NIH National Library of Medicine R01LM013345; NIH National Human Genome Research Institute 5R01HG009174-04; NIH National Heart, Lung, and Blood Institute U01HL121518; German Federal Ministry of Education and Research FKZ 01ZZ1801B
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Each institution reported obtaining proper institutional review board approval for data sharing. Certifications of waivers or approval were collected by the consortium. As data were transmitted in aggregate, no patient level data were available from any site. The following IRBs approved the study as Exempt after review: Mass General Brigham; University of Pennsylvania; University of Pittsburgh; Beth Israel Deaconess Medical Center; University of Michigan; University of California, Los Angeles; Bordeaux University Hospital; Medical Center, University of Freiburg; Boston Children's Hospital The following IRBs approved the study with a waiver of review: Istituti Clinici Scientifici Maugeri; St. Luke's University Health Network; National University Hospital
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
All data collected for this study is presented in the manuscript or appendix. The 4CE Consortium provides additional visualizations and data for other consortium projects: https://covidclinical.net
