Abstract
Introduction Asymptomatic left ventricular dysfunction (ALVD) carries an increased risk for overt heart failure and mortality, yet treatable to mitigate disease progression. An artificial intelligence (AI)-enabled 12-lead electrocardiogram (ECG) model demonstrated promise in ALVD screening but an unexpected drop in performance was observed in external validation. We thus sought to train a de novo model for ALVD detection and investigate its performance across multiple institutions and across a broader set of patient strata.
Methods ECGs taken within 14 days of an echocardiogram were obtained from 4 academic hospitals from the United States (BWH, MGH and UCSF) and Japan (Keio). Four AI models were trained to detect patients with ejection fraction < 40% using ECG from each of the 4 institutions. All the models were then evaluated on the held-out test dataset from the same intuition and all data from the 3 external institutions. Subgroup analyses stratified by patient characteristics and common overt ECG abnormalities were performed.
Results A total of 221,846 ECGs were identified from the 4 institutions. While the BWH-trained and Keio-trained model yielded similar accuracy on its internal test data (AUROC 0.913 and 0.914 respectively), external validity was worse for Keio-trained model (AUROC: 0.905-0.915 for BWH-trained and 0.849-0.877 for Keio-trained model). Although ECG abnormalities including atrial fibrillation, left bundle branch block and paced rhythm reduced detection, the models performed robustly across patient characteristics and other ECG features.
Conclusion Different dataset produces models with different performance highlighting the importance of external validation and extensive stratification analysis.
Competing Interest Statement
RCD is supported by grants from the National Institute of Health, the American Heart Association (One Brave Idea, Apple Heart and Movement Study) and is a cofounder of Atman Health. CAM is supported by grants from the National Institute of Health, the American Heart Association (One Brave Idea, Apple Heart and Movement Study), is a consultant for Pfizer, Clarify Health, Dr. Evidence and Foresite Labs, and is a co-founder of Atman Health. All other authors declare no competing interests.
Funding Statement
This work was supported by One Brave Idea, co-founded by the American Heart Association and Verily with significant support from AstraZeneca and pillar support from Quest Diagnostics (to CAM and RCD), NIH/NHLBI HL140731 (to RCD). SG and MH are supported by the Drs. Morton and Toby Mower Science Innovation Fund Fellowship. SG is partially supported by a grant from The Japanese Society on Thrombosis and Hemostasis.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study complies with all ethical regulations and guidelines. The study protocol was approved by local institutional review boards (IRB) of Mass General Brigham (2019P002651), University of California San Francisco (10-03386), and Keio University (20200030). Since the study collected data retrospectively a waiver of informed consent was approved by the IRB.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
Code availability. The code for training and testing the model is provided at https://github.com/obi-ml-public/ECG-LV-Dysfunction. The model weights may contain personal information from patients and thus, are not shared. We provide a web-interface to run our model and generate predictions at http://onebraveideaml.org
The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, University of Washington, and Vrije Universiteit Amsterdam.
Subject Area
Reviews and Context
0
Comment
0
TRIP Peer Reviews
0
Community Reviews
0
Automated Services
0
Blogs/Media
Author Videos