ABSTRACT
Background ANCA-associated vasculitis (AAV) is a rare but serious disease. Traditional case-identification methods using claims data can be time-intensive and may miss important subgroups. We hypothesized that a deep learning model analyzing electronic health records (EHR) can more accurately identify AAV cases.
Methods We examined the Mass General Brigham (MGB) repository of clinical documentation from 12/1/1979 to 5/11/2021, using expert-curated keywords and ICD codes to identify a large cohort of potential AAV cases. Three labeled datasets (I, II, III) were created, each containing note sections. We trained and evaluated a range of machine learning and deep learning algorithms for note-level classification, using metrics like positive predictive value (PPV), sensitivity, F-score, area under the receiver operating characteristic curve (AUROC), and area under the precision and recall curve (AUPRC). The deep learning model was further evaluated for its ability to classify AAV cases at the patient-level, compared with rule-based algorithms in 2,000 randomly chosen samples.
Results Datasets I, II, and III comprised 6,000, 3,008, and 7,500 note sections, respectively. Deep learning achieved the highest AUROC in all three datasets, with scores of 0.983, 0.991, and 0.991. The deep learning approach also had among the highest PPVs across the three datasets (0.941, 0.954, and 0.800, respectively). In a test cohort of 2,000 cases, the deep learning model achieved a PPV of 0.262 and an estimated sensitivity of 0.975. Compared to the best rule-based algorithm, the deep learning model identified six additional AAV cases, representing 13% of the total.
Conclusion The deep learning model effectively classifies clinical note sections for AAV diagnosis. Its application to EHR notes can potentially uncover additional cases missed by traditional rule-based methods.
SIGNIFICANCE AND INNOVATIONS
– Traditional approaches to identifying AAV cases for research have relied on registries assembled through clinical care and/or on billing codes which may miss important subgroups.
– Unstructured data entered as free text by clinicians document a patient’s diagnosis, symptoms, manifestations, and other features of their condition which may be useful for identifying AAV cases
– We found that a deep learning approach can classify notes as being indicative of AAV and, when applied at the case level, identifies more cases with AAV than rule-based algorithms.
Competing Interest Statement
ZSW reports research support from Amgen, Bristol-Myers Squibb, and Principia/Sanofi, consulting fees from Amgen, Viela Bio, Horizon, Zenas Biopharma, PPD, and MedPace, and has served on advisory boards for Amgen, Horizon, Novartis, Sanofi, Shinogi, and Visterra/Otsuka. All other authors report no disclosures.
Funding Statement
Authors of this work are supported by the National Institutes of Health [grant numbers K23AR073334, R03AR078938, L30AR070520, R03AR078938, K99AG075190]. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The study was approved by the Mass General Brigham Institutional Review Board with a protocol number of 2016P000633.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
jlaurentiev{at}bwh.harvard.edu, lzhou{at}bwh.harvard.edu, ccook13{at}mgh.harvard.edu, ssrivatsan1{at}mgh.harvard.edu, hchoi{at}mgh.harvard.edu, zswallace{at}mgh.harvard.edu, yining_hua{at}hms.harvard.edu, jyang66{at}bwh.harvard.edu, emiloslavsky{at}mgh.harvard.edu
Funding support: Authors of this work are supported by the National Institutes of Health [grant numbers K23AR073334, R03AR078938, L30AR070520, R03AR078938, K99AG075190]. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Conflict of Interest: ZSW reports research support from Amgen, Bristol-Myers Squibb, and Principia/Sanofi, consulting fees from Amgen, Viela Bio, Horizon, Zenas Biopharma, PPD, and MedPace, and has served on advisory boards for Amgen, Horizon, Novartis, Sanofi, Shinogi, and Visterra/Otsuka. All other authors report no disclosures.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
