PT  - JOURNAL ARTICLE
AU  - Thompson, Daniel
AU  - Mofidi, Reza
TI  - Development and comparison of natural language processing models for abdominal aortic aneurysm repair identification and classification using unstructured electronic health records
AID  - 10.1101/2024.12.11.24318852
DP  - 2024 Jan 01
TA  - medRxiv
PG  - 2024.12.11.24318852
4099  - http://medrxiv.org/content/early/2024/12/12/2024.12.11.24318852.short
4100  - http://medrxiv.org/content/early/2024/12/12/2024.12.11.24318852.full
AB  - Background Patient identification for national registries often relies upon clinician recognition of cases or retrospective searches using potentially inaccurate clinical codes, potentially leading to incomplete data capture and inefficiencies. Natural Language Processing (NLP) offers a promising solution by automating analysis of electronic health records (EHRs). This study aimed to develop NLP models for identifying and classifying abdominal aortic aneurysm (AAA) repairs from unstructured EHRs, demonstrating proof-of-concept for automated patient identification in registries like the National Vascular Registry.Method Using the MIMIC-IV-Note dataset, a multi-tiered approach was developed to identify vascular patients (Task 1), AAA repairs (Task 2), and classify repairs as primary or revision (Task 3). Four NLP models were trained and evaluated using 4,870 annotated records: scispaCy, BERT-base, Bio-clinicalBERT, and a scispaCy/Bio-clinicalBERT ensemble. Models were compared using accuracy, precision, recall, F1-score, and area under the receiver operating characteristic curve.Results The scispaCy model demonstrated the fastest training (2 mins/epoch) and inference times (2.87 samples/sec). For Task 1, scispaCy and ensemble models achieved the highest accuracy (0.97). In Task 2, all models performed exceptionally well, with ensemble, scispaCy, and Bio-clinicalBERT models achieving 0.99 accuracy and 1.00 AUC. For Task 3, Bio-clinicalBERT and the ensemble model achieved an AUC of 1.00, with Bio-clinicalBERT displaying the best overall accuracy (0.98).Conclusion This study demonstrates that NLP models can accurately identify and classify AAA repair cases from unstructured EHRs, suggesting significant potential for automating patient identification in vascular surgery and other medical registries, reducing administrative burden and improving data capture for audit and research.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis study did not receive any fundingAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesData/code availability: Models are available for download and use under GNU General Public License v3.0 at https://huggingface.co/danielcthompson with demonstration scripts available at https://github.com/dc-thompson/AAA_classification. The MIMIC IV dataset used for training is available at https://physionet.org/content/mimic-iv-note/2.2/. The underlying code and annotated dataset for model training used in this study is not publicly available but may be made available to qualified researchers on reasonable request to the corresponding author.https://physionet.org/content/mimic-iv-note/2.2/https://github.com/dc-thompson/AAA_classificationhttps://huggingface.co/danielcthompson