RT Journal Article
SR Electronic
T1 Development and Evaluation of Machine Learning Models for the Detection of Emergency Department Patients with Opioid Misuse from Clinical Notes
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2024.12.11.24318875
DO 10.1101/2024.12.11.24318875
A1 Shahid, Usman
A1 Parde, Natalie
A1 Smith, Dale L.
A1 Dickinson, Grayson
A1 Bianco, Joseph
A1 Thorpe, Dillon
A1 Hota, Madhav
A1 Afshar, Majid
A1 Karnik, Niranjan S.
A1 Chhabra, Neeraj
YR 2024
UL http://medrxiv.org/content/early/2024/12/12/2024.12.11.24318875.abstract
AB Objectives The accurate identification of Emergency Department (ED) encounters involving opioid misuse is critical for health services, research, and surveillance. We sought to develop natural language processing (NLP)-based models for the detection of ED encounters involving opioid misuse.Methods A sample of ED encounters enriched for opioid misuse was manually annotated and clinical notes extracted. We evaluated classic machine learning (ML) methods, fine-tuning of publicly available pretrained language models, and a previously developed convolutional neural network opioid classifier for use on hospitalized patients (SMART-AI). Performance was compared to ICD-10-CM codes. Both raw text and text transformed to the United Medical Language System were evaluated. Face validity was evaluated by term feature importance.Results There were 1123 encounters used for training, validation, and testing. Of the classic ML methods, XGBoost had the highest AU_PRC (0.936), accuracy (0.887), and F1 score (0.863) which outperformed ICD-10-CM codes [accuracy 0.870; F1 0.830]. Logistic regression, support vector machine, and XGBoost models had higher AU_PRC using transformed text, while decision trees performed better using raw text. Excluding XGBoost, fine-tuned pre-trained language models outperformed classic ML methods. The best performing model was the fine-tuned SMART-AI based model with domain adaptation [AU_PRC 0.948; accuracy 0.882; F1 0.851]. Explainability analyses showed the most predictive terms were ‘heroin’, ‘opioids’, ‘alcoholic intoxication, chronic’, ‘cocaine’, ‘opiates’, and ‘suboxone’.Conclusions NLP-based models outperform entry of ICD-10-CM diagnosis codes for the detection of ED encounters with opioid misuse. Fine tuning with domain adaptation for pre-trained language models resulted in improved performance.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis work was supported by the following grants from the National Institute on Drug Abuse (NIDA)/NIH: K23DA055061 (Chhabra), R01DA051464 (Afshar), R61DA057629 (Karnik/Chhabra) and, in part, by the National Center for Advancing Translational Sciences (NCATS), through Grant UL1TR002003.The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This study was reviewed and exempted from review as non-human subjects research by the institutional review board of the University of Illinois ChicagoI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesAll data produced in the present study are available upon reasonable request to the authors subject to patient privacy protections