Abstract
Background and Aims Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLM) like BERT have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event detection.
Methods We adapted a new clinical LLM, UCSF BERT, to identify serious adverse events (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection.
Results We annotated 928 outpatient IBD notes corresponding to 928 individual IBD patients for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of 8 candidate models, UCSF BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF BERT was significantly superior at identifying hospitalization events emergent to medication use (p < 0.01).
Conclusions LLMs like UCSF BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared to prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multi-center data and newer architectures like GPT. Our findings support the potential value of using large language models to enhance pharmacovigilance.
Competing Interest Statement
Associations with commercial entities that provided support for the work are reported in the submitted manuscript.
Funding Statement
This publication was supported by the Food and Drug Administration (FDA) of the U.S. Department of Health and Human Services (HHS) as part of a financial assistance award Center of Excellence in Regulatory Science and Innovation grant to University of California, San Francisco, U01FD005978, totaling $79,250 with 33% percentage funded by FDA/HHS and $158,500, 66% percentage funded by the UCSF Division of Gastroenterology and UCSF Bakar Computational Health Sciences Institute, and 1% funded by the National Library of Medicine of the National Institutes of Health under Award Number K99LM014099. Additional support for clinical data resources were provided by National Center for Advancing Translational Sciences, National Institutes of Health, through UCSF-CTSI Grant Number UL1TR001872. The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement, by HHS or the U.S. Government.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This single-center study of natural language processing algorithms for adverse event detection was approved by the University of California San Francisco Institutional Review Board (#18-24588).
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Data Availability
The analytic code to train and evaluate models will be made publicly available at https://github.com/MadhumitaSushil/ADE_detection. A machine-redacted version of the notes-based data can be made available to requesting researchers by mutual agreement and following the execution of a data use agreement.
