RT Journal Article
SR Electronic
T1 LCD Benchmark: Long Clinical Document Benchmark on Mortality Prediction for Language Models
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2024.03.26.24304920
DO 10.1101/2024.03.26.24304920
A1 Yoon, WonJin
A1 Chen, Shan
A1 Gao, Yanjun
A1 Zhao, Zhanzhan
A1 Dligach, Dmitriy
A1 Bitterman, Danielle S.
A1 Afshar, Majid
A1 Miller, Timothy
YR 2024
UL http://medrxiv.org/content/early/2024/07/02/2024.03.26.24304920.abstract
AB Objective The application of Natural Language Processing (NLP) in the clinical domain is important due to the rich unstructured information in clinical documents, which often remains inaccessible in structured data. When applying NLP methods to a certain domain, the role of benchmark datasets is crucial as benchmark datasets not only guide the selection of best-performing models but also enable the assessment of the reliability of the generated outputs. Despite the recent availability of language models (LMs) capable of longer context, benchmark datasets targeting long clinical document classification tasks are absent.Materials and Methods To address this issue, we propose LCD benchmark, a benchmark for the task of predicting 30-day out-of-hospital mortality using discharge notes of MIMIC-IV and statewide death data. We evaluated this benchmark dataset using baseline models, from bag-of-words and CNN to instruction-tuned large language models. Additionally, we provide a comprehensive analysis of the model outputs, including manual review and visualization of model weights, to offer insights into their predictive capabilities and limitations.Results and Discussion Baseline models showed 28.9% for best-performing supervised models and 32.2% for GPT-4 in F1-metrics. Notes in our dataset have a median word count of 1687. Our analysis of the model outputs showed that our dataset is challenging for both models and human experts, but the models can find meaningful signals from the text.Conclusion We expect our LCD benchmark to be a resource for the development of advanced supervised models, or prompting methods, tailored for clinical text.The benchmark dataset is available at https://github.com/Machine-Learning-for-Medical-Language/long-clinical-docCompeting Interest StatementThe authors have declared no competing interest.Funding StatementResearch reported in this publication was supported by the National Library Of Medicine of the National Institutes of Health under Award Number R01LM012973, and by the National Institute Of Mental Health of the National Institutes of Health under Award Number R01MH126977.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:IRB of Boston Children's Hospital gave ethical approval for this work (IRB number:IRB-P00028617). Under PhysioNet Credentialed Health Data Use Agreement 1.5.0 - Data Use Agreement for the MIMIC-IV (v2.2) and - Data Use Agreement for the MIMIC-IV-Note: Deidentified free-text clinical notes (v2.2) All authors are granted access to the database.I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.Yes