RT Journal Article
SR Electronic
T1 From Text to Tables: A Local Privacy Preserving Large Language Model for Structured Information Retrieval from Medical Documents
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2023.12.07.23299648
DO 10.1101/2023.12.07.23299648
A1 Wiest, Isabella C.
A1 Ferber, Dyke
A1 Zhu, Jiefu
A1 van Treeck, Marko
A1 Meyer, Sonja K.
A1 Juglan, Radhika
A1 Carrero, Zunamys I.
A1 Paech, Daniel
A1 Kleesiek, Jens
A1 Ebert, Matthias P.
A1 Truhn, Daniel
A1 Kather, Jakob Nikolas
YR 2023
UL http://medrxiv.org/content/early/2023/12/08/2023.12.07.23299648.abstract
AB Background and Aims Most clinical information is encoded as text, but extracting quantitative information from text is challenging. Large Language Models (LLMs) have emerged as powerful tools for natural language processing and can parse clinical text. However, many LLMs including ChatGPT reside in remote data centers, which disqualifies them from processing personal healthcare data. We present an open-source pipeline using the local LLM “Llama 2” for extracting quantitative information from clinical text and evaluate its use to detect clinical features of decompensated liver cirrhosis.Methods We tasked the LLM to identify five key clinical features of decompensated liver cirrhosis in a zero- and one-shot way without any model training. Our specific objective was to identify abdominal pain, shortness of breath, confusion, liver cirrhosis, and ascites from 500 patient medical histories from the MIMIC IV dataset. We compared LLMs with three different sizes and a variety of pre-specified prompt engineering approaches. Model predictions were compared against the ground truth provided by the consent of three blinded medical experts.Results Our open-source pipeline yielded in highly accurate extraction of quantitative features from medical free text. Clinical features which were explicitly mentioned in the source text, such as liver cirrhosis and ascites, were detected with a sensitivity of 100% and 95% and a specificity of 96% and 95%, respectively from the 70 billion parameter model. Other clinical features, which are often paraphrased in a variety of ways, such as the presence of confusion, were detected only with a sensitivity of 76% and a specificity of 94%. Abdominal pain was detected with a sensitivity of 84% and a specificity of 97%. Shortness of breath was detected with a sensitivity of 87% and a specificity of 96%. The larger version of Llama 2 with 70b parameters outperformed the smaller version with 7b parameters in all tasks. Prompt engineering improved zero-shot performance, particularly for smaller model sizes.Conclusion Our study successfully demonstrates the capability of using locally deployed LLMs to extract clinical information from free text. The hardware requirements are so low that not only on-premise, but also point-of-care deployment of LLMs are possible.Lay summary We leveraged the large language model Llama 2 to extract five key features of decompensated liver cirrhosis from medical history texts, simplifying the analysis of complex text-based healthcare data.Competing Interest StatementJakob Nikolas Kather declares consulting services for Owkin, France; DoMore Diagnostics, Norway; Panakeia, UK; Scailyte, Basel, Switzerland; and Mindpeak, Hamburg, Germany; furthermore JNK holds shares in Kather Consulting, Dresden, Germany; and StratifAI GmbH, Dresden, Germany, and has received honoraria for lectures and advisory board participation by AstraZeneca, Bayer, Eisai, MSD, BMS, Roche, Pfizer and Fresenius. Daniel Truhn has received honoraria for lectures for Bayer and holds shares in StratifAI GmbH, Dresden, Germany. The authors have no other financial or non-financial conflicts of interest to disclose.Funding StatementEuropean Union Horizon Europe and innovation programmeAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:The study used ONLY openly available human data that were originally located at: pyhsionet.orgI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesWe used the MIMIC-IV dataset which is available after request from physionet.org. https://github.com/I2C9W/fromtexttotables/releases/tag/v0.5.0