ABSTRACT
In this study, we propose a scientific framework to detect capability among biomedical large language models (LLMs) for organizing expressions of comorbid disease and temporal progression. We hypothesize that biomedical LLMs pretrained on next-token prediction produce latent spaces that implicitly capture "disease states" and disease progression, i.e., the transitions over disease states over time. We describe how foundation models may capture and transfer knowledge from explicit pretraining tasks to specific clinical applications. A scoring function based on Kullback-Leibler divergence was developed to measure "surprise" in seeing specialization when subsetting admissions along 13 biomedical LLM latent spaces. By detecting implicit ordering of longitudinal data, we aim to understand how these models self-organize clinical information and support tasks such as phenotypic classification and mortality prediction. We test our hypothesis along a case study for obstructive sleep apnea (OSA) in the publicly available MIMIC-IV dataset, finding ordering of phenotypic clusters and temporality within latent spaces. Our quantitative findings suggest that increased compute, conformance with compute-optimal training, and widening contexts promote better implicit ordering of clinical admissions by disease states, explaining 60.3% of the variance in our proposed implicit task. Preliminary qualitative findings suggest LLMs’ latent spaces trace patient trajectories through different phenotypic clusters, terminating at end-of-life phenotypes. This approach highlights the potential of biomedical LLMs in modeling disease progression, identifying new patterns in disease pathways and interventions, and evaluating clinical hypotheses related to drivers of severe illness. We underscore the need for larger, high-resolution longitudinal datasets to further validate and enhance understanding of the utility of LLMs in modeling patient trajectories along clinical text and advancing precision medicine.
Question Do LLMs sensibly organize cilnical data with respect to applications in precision medicine?
Findings Biomedically-trained LLMs show increasing potential in promoting the organization of patient data to reflect disease progression. In a subcohort of OSA patients, maps derived from LLMs’ latent representations reveal traceable disease trajectories.
Meaning Maps of disease progression offer an explanation to the utility of LLMs in precision medicine. Following current pretraining conventions in foundation modeling, scientific inquiry into these maps may help anticipate progress in applications of LLMs for healthcare.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was supported in part by the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS) under the Science Undergraduate Laboratory Internship (SULI) Program and the Applied Mathematics and Computational Research Division (AMCR) of the Lawrence Berkeley National Lab. The authors would like to thank the National Energy Research Scientific Computing (NERSC) Center for the allocation of computing hours on the Perlmutter supercomputer.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Study used the public MIMIC-IV dataset.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Grammar errors and misspellings were corrected throughout the manuscript. Wording of theoretical concepts were adjusted in methods section " Scoring Implicit Ordering in Clinically-Aware Foundation Models ", specifically changing "intelligent designs" to "synthesized designs" as a more accurate descriptor for the concept. Wording of trajectory episodes were adjusted for accuracy in section "Trajectories Along Corpus Manifolds". The inclusion of results for a chart review was finished in section " Characterizing Drivers of Severe Illness in Obstructive Sleep Apnea". Adjustments of labels in the figure 5 were made to help the reader. Method sections were rearranged to improve readability. An additional figure for describing the framework was added (Figure 1). Names of section headers were adjusted to improve readability.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
