Abstract
Background The broad adoption of Electronic Health Records (EHRs) provides great opportunities to conduct health care research and solve various clinical problems in medicine. With recent advances and success, methods based on machine learning and deep learning have become increasingly popular in medical informatics. However, while many research studies utilize temporal structured data on predictive modeling, they typically neglect potentially valuable information in unstructured clinical notes. Integrating heterogeneous data types across EHRs through deep learning techniques may help improve the performance of prediction models.
Methods In this research, we proposed 2 general-purpose multi-modal neural network architectures to enhance patient representation learning by combining sequential unstructured notes with structured data. The proposed fusion models leverage document embeddings for the representation of long clinical note documents and either convolutional neural network or long short-term memory networks to model the sequential clinical notes and temporal signals, and one-hot encoding for static information representation. The concatenated representation is the final patient representation which is used to make predictions.
Results We evaluate the performance of proposed models on 3 risk prediction tasks (i.e., in-hospital mortality, 30-day hospital readmission, and long length of stay prediction) using derived data from the publicly available Medical Information Mart for Intensive Care III dataset. Our results show that by combining unstructured clinical notes with structured data, the proposed models outperform other models that utilize either unstructured notes or structured data only.
Conclusions The proposed fusion models learn better patient representation by combining structured and unstructured data. Integrating heterogeneous data types across EHRs helps improve the performance of prediction models and reduce errors.
Availability The code for this paper is available at: https://github.com/onlyzdd/clinical-fusion.
Background
Electronic Health Records (EHRs) are longitudinal electronic records of patients’ health information, including structured data (patient demographics, vital signs, lab tests, etc.) and unstructured data (clinical notes and reports). In the United States, for example, over 30 million patients visit hospitals each year, and the percent of non-Federal acute care hospitals with the adoption of at least a Basic EHR system increased from 9.4% to 83.8% over the 7 years between 2008 and 2015 [1]. The broad adoption of EHRs provides unprecedented opportunities for data mining and machine learning researchers to conduct health care research.
With recent advances and success, machine learning and deep learning-based approaches have become increasingly popular in health care and shown great promise in extracting insights from EHRs. Accurately predicting clinical outcomes, such as mortality and readmission prediction, can help improve health care and reduce cost. Traditionally, some knowledge-driven scores are used to estimate the risk of clinical outcomes. For example, SAPS scores [2] and APACHE IV [3] are used to identify patients at high risk of mortality; LACE Index [4] and HOSPITAL Score [5] are used to evaluate hospital readmission risk. Recently, lots of research studies have been conducted for these prediction tasks based on EHRs using machine learning and deep learning techniques. Caruana [6] predicts hospital readmission using traditional logistic regression and random forest models. Tang [7] shows that recurrent neural networks using temporal physiologic features from EHRs provide additional benefits in mortality prediction. Rajkomar [8] combines 3 deep learning models and develops an ensemble model to predict hospital readmission and long length of stay. Besides, Min [9] compared different types of machine learning models for predicting the readmission risk of Chronic Obstructive Pulmonary Disease patients. Two benchmarks studies [10, 11] show that deep learning models consistently outperform all the other approaches over several clinical prediction tasks. In addition to structured EHR data such as vital signs and lab tests, unstructured data also offers promise in predictive modeling [12, 13]. Boag [14] explores several representations of clinical notes and their effectiveness on downstream tasks. Liu’s model [15] forecasts the onset of 3 kinds of diseases using medical notes. Sushil [16] utilizes a stacked denoised autoencoder and a paragraph vector model to learn generalized patient representation directly from clinical notes and the learned representation is used to predict mortality.
However, most of the previous works focused on prediction modeling by utilizing either structured data or unstructured clinical notes and few of them pay enough attention to combining structured data and unstructured clinical notes. Integrating heterogeneous data types across EHRs (unstructured clinical notes, time-series clinical signals, static information, etc.) presents new challenges in EHRs modeling but may offer new potentials.
Recently, some works [15, 17] extract structured data as text features such as medical named entities and numerical lab tests from clinical notes and then combine them with clinical notes to improve downstream tasks. However, their approaches are domain-specific and cannot easily be transferred to other domains. Besides, their structured data are extracted from clinical notes and may introduce errors compared to original signals.
In this paper, we aim at combining structured data and unstructured text directly through deep learning techniques for clinical risk predictions. Deep learning methods have made great progress in many areas [18] such as computer vision [19], speech recognition [20] and natural language processing [21] since 2012. The flexibility of deep neural networks makes it well-suited for the data fusion problem of combining unstructured clinical notes and structured data. Here, we propose 2 multi-modal neural network architectures learn patient representation and the patient representation is then used to predict patient outcomes. The proposed multi-modal neural network architectures are purposegeneral and can be applied to other domains without effort.
To summarize, the contributions of our work are:
We propose 2 general-purpose fusion models to combine temporal signals and clinical text which lead to better performance on 3 prediction tasks.
We examine the capability of unstructured clinical text in predictive modeling.
We present benchmark results of in-hospital mortality, 30-day readmission and long length of stay prediction tasks. We show that deep learning models consistently outperform baseline machine learning models.
We compare and analyze the running time of proposed fusion models and baseline models.
Methods
In this section, we describe the dataset, patient features, predictive tasks, and proposed general-purpose neural network architectures for combining unstructured data and structured data using deep learning techniques.
Dataset description
Medical Information Mart for Intensive Care III (MIMIC-III) [22] is a publicly available critical care database maintained by the Massachusetts Institute of Technology (MIT)’s Laboratory for Computational Physiology. MIMIC-III comprises deidentified health-related data associated with over forty thousand patients who stayed in critical care units of the Beth Israel Deaconess Medical Center (BIDMC) between 2001 and 2012. This database includes patient health information such as demographics, vital signs, lab test results, medications, diagnosis codes, as well as clinical notes.
Patient features
Patient features consist of features from both structured data (static information and temporal signals) and unstructured data (clinical text). In this part, we describe the patient features that are utilized by our model and some data preprocessing details.
Static information
Static information refers to demographic information and admission-related information in this study. For demographic information, patient’s age, gender, marital status, ethnicity, and insurance information are considered. Only adult patients are enrolled in this study. Hence, age was split into 5 groups (18, 25), (25, 45), (45, 65), (65, 89), (89,). For admission-related information, admission type is included as features.
Temporal signals
For temporal signals, we consider 7 frequently sampled vital signs: heart rate, systolic blood pressure (SysBP), diastolic blood pressure (DiasBP), mean arterial blood pressure (MeanBP), respiratory rate, temperature, SpO2; and 19 common lab tests: anion gap, albumin, bands, bicarbonate, bilirubin, creatinine, chloride, glucose, hematocrit, hemoglobin, lactate, platelet, potassium, partial thromboplastin time (PTT), international normalized ratio (INR), prothrombin time (PT), sodium, blood urea nitrogen (BUN), white blood cell count (WBC). Statistics of temporal signals are shown in Table 1. Additional statistics of temporal signals are provided in Supplementary Materials. After feature selection, we extract values of these time-series features up to the first 24 hours of each hospital admission. For each temporal signal, average is used to represent the signal at each timestep (hour). Then, each temporal variable was normalized using min-max normalization. To handle missing values, we simply use “0” to impute [23].
Label statistics and characteristics of 3 prediction tasks. SD represents standard deviation.
Sequential clinical notes
In addition to the aforementioned types of structured data, we also incorporate sequential unstructured notes, which contain a vast wealth of knowledge and insight that can be utilized for predictive models using Natural Language Processing (NLP). In this study, we considered Nursing, Nursing/Other, Physician, and Radiology notes, because these kinds of notes are in the majority of clinical notes and are frequently recorded in MIMIC-III database. We only extract the first 24 hours’ notes for each admission to enable early prediction of outcomes.
Predictive tasks
Here, 3 benchmark prediction tasks are adopted which are crucial in clinical data problems and have been well studied in the medical community [7, 8, 24, 25, 26, 27, 28].
In-hospital mortality prediction
Mortality prediction is recognized as one of the primary outcomes of interest. The overall aim of this task is to predict whether a patient passes away during the hospital stay. This task is formulated as a binary classification problem, where the label indicates the occurrence of a death event. To evaluate the performance, we report the F1-score (F1), area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve (AUPRC). AUROC is the main metric.
Long length of stay prediction
Length of stay is defined as the time interval between hospital admission and discharge. In the second task, we predict a long length of stay whether a length of stay is more than 7 days [8, 28]. Long length of stay prediction is important for hospital management. This task is formulated as a binary classification problem with the same metrics of the mortality prediction task.
Hospital readmission prediction
Hospital readmission refers to unplanned hospital admissions within 30 days following the initial discharge. Hospital readmission has received much attention because of its No impacts on healthcare systems’ budgets. In the US, for example, roughly 2 million hospital readmissions each year costs Medicare 27 billion dollars, of which 17 billion dollars are potentially avoidable [29]. Reducing preventable hospital readmissions represents an opportunity to improve health care, lower costs, and increase patient satisfaction. Predicting unplanned hospital readmission is a binary classification problem with the same metrics as the in hospital mortality prediction task.
Neural Network Architecture
In this part, we present 2 neural network architectures for combining clinical structured data with sequential clinical notes. The overview of the proposed models, namely Fusion-CNN and Fusion-LSTM, are illustrated in Figure 1 and Figure 2. Each model mainly consists of 5 parts, static information encoder, temporal signals embedding, sequential notes representation, patient representation, and output layer. Fusion-CNN is based on convolutional neural networks (CNN) and Fusion-LSTM is based on long-short term memory (LSTM) networks. The 2 models have common in model inputs and outputs but differ in the way how they model the temporal information.
Architecture of CNN-based Fusion-CNN. Fusion-CNN uses document embeddings, 2-layer CNN and max-pooling to model sequential clinical notes. Similarly, 2-layer CNN and max-pooling are used to model temporal signals. The final patient representation is the concatenation of the latent representation of sequential clinical notes, temporal signals, and the static information vector. Then the final patient representation is passed to output layers to make predictions.
Static information encoder
The static categorical features including patient demographics and admission-related information are encoded as one-hot vectors through the static information encoder. The output of the encoder is zstatic = [zdemo; zadm] with size dstatic = ddemo + dadm and zstatic is part of patient representation.
Temporal signals representation
In this part, Fusion-CNN and Fusion-LSTM leverage different techniques to model temporal signals. The learned vector for temporal signals representation is ztemporal with size of dtemporal.
Fusion-CNN Convolutional neural networks (CNNs) can automatically learn the features through convolution and pooling operations and can be used for time-series modeling. Fusion-CNN uses 2-layer convolution and max-pooling to extract deep features of temporal signals as shown in Figure 1.
Fusion-LSTM Recurrent neural networks (RNNs) are considered since RNN models have achieved great success in sequences and time series data modeling. However, RNNs with simple activations suffer from vanishing gradients. Long short-term memory (LSTM) neural networks are a type of RNNs that can learn and remember long sequences of input data. 2-layer LSTM is utilized in Fusion-LSTM model to learn the representations of temporal signals as shown in Figure 2. To prevent the model from overfitting, dropout on nonrecurrent connections is applied between RNN layers and before outputs.
Architecture of LSTM-based Fusion-LSTM. Fusion-LSTM uses document embeddings, a BiLSTM layer, and a max-pooling layer to model sequential clinical notes. 2-layer LSTMs are used to model temporal signals. The concatenated patient representation is passed to output layers to make predictions.
Sequential notes representation
Word embedding is a popular technique in natural language processing that is used to map words or phrases from vocabulary to a corresponding vector of continuous values. However, directly modeling sequential notes using word embeddings and deep learning can be time-consuming and may not be practical since clinical notes are usually very long and involve multiple timestamps. To solve this problem, we present the sequential notes representation component based on document embeddings. Here, we utilize paragraph vector (aka. Doc2Vec) [30] to learn the embedding of each clinical note. Time-series document embeddings are inputs to Fusion-CNN and Fusion-LSTM as shown in Figure 1 and Figure 2. The sequential notes representation component produces znote with size of dnote as the latent representation of sequential notes.
Fusion-CNN As shown in Figure 1, the sequential notes representation part of Fusion-CNN model is made up of document embeddings, a series of convolutional layers and max-pooling layers, and a flatten layer. Document embedding inputs are passed to these convolutional layers and max-pooling layers. The flatten layer takes the output of the max-pooling layer as input and outputs the final text representation.
Fusion-LSTM model is demonstrated in Figure 2, the sequential notes representation part is made up of document embeddings, a BiLSTM (bidirectional LSTM) layer, and a max-pooling layer. The document embedding inputs are passed to the BiL-STM layer. The BiLSTM layer concatenates the outputs (
, 
) from 2 hidden layers of opposite direction to the same output 
and can capture long term dependencies in sequential text data. The max-pooling layer takes the hidden states of the BiLSTM layer as input and outputs the final text representation.
Patient representation
The final patient representation z is obtained by concatenating the representations of clinical text, temporal signals, along with static information. The representation of each patient is zp = [zstatic; ztemporal; ztext], the size of this vector is dstatic + dtemporal + dtext. The patient representation is then fed to a final output layer to make predictions.
Output layer
The output layer takes patient representation as input and makes predictions. For each patient representation zp, we have a task-specific target y. y ∊ {0, 1} is a single binary label indicating whether the in-hospital mortality, 30-day readmission, or long length of stay event occurs.
For each prediction task, the output layer receives an instance of patient representation zp as input and tries to predict the ground truth y. For binary classification tasks, the output layer is:
The W matrices and b vectors are the trainable parameters, σ represents a sigmoid activation function.
For each of these 3 tasks, the loss functions is defined as the binary cross entropy loss:
Results and Discussion
Experiment setup
Cohort preparation
Based on the MIMIC-III dataset, we evaluated our proposed models on 3 predictive tasks (i.e. in-hospital mortality prediction, 30-day readmission prediction, and long length of stay prediction). To build corresponding cohorts, we first removed all patients whose age <18 years old and all hospital admissions whose length of stay is less than 1 day. Besides, patients without any records of required temporal signals and clinical notes were removed. In total, 39,429 unique admissions are eligible for prediction tasks. Label statistics and characteristics of 3 prediction tasks are provided in Table 1.
Implementation details
In this part, we describe the implementation details. We train the unsupervised Doc2Vec model on the training set to obtain the document-level embeddings for each note using the popular Gensim toolkit [31]. We use PV-DBOW as the training algorithm, number of training epochs as 30, initial learning rate as 0.025, learning rate decay as 0.0002, and dimension of vectors as 200 to train.
We implement baseline models (i.e., logistic regression and random forest) with scikit-learn [32]. Deep learning models are implemented using PyTorch [33]. All deep learning models are trained with Adam optimizer with a learning rate of 0.0001 and ReLU as the activation function. The batch size is chosen as 64 and the max epoch number is set to 50.
For evaluation, 70% of the data are used for training, and 10% for validation, 20% for testing. For binary classification tasks, AUROC is used as the main metric. Besides, we report F1 score, and AUPRC to aid the interpretation of AUROC for imbalanced datasets.
Baselines
We compared our model with the following baseline methods: logistic regression (LR), random forest (RF). Because these standard machine learning methods cannot work directly with temporal sequences, the element-wise mean vector across sequential notes and aggregations (i.e. mean value, minimum value, maximum value, standard deviation, and count of observations) of temporal signals are used as model inputs.
Ablation study
To evaluate the contribution of different components and gain a better understanding of the proposed fusion model’s behavior, an ablation study is adopted and we have conducted extensive experiments on different models. Let U, T, S denote the unstructured clinical notes, temporal signals, and static information.
Results
In this section, we report the performance of proposed models on 3 prediction tasks. The results are shown in Table 2, Table ??, and Table 6. Each reported performance metric is the average score of 5 runs with different data splits. To measure the uncertainty of a trained model’s performance, we calculated 95% confidence interval using t-distribution and the results are reported. Besides, to better compare model performances on each task, we performed statistical testing and calculated P-value of AUROC score across various models using statistical t-testingP-value matrix of AUROC scores on in-hospital mortality prediction, long length of stay prediction, 30-day readmission prediction tasks are shown in Table 3, Table 5, and Table 7. In summary, the results shows significant improvements and it matches our expectations: (1) Deep learning models outperformed traditional machine learning models by comparing the performances of different models on the same model inputs. (2) Models could make more accurate predictions by combining unstructured text and structured data.
In-hospital mortality prediction on MIMIC-III. U, T, S represents unstructured data, temporal signals, and static information respectively.
P-value matrix of various model performances (AUROC) for in-hospital mortality prediction. U, T, S represents unstructured data, temporal signals, and static information respectively.
In-hospital mortality prediction
Table 2 shows the performance of various models on the in-hospital mortality prediction task. From Table 2 and Table 3, deep models outperformed baseline models. We speculate the main reasons why deep models work better are two-fold: (1) Deep models can automatically learn better patient representations as the network grows deeper and yield more accurate predictions. (2) Deep models can capture temporal information and local patterns, while logistic regression and random forest simply aggregate time-series features and hence suffer from information loss.
For each kind of classifier, the performance of classifier trained on all data (U+T+S) is significantly higher than that trained on either structured data (T + S) or unstructured data (U) only. Especially by considering unstructured text, the AUROC score of Fusion-CNN and Fusion-LSTM increased by 0.043 and 0.034 respectively. Structured data contains a patient’s vital signs and lab test results, while sequential notes provide the patient’s clinical history including diagnoses, medications, and so on. This observation implicitly explains why unstructured text and structured data can complement each other to some extent in predictive modeling which leads to performance improvement.
Long length of stay prediction
Table 4 shows the performance measured by F1, AUROC, and AUPRC of different models on the long length of stay prediction. We observe that (1) Logistic regression serves as a very strong baseline while Fusion-CNN achieves a slightly higher F1 score and AUROC score than Fusion-LSTM. (2) By integrating multi-modal information, all models yield more accurate predictions and the improvement is significant as shown in Table 5.
Long length of stay prediction on MIMIC-III. U, T, S represents unstructured data, temporal signals, and static information respectively.
P-value matrix of various model performances (AUROC) for long length of stay prediction. U, T, S represents unstructured data, temporal signals, and static information respectively.
Hospital readmission prediction
Table 6 and Table 7 summarize the results of various approaches of the hospital readmission task. For this task, logistic regression performed well but random forest performed badly. Fusion-CNN and Fusion-LSTM yielded comparably better predictions of AU-ROC score around 0.67. Incorporating clinical notes led to performance improvement for logistic regression, Fusion-CNN, and Fusion-LSTM. However, combining unstructured notes with structured data hurt the performance of random forest.
We noted the AUROC score for hospital readmission prediction is significantly lower than in-hospital mortality one which means readmission risk modeling is more complex and difficult compared to in hospital mortality prediction. This is probably because the given features are inadequate for building a good hospital readmission risk prediction model. Besides, we only used the first day’s data which is far away from patient discharge that may not be very helpful in readmission prediction modeling.
30-day readmission prediction on MIMIC-111. U, T, S represents unstructured data, temporal signals, and static information respectively.
P-value matrix of various model performances (AUROC) for 30-day readmission prediction. U, T, S represents unstructured data, temporal signals, and static information respectively.
Discussion
In this study, we examined proposed fusion models on 3 outcome prediction tasks, namely mortality prediction, long length of stay prediction, and readmission prediction. The results showed that deep fusion models (Fusion-CNN, Fusion-LSTM) outperformed baselines and yielded more accurate predictions by incorporating unstructured text.
In 3 tasks, logistic regression was a quite strong baseline and was consistently more useful than random forest. Deep models achieved the best performance for each task while training time of deep models is also acceptable as demonstrated in Figure 3. All experiments were performed on a 32-core Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz machine with NVIDIA TITAN RTX GPU processor. For a fair comparison, we report the training time per epoch for Fusion-CNN and Fusion-LSTM.
Comparison of model running time with different inputs.
Conclusion
In this paper, we proposed 2 multi-modal deep neural networks that learn patient representation by combining unstructured clinical text and structured data. The 2 models make use of either LSTMs or CNNs to model temporal information. The proposed models are quite general data fusion methods and can be which mainly due to the learned patient representation consists of medication, diagnosis information from sequential unstructured notes, and vital signs, lab test results from structured data. In future work, we plan to apply the proposed fusion methods to more real-world applications.
Data Availability
MIMIC-III database analyzed in the study is available on PhysioNet repository.
Additional Files
Additional File 1 — Additional statistics.
Figure S1. Length of stay distribution of the processed MIMIC-III cohort.
Table S1. Statistics of collected vital signs and lab tests.
Author’s contributions
PZ conceived the project. DZ and PZ developed the method. DZ conducted the experiments. DZ, CY, JZ, and PZ analyzed experimental results. DZ, CY, JZ, XY, and PZ wrote the manuscript. All authors read and approved the final manuscript.
Funding
This project was funded in part under a grant with Lyntek Medical Technologies, Inc.
Availability of data and materials
MIMIC-III database analyzed in the study is available on PhysioNet repository. The source code is provided for reproducing and is available at https://github.com/onlyzdd/clinical-fusion.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
PZ is the member of the editorial board of BMC Medical Informatics and Decision Making. The authors declare that they have no other competing interests.
Acknowledgements
Not applicable.
References
