Multimodal deep learning enhances diagnostic precision in left ventricular hypertrophy

Data Acquisition and selection

With the approval of Stanford Institutional Review Board (IRB), we retrieved electrocardiograms and echocardiograms from patients between 2006 and 2018 at Stanford Medicine (Table 1). The data was split into training, validation, and test sets with no patient overlap between sets. Due to the fact that multiple electrocardiograms and echocardiograms are present within the healthcare system record, we explored various data selection scenarios to understand what selection methods are best suited for this specific task. The quantitative comparison of all data selection used can be found in Supplementary Table S1. The final model was trained using a patient’s first ECG and first echocardiogram in the system.

Model performance

Four multimodal fusion models were explored: late-average, late-ranked, pre-trained late fusion and random late fusion (Figure 1). The performance metrics of each model is detailed in Table 2. The late average model achieved the highest F1-score and specificity rates 0.711 (0.571 - 0.826) and 0.952 (0.921 - 0.979) respectively on the held-out test set. We conducted experiments to study the performance of single-modal models trained on only ECG and echo to demonstrate the benefit of multimodal models. The multimodal models outperform single-modal model F1-scores, which increase from 0.63 to 0.71. Furthermore, the false-discovery rates are significantly reduced from 0.45 to 0.3. To provide context for these results, we also trained the single-modal models to predict left ventricular etiology using standard quantitative features from the electrocardiogram. This baseline model achieved sensitivity rates of 0.51 for predicting HCM which is considerably lower than LVH-Fusion (Supplementary Table S2). These results show that the proposed electrocardiogram signals model discover novel characteristics not accounted for with the quantitative features. Lastly, to examine the discriminatory power of our methodology, we performed a sensitivity analysis for predicting LVH etiology including the additional classification task of “normal.” In this context, LVH-fusion maintains high discriminatory power in predicting LVH from normal ECG and echocardiogram videos, suggesting that false positive rates of hypertension or hypertrophic cardiomyopathy would be low if the model was extended to this use case (Supplementary Table S3 and S4).

Understanding model performance

In order to improve our understanding of how LVH-Fusion classifies left ventricular etiology, we implemented a series of ablation studies similar to Hughes et al. 2021¹⁸ to determine what information models rely on to make predictions. For electrocardiogram single-modal models we examined the impact of varying the number of leads from the standard 12 leads to 8 leads and masking each lead to understand the impact each lead holds for prediction estimates. We find that although no single lead harbors a statistically significant impact on the overall model performance, masking out lead V3 and aVR had the highest negative impact on prediction estimates, Figure 2. Next, since the standard 12 lead ECG contains 8 algebraically independent leads, we considered the impact of masking multiple leads combinations. We observe an overall reduction in classification metrics when masking multiple leads at a time with no significant difference between masking the 4 dependent leads (III, aVL, aVF, aVR) and a random subselection of 4 leads, Supplementary Figure S2. These results suggest our model benefits from the complete 12 lead input and classification metrics are negatively impacted with any nonspecific reduction in leads.

• Download figure
• Open in new tab

Figure 2 Ablation studies impact on LVH-Fusion performance

Bootstrap 95% CI for performance metrics, F1-score and average precision score, for each model trained on ablated input data. for each prediction metric is shown. (TOP row) Results from ablating ECG input. (BOTTOM row) Results from ablating echocardiogram input. For each ablation setting, a separate model was trained on that type of ablated data to quantify the information content in the data.

For the echocardiogram single-modal model, we examined segmentation, restricting the prediction algorithm to i) only the region around the left ventricle, ii) random single frames, and iii) single end diastolic frames. Restricting the echocardiogram model to the area around the left ventricle caused a decrease in accuracy, showing the model relies on information outside of that region to make classifications. This is interesting given the focus of clinicians on the left ventricle when considering LVH, even despite the fact that hypertension could impact the left atrium by causing restriction and HCM affects all four chambers. Restricting the model’s input to a single frame further decreases accuracy, demonstrating that motion information is important in distinguishing between HCM and hypertension. Figure 2 details the performance of each ablation experiment.

Model interpretations

In order to improve our understanding of how LVH-Fusion classifies left ventricular etiology, we implemented SHAP GradientExplainer, a game theory approach to explain the output of a machine learning algorithm¹⁹. Relating this method to the ECG model, this approach takes the prediction of a model and estimates the gradient with respect to each individual timestep for every lead from the input signal. For echocardiogram videos, an analogous methodology applies: the gradient of the model’s prediction was calculated with respect to every pixel from the input video. In each case, the calculated value is then compared to a provided background distribution, the training data. The value of the calculated gradients for each timestep/pixel is then assigned an importance score such that highly impactful scores (denoted in red) hold positive impacts on prediction estimates. Values with low importance scores negatively influence prediction estimates (denoted in blue).

We emphasize samples of ECG and echocardiograms from the test partition to deduce regions the model found most impactful to prediction estimates, Figure 3 and 4. In Figure 3, the ECG interpretation results highlight an overall focus on V3 and T-wave inversion in leads V1-V6. Both the observed early R wave progression and T-wave inversion are indications of HCM. Summarized local interpretations for each lead provides explanations of the overall impact each lead has on prediction estimates. Additional examples of ECG interpretation tracings can be found in the Supplement Figure S1. Comparably, the interpretation results of the echocardiogram videos, Figure 4, clearly depicts asymmetric proximal septal thickness, a hallmark distinction of HCM across all frames of the video. Next, to examine local summary interpretations, we segmented the left ventricle on each frame for duration of a video’s length. This allowed us to quantitatively compare the positive and negative impacts the estimated LV size had on overall prediction estimates, Supplemental Figure S3.

• Download figure
• Open in new tab

Figure 3 LVH-Fusion ECG interpretations.

SHAP explanations of one true positive, HCM sample (A). Red areas indicate timesteps that hold a positive impact on prediction, while blue timesteps indicate a negative impact on prediction, no color is neutral. (B) Selected regions of ECG leads denote timesteps of high estimated importance, focusing on inverted T-waves and lead V3 R peaks. (C) Local explanations of the cumulative SHAP values on prediction output across leads. Lead V3 overall contains the highest values of SHAP values for this sample presented.

• Download figure
• Open in new tab

Figure 4 LVH-Fusion echocardiogram interpretations.

SHAP explanations for two true positive samples, HCM (top row) and HTN (bottom row). Each class has 3 frames selected with SHAP values overlaid. Red areas indicate pixels that hold a positive impact on prediction, while blue pixels indicate a negative impact on prediction, no color is neutral. We observe red areas of importance converging on the asymmetric septal wall in the HCM example.

To further examine if the regions of importance identified in distinct samples are globally similar across all predictions, a summation or averaging across all local instances was performed. This approach provides a highly compressed, global insight into the model’s behavior. We considered per lead contributions to predictions in ECGs and left ventricular segmentation in echocardiogram videos. Global summary results for ECG corroborates our results from the ablation studies, lead V3 and aVR holds valuable information for model’s prediction estimates, Supplement Figure S4.

Comparison against physician interpretation

We had two expert readers review ECG tracings and echocardiogram videos and asked them to make a diagnosis of HTN or HCM. We selected 45 samples (40 HTN and 5 HCM) from the test set to compare LVH-fusion. The LVH-fusion model outperformed these expert cardiologists (one of whom has 20 years of experience in diagnosing HCM). LVH-fusion correctly classified 3 out of the 5 ECG and echocardiogram HCM samples. Variability between cardiologists varied greatly, with one cardiologist matching LVH-fusion sensitivity estimates but with a reduction in specificity, while cardiologist two failed to correctly classify any of the HCM ECG samples provided.

Data acquisition and study population

Hypertrophic cardiomyopathy patients were selected for this study from the Hypertrophic Cardiomyopathy clinics at the Stanford Center for Inherited Cardiovascular Disease. Hypertension patients were selected from individuals that were found to be persistently hypertensive (SBP >150) with at least 5 consecutive systolic blood pressure readings over 150. Exclusion criteria included any ECG clinical annotations of ventricular-pacing or left bundle branch block. In addition, we excluded any data from both electrocardiograms and echocardiograms datasets if the date acquired was after a documented myectomy procedure.

We retrieved 15,761 electrocardiograms (ECGs) and 3,234 transthoracic echocardiograms from 2,728 unique individuals at Stanford Health Care, Table 1. Standard 12 lead ECGs were divided into training, validation, and test partitions based on a unique patient identification number to ensure that no patient overlap existed across data partitions. Echocardiogram videos from Stanford Medicine were curated for apical 4-chamber view videos.

Data Processing and selection

Electrocardiogram signals were filtered to remove any baseline wander and powerline interference. Normalization of 12 lead ECGs was performed by lead over a random subset of the study sample population, using mean and standard deviation. Echocardiogram videos were processed in an identical method as Oyuang et al¹³. Given multiple electrocardiograms and echocardiograms per individual present within our dataset, we examined the effects of different data selection methods on model training and performance metrics. We selected three different data selection methods: 1) first clinical presentation for all data partitions, 2) all clinical presentations in the training partition with only first clinical presentation selected for the validation and test partitions, and 3) all clinical presentations for all partitions. Extended details of each selection method can be found in Supplemental Table 1.

Overview of model training framework

Training for the single-modal and multimodal neural network models were executed independently.

Models were trained using a two-stage grid search approach to find the optimal hyperparameters. In the initial hyperparameter search, evaluation metrics from the validation set can be found in the Supplementary Tables S5, S6. The hyperparameters that yielded the best performing models were selected for additional training and hyperparameter search considering various loss functions, loss weighting for minority class and minority class oversampling. Final models were selected from the lowest validation loss.

Single-modal model training

For electrocardiogram single-modal model training, the following hyperparameters included: model architecture: {VGG11, VGG13, VGG16, VGG19, densenet169, densenet121, densenet201, densenet161 resnet18, resnet34, resnet50, resnet101, resnet152, resnext50_32×4d, resnext101_32×8d, wide_resnet50_2 wide_resnet101_2}; batch size: {32, 64, 75}; Optimizer: {SGD, adam}, and Hz: {500, 250}. The first hyperparameter search involved training all combinations of hyperparameters above for 100 epochs and saving results from the epoch with the lowest loss. Furthermore, we explored a second hyperparameter search which explored class weighted loss functions, oversampling minority class samples and setting final bias term to the expected class ratios from top performing models from the initial hyperparameters search. We examined expanding training to 150 epochs and considering both loss and auPRC results for selection of the final model. The selected hyperparameters that resulted in best performance on the validation set were the following: resnet 34 model, oversampling minority class, adam optimizer, batch size of 64, and sampling rate of 500.

For echocardiogram unimodal model training, the following hyperparameters included: Model architecture: {r2plus1d_18, mc3_18, r3d_18}, Number of frames: {96, 64, 32, 16, 8, 4, 1}; Period: {2, 4}; Pretrained weights: {True, False}. The first hyperparameter search involved training all combinations of hyperparameters above for 100 epochs and saving results from the epoch with the lowest loss. Furthermore, we explored a second hyperparameter search which explored class weighted loss functions, oversampling minority class samples and setting final bias term to the expected class ratios from top performing models from the initial hyperparameters search. We examined expanding training to 300 epochs and considering both loss and auPRC results for selection of the final model. The selected hyperparameters that resulted in best performance on the validation set were the following: r2plus1d_18 model, pretrained weights, weighted minority class, adam optimizer, batch size of 20, and frames 16 with sampling period of 4.

Multimodal model training

For multimodal training models, the electrocardiogram and echocardiogram data were paired according to unique patient identifiers. Data selection for the earliest clinical encounter was selected for all training, validation and test set partitions; this resulted in a total of 1,414 training, 176 validation, and 168 internal test samples. The detailed characteristics of the dataset can be found in Table 1. We hypothesized that using the learned weights from the single-modal models would benefit training so we explored both pre-trained late fusion and random late fusion models. All multimodal models were trained to 300 epochs and we considered both loss and auPRC results for selection of the final multimodal model. We implemented LVH-Fusion using PyTorch on the Stanford University Research cluster, Sherlock. The selected hyperparameters that resulted in best performance on the validation set were the following: r2plus1d_18 model + resnet 34, pretrained weights, weighted minority class, adam optimizer, batch size of 10, and frames 16 with sampling period of 4.

Comparison to feature based models

Standard reported features from Tracemaster electrocardiogram machines were extracted for each ECG considered in this study. We used these features for input into a XGboost model to determine if a feature-based method would exceed the performance metrics of the unimodal neural network models. The list of ECG features used were modeled from Kwon et al. 2020 ¹⁰.

Comparison with normal samples

In order to explore how our neural networks, perform on non-left ventricular hypertrophy individuals, we sampled electrocardiograms with clinical annotations of sinus rhythm and echocardiograms with a normal ejection fraction greater than 45. We took the best performing single-modal model and retrained them to include an additional non-LVH class; details of sample size and performance metrics can be found in Supplementary Table 5 and Supplementary Table 6, respectively.

Ablation experiments

To further understand how the neural networks make their predictions, we explored various ablation studies.

We retrained the single-modal echo model with data ablated in the following ways:

1. a single randomly selected frame of each echo, repeated for the length of the original video to compare to the best performing unimodal model.

2. The end diastolic frame from each echo, repeated for the length of the original video to fairly compare to the best performing unimodal model. The end diastolic frame was identified by a trained sonographer from EchoNet-dynamic¹³.

3. Using the estimated left ventricular segmentation from EchoNet-dynamic¹³, we set all pixels to zero except a segmented box around the left ventricle.

For electrocardiogram we retrained the single-modal models for the following experiments:

1. Using 8 of the 12 leads, to compare to the best performing unimodal model.

2. Masking out each lead independently to compare to the best performing single-modal model and understand impacts each lead holds on performance.

Echocardiogram models were trained to 300 epochs and electrocardiogram models were trained for 150 epochs.

SHAP Interpretation experiments

SHAP GradientExplainer¹⁹ uses an extension of integrated gradient values and SHAP values, which aims to attribute an importance value to each input feature by integrating the gradients of all interpolations between a foreground sample (test samples) and a provided background samples (training data). The importance scores sum up to approximately the difference between the expected value of all background samples and the individual prediction estimate of interest. We applied this method to both ECG and echocardiogram models; 1500 samples were used to build the background distribution for the ECG model and 80 samples were used to build the background distribution for the echocardiogram model. In both cases, the full test set was used as foreground samples.