Precision phenotyping from routine laboratory parameters for machine learning out-of-hospital survival prediction using 4D time-dependent SHAP plots in an all-comers prospective PCI registry
==============================================================================================================================================================================================

* Paul-Adrian Călburean
* Anda-Cristina Scurtu
* Paul Grebenisan
* Ioana-Andreea Nistor
* Victor Vacariu
* Reka-Katalin Drincal
* Ioana Paula Sulea
* Tiberiu Oltean
* László Hadadi

## Abstract

**Introduction** Out-of-hospital mortality in coronary artery disease (CAD) is particularly high and established adverse event prediction tools are yet to be available. Our study aimed to investigate whether precision phenotyping can be performed using routine laboratory parameters for the prediction of out-of-hospital survival in a CAD population treated by percutaneous coronary intervention (PCI).

**Materials and methods** All patients treated by PCI and discharged alive in a tertiary center between January 2016 – December 2022 that have been included prospectively in the local registry were analyzed. 115 parameters from the PCI registry and 266 parameters derived from routine laboratory testing were used. An extreme gradient-boosted decision tree machine learning (ML) algorithm was trained and used to predict all-cause and cardiovascular-cause survival.

**Results** A total of 7186 PCI hospitalizations for 5797 patients were included with more than 610.000 laboratory values. All-cause and cardiovascular cause mortality was 17.5% and 12.2%, respectively, during a median follow-up time of 1454 (687 – 2072) days. The integrated area under the receiver operator characteristic curve for prediction of all-cause and cardiovascular cause mortality by the ML on the validation dataset was 0.844 and 0.837, respectively (all p<0.001). The integrated area under the precision-recall curve for prediction of all-cause and cardiovascular cause mortality by the ML on the validation dataset was 0.647 and 0.589, respectively (all p<0.001).

**Conclusion** Precise survival prediction in CAD can be achieved using routine laboratory parameters. ML outperformed clinical risk scores in predicting out-of-hospital mortality in a prospective all-comers PCI population.

Keywords
*   Machine learning
*   coronary artery disease
*   percutaneous coronary intervention
*   survival analysis

## Introduction

Extensive research has been dedicated to the prediction of adverse events in the coronary artery disease (CAD) population. Numerous clinical, angiographic, and combined risk scores were developed to estimate survival, such as Global Registry of Acute Coronary Events (GRACE) score, Age, Creatinine, and Ejection fraction (ACEF) score, thrombolysis in myocardial infarction (TIMI) score and Synergy Between PCI with TAXUS and Cardiac Surgery (SYNTAX) score.1–4 However, in many available risk scores the prediction performance is mainly driven by high accuracy in predicting in-hospital death, since the presence of very high-risk clinical features almost invariably leads to in-hospital death (e.g., cardiogenic shock, acute pulmonary edema, resuscitated cardiac arrest, mechanical complications of acute myocardial infarction, severe anemia or acute kidney injury), while out-of-hospital death is more difficult to predict. Moreover, long-term survival in CAD is less determined by the complexity of coronary anatomy.5 Thus, more research is needed to find robust predictors of impaired survival in the CAD population, especially after hospital discharge.

Phenotype, initially defined as “the observable traits of an organism”, nowadays includes clinical and paraclinical characteristics of a patient (e.g., diagnostic imaging, laboratory parameters, electrocardiographic changes).6 According to the patient’s phenotypic abnormalities, both diagnostic and prognostic assessments can be made.6 Precision phenotyping is a concept that implies a comprehensive analysis of a wide panel of clinical parameters usually using artificial intelligence (AI) techniques which can quantify even slight changes in a patient’s phenotype and offer a precise prognostic estimate. Precision phenotyping has been previously proposed using clinical, ultrasound, computed tomography, or big “-omic” data.7–9 Whether a precise survival estimate can be given using a spectrum of laboratory parameters routinely measured in clinical practice is yet to be proven.

The aim of the present study was (1) to train an ML model to predict out-of-hospital all-cause and cardiovascular cause mortality using a wide panel of laboratory parameters, (2) to identify predictors of out-of-hospital adverse events, and (3) to compare the predictive performance of ML model with clinical risk scores relevant for CAD population, in an all-comers patient population treated by PCI in a Romanian tertiary cardiovascular center.

## Materials and methods

### Study population

The study design is illustrated in Figure 1. All patients treated by PCI in the Emergency Institute for Cardiovascular Diseases and Transplantation of Târgu Mureş have been prospectively included at discharge in the local PCI Registry of the Institute since January 2016. For the current analysis, all patients treated between January 2016 – December 2022 were included, in order to have a minimum of 6 months of follow-up. Subjects were analyzed by PCI hospitalization (hospitalization in which PCI was performed). Exclusion criteria consisted of (1) age less than 18 years old, (2) presence of in-hospital death, or (2) lack of available survival data (e.g., foreign patients). The Registry is accessible online at the website [http://pci.cardio.ro/](http://pci.cardio.ro/) and is based on the criteria of Cardiology Audit and Registration Data Standards (CARDS) developed by the Department of Health and Children, European Society of Cardiology, Irish Cardiac Society, and the European Commission.10 Briefly, the CARDS recommendations address data regarding demographics, relevant medical history, and comorbid conditions, clinical status at hospital admission, PCI indication, affected and treated coronary artery segments, usage of invasive diagnostic or therapeutic devices, procedural complications, medical treatment during hospitalization and at discharge and in-hospital evolution. All the information available regarding all the variables proposed in that document was collected for each included patient, at every PCI. A number of 115 parameters were available from the PCI registry. Moreover, to complete and update the original CARDS recommendations regarding PCI procedures, an additional 266 laboratory parameters were also collected (Supplemental Table 1). Noteworthy, all laboratory data during the PCI hospitalization was acquired from the electronic health record (EHR). Since the timing and number of determinations of laboratory parameters differ between patients and hospitalizations, each laboratory parameter was considered separately first value in hospital, last value before discharge, average value during hospitalization, absolute maximum and minimum value, and number of each separate determination of that certain laboratory parameter (e.g., first creatinine, last creatinine, average creatinine, maximum and minimum creatinine, number of creatinine determinations, Supplemental Table 1). A total of 381 parameters were used for the final analysis. Furthermore, clinical, and angiographic risk scores such as ACEF, GRACE, and SYNTAX score were also calculated for comparison with the ML model’s survival prediction performance.

![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/09/03/2024.08.31.24312888/F1.medium.gif)

[Figure 1:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/F1)

Figure 1: 
Illustration of the study design.

All patients or their legal representatives (e.g., for critical patients) provided written signed informed consent regarding the PCI procedure and their participation in the study. The study was approved by the ethical committee of our institution (decision number 8646 from 22 December 2015 approved by the Ethical Committee of the Emergency Institute for Cardiovascular Diseases and Transplantation of Târgu Mureş). The protocol was carried out in accordance with the ethical principles for medical research involving human subjects established by the Declaration of Helsinki, protecting the confidentiality of personal information of the patients.

### Follow-up and clinical outcomes

The clinical endpoint of this study was the incidence of out-of-hospital cardiovascular and all-cause mortality. Out-of-hospital mortality was defined as death that occurred after discharge, but not occurring during the initial hospital admission when the PCI procedure was performed. The Romanian National Health Insurance System database supplied mortality rates as of July 2023 for all the patients. For patients who had died during follow-up, the Regional Statistics Office of the Romanian National Institute of Statistics supplied the exact date and cause of death according to the tenth revision of the International Classification of Diseases (ICD-10). If the cause of death belonged to diseases of the circulatory system, then death was considered to be of cardiovascular cause.

### Machine learning

All patients had a minimum of 6 months survival status available and up to 6 years of follow-up. Patients were dichotomized every 6 months into alive/deceased groups and censored events were removed. Afterward, for each timeframe, the dataset was randomly divided into 70% training and 30% validation datasets. To prevent data leaks since a patient could have multiple hospitalizations, each patient was assigned to either a training or validation set. On the 70% training dataset, a 5-fold cross-validation was performed and to prevent data leak between folds, each patient was only assigned to one-fold. An extreme gradient boosted decision tree (XGBoost) algorithm11 was evaluated as a binary classifier for predicting both all-cause- and cardiovascular-cause mortality occurred out of hospital, for each timeframe. XGBoost algorithm was implemented in Python version 3.9.13. Hyperparameter optimization was performed by using Bayesian search on 6 years survival status, being the least imbalanced dataset, and those best training parameters were used for all timeframes. The accessibility of underlying supporting scripts is detailed in the data availability section. In order to explain the ML decision process, the open-source Shapley additive explanations (SHAP) framework was used.12 Most important features were obtained as follows: (1) SHAP values were obtained for each of the database values at each timeframe, for both all-cause and cardiovascular cause events; (2) at each timeframe, absolute SHAP values were added, and parameters’ importance was considered by the highest sum of absolute SHAP value (e.g., the parameter with the highest absolute SHAP values sum at 5-years cardiovascular survival was the most important); (3) overall most important predictors were obtained by ascending classification of averaged positions at all timeframes for all-cause and cardiovascular cause survival, separately (see supporting code). Illustration of time-dependent SHAP values for a certain parameter on a specified survival (e.g., time-dependent SHAP values for age impact on all-cause survival) was performed as follows: (1) SHAP values were obtained for every value of that parameter at each timeframe on the specified survival; (2) for the specified parameter, ten percentile intervals were obtained according to the minimum and maximum value; (3) SHAP values were averaged for each percentile at each timeframe; (4) 3D mesh plot was illustrated using the ten percentiles of that parameter, all investigated timeframes and averaged SHAP values for that percentile; (5) in order to correlate averaged SHAP values with actual observed survival, the now 4D mesh plot was colored by averaged survival for each parameter percentile at each timeframe (see supporting code).

### Statistical analysis

A significance level α of 0.05 and a 95% confidence interval (CI) were considered. Continuous variables were evaluated for normal distribution using the Shapiro-Wilk test. Continuous variables with parametric distributions were reported as mean ± standard deviation and compared using non-paired Student’s t-test, while continuous variables with non-parametric distributions and discrete variables were reported as median (interquartile range) and compared using Mann Whitney test. Categorical variables were reported as absolute and relative frequencies and compared using Fisher’s exact test for clinical parameters with frequencies less than 5 and the Chi2 test otherwise. For baseline clinical characteristics, univariate Cox proportional hazard was used to predict the association in the form of hazard ratio (HR) between observed survival and an independent categorical variable. All continuous variables were dichotomized related to the median to facilitate comparability of HR and CI between variables. Statistical analysis was performed using R version 4.1.1 and R Studio version 1.4.17. Performance on survival prediction was measured by integrating the time-dependent area under the curve of receiver-operator characteristic (AUC-ROC), the area under the curve of precision-recall (AUC-PR), Matthews correlation coefficient (MCC), F1 score, and Brier score for a specified survival. While for AUC-ROC the baseline is constant at 0.5, in the case of AUC-PR, the baseline is determined by positive cases per total cases ratio.13 Thus, a baseline-corrected AUC-PR was calculated by subtracting the baseline value from the AUC-PR value. MCC and F1 score calculation require dichotomization from probability for ML model and score value for clinical risk scores into predicted deceased or alive, which was performed using Youden’s method for cut-off calculation.

## Results

### Study population

A total of 7186 PCI hospitalizations for 5797 patients were included in the present study, with a total of 611309 laboratory values. Of those patients, 4062 (70.07%) were male sex, the median age was 65.0 (57.2-71.8) years and the median BMI was 28.3 (25.9-31.8) kg/m2. Complete clinical characteristics of the studied patients are reported in Table 1 and Supplemental Table 1. Complete clinical characteristics of the studied PCI hospitalizations are reported in Table 2 and Supplemental Table 2. A total of 1017 (17.5%) patients died of all causes during a median follow-up time of 1454 (687 – 2072) days. When censoring events of non-cardiovascular cause, a total of 710 (12.2%) patients died of cardiovascular causes during a median follow-up time of 1544 (748– 2155) days. Cumulative per-patient mortality incidence is illustrated in Figure 2A and cumulative per-PCI mortality incidence is illustrated in Figure 2B. A total of 115 clinical parameters were available from the PCI registry. A median of 114 (110-115) parameters were available for each patient as some parameters were missing or not applicable (e.g., time from symptom onset to PCI, which is measured in acute but not in chronic coronary syndrome). A total of 611309 laboratory values were analyzed for all 7186 hospitalizations. Per hospitalization, a median of 78 (65-116) laboratory values were available. For each laboratory parameter, the first and last in-hospital value, maximum and minimum value during hospitalization, averaged value during hospitalization, and number of determinations for that parameter during hospitalization were calculated, therefore 266 additional parameters were analyzed.

![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/09/03/2024.08.31.24312888/F2.medium.gif)

[Figure 2:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/F2)

Figure 2: 
Observed survival in the studied population.

View this table:
[Table 1:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/T1)

Table 1: Baseline characteristics of the studied parameters on a per patient analysis.

View this table:
[Table 2:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/T2)

Table 2: Baseline characteristics of the studied parameters on a per PCI analysis.

### Machine learning analysis

Machine learning models accurately predicted out-of-hospital survival at all timeframes on all time-dependent performance metrics (Figure 3). While AUC-ROC was consistently high (Figure 3A, 3C), AUC-PR gradually increased as the deceased-to-alive ratio became less imbalanced (Figure 3B, 3D). Minor differences were between testing dataset predictions and validations dataset predictions, revealing that overfitting did not occur (Figure 3). Tuned hyperparameters found by Bayesian search included a total of 5000 aggregated decision trees, with a maximum tree depth of 12 levels and a learning rate of 0.01 (see supporting code). Most important survival predictors were obtained by classifying the averaged importance position at all timeframes. For all-cause mortality, left ventricular ejection fraction (LVEF), age, hospitalization cost, heart rate at presentation, renal function reflected by maximum creatinine during hospitalization, standard deviation of red cell distribution width and last determined before discharge lymphocyte per monocyte ratio were among the most important parameters. For cardiovascular cause mortality, left ventricular ejection fraction (LVEF), age, systolic and diastolic blood pressure at presentation, heart rate at presentation, hospitalization cost, renal function reflected by maximum creatinine during hospitalization, the standard deviation of red cell distribution width and last determined before discharge lymphocyte per monocyte ratio was among the most important parameters. Furthermore, the most important factors for the occurrence of adverse events and most important factors against the occurrence of adverse events of any cause and cardiovascular cause are reported in Supplemental Figure 1. Time-dependent SHAP values are reported in Figure 5 and reveal how the predictor’s values (e.g., advanced age or reduced LVEF) predispose to adverse events. Patient individualized survival prediction can be illustrated using ML probability output at each timeframe with event occurrence estimated at the point where probability is less than the cut-off for that specific timeframe (Figure 6).

![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/09/03/2024.08.31.24312888/F3.medium.gif)

[Figure 3:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/F3)

Figure 3: 
Time-dependent AUC-ROC and AUC-PR for cardiovascular cause and all-cause death prediction.

### Comparison between machine learning and clinical scores

To illustrate the predictive performance of ML models, some important clinical risk scores, such as ACEF, GRACE, or SYNTAX scores, were also calculated and compared with ML models. ML consistently and significantly outperformed clinical risk scores on all timeframes (Figure 3) and all performance metrics (Table 3). Among clinical scores, the ACEF score had the highest performance metrics, while the SYNTAX score had the lowest performance metrics. The integrated AUC-ROC for prediction of all-cause mortality by the ML model versus ACEF score was 0.844 and 0.735, respectively (Table 3). The integrated AUC-ROC for prediction of cardiovascular cause mortality by the ML model versus ACEF score was 0.837 and 0.761, respectively (Table 3). A more striking increase in prediction performance is reflected by the integrated AUC-PR and integrated baseline-corrected AUC-PR. The integrated AUC-PR for prediction of cardiovascular and all-cause mortality by the ML model was 0.647 and 0.589, respectively, while for ACEF score was 0.380 and 0.410, respectively (Table 3). The integrated baseline corrected AUC-PR for prediction of cardiovascular and all-cause mortality by the ML model was 0.407 and 0.428, respectively, while for ACEF score was 0.199 and 0.190, respectively (Table 3). Additional details regarding performance metrics are reported in Supplemental Table 3. Other metrics, such as integrated MCC, F1, and Brier score were also in favor of ML models.

View this table:
[Table 3:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/T3)

Table 3: Performance metrics for ML models and clinical risk scores.

## Discussions

The main findings of our study are: (1) In a prospective all-comers PCI registry, both all-cause and cardiovascular cause out-of-hospital mortality is high, with 6-years mortality of up to 42%; (2) ML models can accurately predict long-term out-of-hospital survival in a CAD population treated by PCI using routine laboratory parameters; (3) Predictive performance of ML models was better than relevant clinical scores, such as GRACE, ACEF or SYNTAX score on all performance metrics, including integrated AUC-ROC, AUC-PR, MCC, F1 and Brier score; (3) Important factors for survival prediction includes age, LVEF, blood pressure and heart rate on presentation. However, most important factors were those derived from laboratory parameter analysis; (4) Time-dependent SHAP values reveal how the values of a parameter impact predictions for each timeframe.

Ischemic heart disease is the leading cause of death14, although precise long-term follow-up from the moment of CAD diagnosis, especially out-of-hospital, is scarce. In our study 5-year and 6-year all-cause, out-of-hospital mortality was 21.7% and 33.2%, respectively, relatively lower than the 5-year all-cause mortality of 37.3% from the SYNTAX trial although it included only three-vessel CAD disease.15 In patients with established CAD, clinical risk prediction tools for secondary prevention are useful, considering the high mortality associated with CAD. For this purpose, certain clinical scores such as ACEF, SYNTAX, and GRACE scores were developed.1,2,4 Refinements of clinical scores were attempted to improve predictive performance, particularly for the original SYNTAX score, from which several SYNTAX-derived scores were reported.16 In our analysis, ML models outperformed clinical scores on all timeframes and all statistical metrics (Table 3). Accordingly, the use of machine learning instead of continuously refined traditional risk scores seems to be more efficient in long-term mortality risk prediction in CAD patients, especially if clinical data can reliantly be obtained from EHR. In the present study, over 600,000 laboratory values were automatically obtained from the EHR, making future integration in clinical practice possible.

In our analysis, LVEF, age, blood pressure, and heart rate were among the most important predictors, which are also partially included in clinical scores such as ACEF and GRACE scores. High hospitalization cost and increased contrast volume were also predictors of events, reflecting the need for advanced therapies and long or complex PCI procedures, respectively. Interestingly, numerous important clinical features were obtained from laboratory values, reflecting renal function (first and maximum serum creatinine and urea levels), hematologic function (red cell distribution width, mean corpuscular hemoglobin concentration, platelet distribution width), inflammatory status (lymphocyte per monocyte ratio, maximum neutrophil count) and glycemic status (Figure 4). Most of the mentioned parameters were previously reported as predictors of adverse events. The association between lymphocyte-to-monocyte ratio and impaired prognosis in the CAD population has been documented in a recent meta-analysis.17 Other studies showed that red cell distribution width is an inexpensive prognostic marker in both myocardial infarction and heart failure.18,19 This illustrates that precision phenotyping can be achieved using laboratory parameters. Moreover, while the clinician works with dichotomized outcomes from continuous variables (e.g., patient has heart failure with reduced ejection fraction if LVEF ≤40%) leading to an information loss20, ML models are particularly powerful in analyzing continuous data without the need for fitting into categories – for this reason, the most important parameters were laboratory values instead of disease labels. Interestingly, the severity of coronary lesions was not particularly important for long-term prognosis. Similarly, SYNTAX score performance decreased over time, revealing that coronary anatomy is less important in the long term.

![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/09/03/2024.08.31.24312888/F4.medium.gif)

[Figure 4:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/F4)

Figure 4: 
Most important parameters in the prediction of cardiovascular cause and all-cause mortality (lower is better).

![Figure 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/09/03/2024.08.31.24312888/F5.medium.gif)

[Figure 5:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/F5)

Figure 5: 
4D time-dependent SHAP mesh plot for various predictors of all-cause mortality.

![Figure 6:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/09/03/2024.08.31.24312888/F6.medium.gif)

[Figure 6:](http://medrxiv.org/content/early/2024/09/03/2024.08.31.24312888/F6)

Figure 6: 
An example of a patient specific estimated survival curve.

The emergence of AI brought numerous tools for prediction making, which can be simplistically categorized as supervised learning using machine learning techniques (e.g., support vector machines or gradient-boosted decision trees) or unsupervised learning using deep learning techniques (e.g., neural networks). Depending on the underlying task, certain algorithms could be more suitable. Indeed, for tabular data, deep learning seldom outperforms machine learning.21 Regarding survival analysis, Kaplan–Meier and Cox proportional hazards models have successfully been used for decades.22 However, both techniques consider a linear relationship between the risk factor and the log hazard function23. This constant effect of the risk factor during the follow-up assumed by the Kaplan–Meier and Cox models could be an oversimplification.23 Recently, ML methods were proposed to analyze survival data.24 More specifically, standard ML techniques use binary classification to predict the outcome at a certain timeframe. This requires dichotomization with removal of censored data, does not consider hazards a function of time, but has a high interpretability of the model. On the other hand, modified ML techniques for survival data are adjusted to consider hazards as a function of time, to handle censored and time-to-event data, but have lower interpretability.24 Moreover, numerous performance metrics to evaluate survival prediction models are time-dependent and require the removal of censored events (e.g., Brier score or time-dependent area under receiver operator characteristic).25 Indeed, there are a variety of ML and deep learning techniques to analyze survival data, and the best technique to perform survival analysis is yet to be determined.

A reproducibility crisis is increasingly recognized in the area of AI, driven mainly by data leakage and lack of transparency by unpublished code.26,27 Indeed, data leakage – the ability of the AI model to see the outcome of the training dataset – is an important problem that may remain undetected.28 In our previous study, an unexpected data leak was observed – the ML model was making an unrealistic nearly perfect in-hospital mortality prediction by looking at treatment at discharge, recognizing that if a patient did not have medical treatment at hospital discharge it was because the patient was deceased.29 In the current study, we consider data leak to be highly improbable, since (1) only in-hospital data was analyzed, (2) only out-of-hospital death was considered, and (3) survival data was acquired from a different institution. Moreover, survival data was not merged with clinical data during the analysis. Regarding underlying code, a report revealed that 6% of AI papers are accompanied by complete code, while half of the AI papers contained “pseudocode”, a brief description of the algorithm.30 In line with current recommendations, the underlying algorithm code for the current study is publicly disclosed.

### Study limitations and future research directions

The main limitation of our study is the lack of external validation. In addition, the study population is typical for Eastern Europe, thus extrapolating to other populations could be limited. Although not a limitation per se, we are the first to illustrate a time-dependent SHAP plot and certain time-integrated statistical indicators (e.g., integrated AUC-PR and baseline corrected AUC-PR) which warrants further investigation.

## Conclusions

Machine learning can accurately predict out-of-hospital all-cause and cardiovascular cause death using routinely performed laboratory parameters and outperforms classic clinical parameter-based risk scores in a prospectively followed CAD population treated by PCI.

## Author contributions

P.-A.C. and L.H. were involved in the study design. P.-A.C. and L.H. performed the literature search. L.H. supervised and coordinated the data collection in the registry. P.-A.C. was involved in the development of machine learning models and explainable artificial intelligence. All authors were involved in data collection. T.O. was responsible for the development and maintenance of the online platform. P.-A.C. and L.H. analyzed the data. P.-A.C. and L.H. were involved in the writing of the initial draft. All authors were involved in the writing of the final draft and approved the manuscript.

## Conflict of interest

The authors have nothing to disclose.

## Data availability

Due to national and EU regulations, particularly the General Data Protection Regulation, the data used in this study cannot be made publicly available and shared with the wider research community. The data can be shared by the corresponding author for use in secure environments upon request. However, all Python scripts underlying this study are publicly available at [https://github.com/paul-md/Survival-ML](https://github.com/paul-md/Survival-ML).

## Acknowledgments

The authors would like to thank all the resident colleagues who continuously introduced the data regarding PCIs performed in the Emergency Institute for Cardiovascular Diseases and Transplantation since 2016. We also would like to thank Mr. Ioan Matei for his help in achieving long-term mortality data.

## Footnotes

*   **Funding:** This research received no funding.

*   Received August 31, 2024.
*   Revision received August 31, 2024.
*   Accepted September 3, 2024.


*   © 2024, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## References

1.  1.Tang, E. W., Wong, C.-K. & Herbison, P. Global Registry of Acute Coronary Events (GRACE) hospital discharge risk score accurately predicts long-term mortality post acute coronary syndrome. American Heart Journal 153, 29–35 (2007).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ahj.2006.10.004&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17174633&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000243706000009&link_type=ISI) 

2.  2.Chichareon, P. et al. Predictive ability of ACEF and ACEF II score in patients undergoing percutaneous coronary intervention in the GLOBAL LEADERS study. International Journal of Cardiology 286, 43–50 (2019).
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 

3.  3.Mok, Y. et al. International Validation of the Thrombolysis in Myocardial Infarction (TIMI) Risk Score for Secondary Prevention in PostLMI Patients: A Collaborative Analysis of the Chronic Kidney Disease Prognosis Consortium and the Risk Validation Scientific Committee. Journal of the American Heart Association 7, e008426 (2018).
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiYWhhb2EiO3M6NToicmVzaWQiO3M6MTI6IjcvMTQvZTAwODQyNiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDI0LzA5LzAzLzIwMjQuMDguMzEuMjQzMTI4ODguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

4.  4.Sianos, G. et al. The SYNTAX Score: an angiographic tool grading the complexity of coronary artery disease. EuroIntervention [https://eurointervention.pcronline.com/article/the-syntax-score-an-angiographic-tool-grading-the-complexity-of-coronary-artery-disease](https://eurointervention.pcronline.com/article/the-syntax-score-an-angiographic-tool-grading-the-complexity-of-coronary-artery-disease).
    
    

5.  5.Thuijs, D. J. F. M. et al. Percutaneous coronary intervention versus coronary artery bypass grafting in patients with three-vessel or left main coronary artery disease: 10-year follow-up of the multicentre randomised controlled SYNTAX trial. The Lancet 394, 1325– 1334 (2019).
    
    

6.  6.Robinson, P. N. Deep phenotyping for precision medicine. Human Mutation 33, 777– 780 (2012).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/humu.22080&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22504886&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 

7.  7.Precision Phenotyping in Heart Failure and Pattern Clustering of Ultrasound Data for the Assessment of Diastolic Dysfunction. 10.1016/j.jcmg.2016.10.012 doi:10.1016/j.jcmg.2016.10.012.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiamltZyI7czo1OiJyZXNpZCI7czoxMDoiMTAvMTEvMTI5MSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDI0LzA5LzAzLzIwMjQuMDguMzEuMjQzMTI4ODguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

8.  8.Oikonomou, E. K., Siddique, M. & Antoniades, C. Artificial intelligence in medical imaging: A radiomic guide to precision phenotyping of cardiovascular disease. Cardiovascular Research 116, 2040–2054 (2020).
    
    

9.  9.Delude, C. M. Deep phenotyping: The details of disease. Nature 527, S14–S15 (2015).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/527S14a&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26536218&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 

10. 10.Flynn, M. R. et al. The Cardiology Audit and Registration Data Standards (CARDS), European data standards for clinical cardiology practice. European Heart Journal 26, 308– 313 (2005).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/eurheartj/ehi079&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15618029&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000228892200016&link_type=ISI) 

11. 11.Chen, T. & Guestrin, C. XGBoost: A Scalable Tree Boosting System. in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (2016). doi:10.1145/2939672.2939785.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1145/2939672.2939785&link_type=DOI) 

12. 12.Lundberg, S. & Lee, S.-I. A Unified Approach to Interpreting Model Predictions. arXiv:1705.07874 [cs, stat] (2017).
    
    

13. 13.Saito, T. & Rehmsmeier, M. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE 10, e0118432 (2015).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0118432&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25738806&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 

14. 14.Nowbar, A. N., Howard, J. P., Finegold, J. A., Asaria, P. & Francis, D. P. 2014 Global geographic analysis of mortality from ischaemic heart disease by country, age and income: Statistics from World Health Organisation and United Nations. International Journal of Cardiology 174, 293–298 (2014).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijcard.2014.04.096&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24794549&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 

15. 15.Mohr, F. W. et al. Coronary artery bypass graft surgery versus percutaneous coronary intervention in patients with three-vessel disease and left main coronary disease: 5-year follow-up of the randomised, clinical SYNTAX trial. The Lancet 381, 629–638 (2013).
    
    

16. 16.Yadav, M. et al. Prediction of Coronary Risk by SYNTAX and Derived Scores. Journal of the American College of Cardiology 62, 1219–1230 (2013).
    
    [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJhY2NqIjtzOjU6InJlc2lkIjtzOjEwOiI2Mi8xNC8xMjE5IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDkvMDMvMjAyNC4wOC4zMS4yNDMxMjg4OC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

17. 17.Quan, X.-Q., Wang, R.-C., Zhang, Q., Zhang, C.-T. & Sun, L. The predictive value of lymphocyte-to-monocyte ratio in the prognosis of acute coronary syndrome patients: a systematic review and meta-analysis. BMC Cardiovasc Disord 20, 338 (2020).
    
    

18. 18.Al-Najjar, Y., Goode, K. M., Zhang, J., Cleland, J. G. F. & Clark, A. L. Red cell distribution width: an inexpensive and powerful prognostic marker in heart failure. European Journal of Heart Failure 11, 1155–1162 (2009).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/eurjhf/hfp147&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19926599&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000272082700009&link_type=ISI) 

19. 19.Dabbah, S., Hammerman, H., Markiewicz, W. & Aronson, D. Relation Between Red Cell Distribution Width and Clinical Outcomes After Acute Myocardial Infarction. The American Journal of Cardiology 105, 312–317 (2010).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.amjcard.2009.09.027&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20102941&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000278136500007&link_type=ISI) 

20. 20.Fedorov, V., Mannino, F. & Zhang, R. Consequences of dichotomization. Pharmaceutical Statistics 8, 50–61 (2009).
    
    

21. 21.Shwartz-Ziv, R. & Armon, A. Tabular Data: Deep Learning is Not All You Need. Preprint at doi:10.48550/arXiv.2106.03253 (2021).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.48550/arXiv.2106.03253&link_type=DOI) 

22. 22.Cox, D. R. Regression models and lifeLtables. Journal of the Royal Statistical Society: Series B (Methodological) 34, 187–202 (1972).
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1972N572600003&link_type=ISI) 

23. 23.Harrell, F. E. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. vol. 608 (Springer, 2001).
    
    

24. 24.Cygu, S., Seow, H., Dushoff, J. & Bolker, B. M. Comparing machine learning approaches to incorporate time-varying covariates in predicting cancer survival time. Sci Rep 13, 1370 (2023).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41598-023-28393-7&link_type=DOI) 

25. 25.Park, S. Y., Park, J. E., Kim, H. & Park, S. H. Review of Statistical Methods for Evaluating the Performance of Survival or Other Time-to-Event Prediction Models (from Conventional to Deep Learning Approaches). Korean J Radiol 22, 1697–1707 (2021).
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34269532&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F03%2F2024.08.31.24312888.atom) 

26. 26.Hutson, M. Artificial intelligence faces reproducibility crisis. Science 359, 725–726 (2018).
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEyOiIzNTkvNjM3Ny83MjUiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyNC8wOS8wMy8yMDI0LjA4LjMxLjI0MzEyODg4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

27. 27.Gibney, E. Could machine learning fuel a reproducibility crisis in science? Nature 608, 250–251 (2022).
    
    

28. 28.Kapoor, S. & Narayanan, A. Leakage and the Reproducibility Crisis in ML-based Science. Preprint at doi:10.48550/arXiv.2207.07048 (2022).
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.48550/arXiv.2207.07048&link_type=DOI) 

29. 29.Călburean, P.-A. et al. Prediction of 3-year all-cause and cardiovascular cause mortality in a prospective percutaneous coronary intervention registry: Machine learning model outperforms conventional clinical risk scores. Atherosclerosis 350, 33–40 (2022).
    
    

30. 30.Gundersen, O. E. & Kjensmo, S. State of the art: Reproducibility in artificial intelligence. in Proceedings of the AAAI Conference on Artificial Intelligence vol. 32 (2018).