GraftIQ: A Hybrid Multi-Class Neural Network Integrating Clinical Insight for Multi-Outcome Prediction in Liver Transplant Recipients

INTRODUCTION

Liver transplantation (LT) is a life-saving measure for selected patients with end stage liver disease ¹. Despite tremendous improvement in LT outcomes over recent decades, liver transplant recipients (LTRs) remain at risk of developing graft injury of various etiologies. Graft injury can result in fibrosis and cirrhosis over time, potentially resulting in graft loss in 25% of LTRs ². Graft viability after LT is dependent on the prompt recognition of post-transplant pathologies. Promptly starting treatments like high-dose steroids for rejection or antiviral therapy relies on identifying the cause of graft injury to prevent long-term dysfunction.

Graft injury is often suspected via elevated liver enzymes during routine bloodwork. However, biochemical tests are non-specific and it is difficult to establish a cause of injury based on these alone ³. Liver biopsy has therefore remained the gold standard for the diagnosis of graft pathology⁴. In fact, with improvements in post-transplant survival over the last 2 decades, repeat evaluations of graft function via biopsy have become more frequent⁵. Liver biopsy is subject to sampling error as well as complications such as bleeding, infection⁶ and is often unavailable in a timely manner. Time constraints mean that hepatologists often have to make empiric decisions before liver biopsy is carried out. Such decisions are based on clinical data (age, indication for transplant, time after transplant, diabetes, obesity, immunosuppression regimen) and liver biochemical patterns. Thus, there is a clinical need to develop reliable, non-invasive methodologies that can establish or rule out specific etiologies of graft injury prior to biopsy, allowing rational therapeutic decisions to be made as quickly as possible.

Machine learning (ML), specifically Neural Networks (NN) are efficient in analyzing large, complex, and heterogeneous datasets, generating reproducible predictions and classifications on previously unseen data⁷. Previous studies have demonstrated the feasibility of convolutional NNs to generate accurate prognostic predictions and fibrosis detection in various chronic liver diseases by leveraging blood test patterns and interrelationships between variables^8,9. When applied to liver transplantation, ML has emerged as a promising methodology to stratify patient risk and predict post-transplant outcomes^10-12,30. Convolutional NNs have been applied to help predict waitlist mortality¹³, donor-recipient matching^14,15 and HCC recurrence¹⁶. While ML models have the potential to process and analyze vast amounts of data quickly and efficiently, it is important to recognize that ML models may lack the nuanced understanding, and contextual knowledge that experienced clinicians bring to patient care.

In this study, we hypothesized that combining the extensive, prior knowledge of causal and correlational associations that human experts possess with a machine-learned model would increase model generalizability. To address the same, we have developed a novel hybrid multiclass neural network model ‘GraftIQ’, that aims to predict the etiology of a graft injury using richly annotated clinical, demographic and laboratory data from LT recipients. As opposed to neural networks being commonly used for binary classification, the novelty of our approach lies in its multi-class classification of 6 diagnostic categories. This enables our approach to be a one-step methodology for multicategory classification through a single comprehensive model. Beyond this, another novelty of our methodology is the Bayesian fusion-based integration of clinician feedback to refine predictions of our ML model for being applicable in the clinical setting. Our hybrid ML tool ‘GraftIQ’ could potentially reduce dependence on longitudinal liver biopsies and lead to earlier therapeutic interventions, improving graft viability and patient survival over time.

MATERIALS and METHODS

2. Study Design

2.1 Definition and diagnosis of post-transplant complications

The first part of the study was to establish the most common etiologies for graft injury in LTRs from available biopsies. Biopsies were reviewed by two separate reviewers and labeled according to the appropriate diagnosis from the pathologist’s biopsy report. Biopsies were labeled as normal, acute cellular rejection (ACR), antibody-mediated rejection, biliary obstruction (BO), congestion, alloimmune hepatitis (AIH), viral (Hepatitis C, Hepatitis B, Cytomegalovirus (CMV), Epstein Barr virus (EBV)), metabolic-associated steatohepatitis (MASH) and toxic/drug-induced graft injury. Dual diagnoses were excluded from the analysis. The categories that were considered for statistical and ML analysis included ACR, BO, AIH, Hepatitis C infection (HCV), congestion, and MASH. The biopsies that read as normal as well as all other remaining diagnoses were grouped together as ‘Others’ due to the small number of samples as shown in Figure 1(b).

• Download figure
• Open in new tab

Figure 1: Overall Framework and Study Design.

(a) Schematic representation of our hybrid multiclass neural network, ‘Graft IQ,’ which combines clinician expertise with multiclass neural network capabilities to predict the cause of graft injury. (b) Flowchart detailing study design and data distribution

RESULTS

3. Results of implementation analysis

As shown in Figure 2, for the 30 cases chosen for the implementation analysis, the ML model exhibited higher predictive accuracies, surpassing hepatologists in every evaluated category. Notably, the ML tool achieved a perfect 100% accuracy for Alloimmune hepatitis, BO, HCV, MASH, showcasing its robust diagnostic capabilities. In contrast, hepatologists demonstrated comparatively lower accuracy rates, particularly in predicting ACR, BO, congestion, and MASH. The instances where our ML model misclassified ACR (67%) and congestion (80%) categories shed light on the importance of integrating clinical expertise into our predictive framework. In the case of ACR, cases were misclassified as MASH, despite elevated liver enzymes and blood tests, as the ML model failed to consider the patient’s low age of graft, a key indicator against MASH. Similarly, in congestion misclassified as HCV, the model overlooked the absence of HCV as an indication for transplant. These discrepancies underscore the necessity of incorporating clinical insights to align more closely with the complexities of real-world clinical scenarios.

• Download figure
• Open in new tab

Figure 2: Expert vs Machine Implementation analysis.

Comparison of multiclass NN based ML model’s prediction using clinical and lab data vs averaged prediction accuracy provided by 12 hepatologists (medics)

4. Predictive Performance evaluation

4.1 Results on the test set

4.1.1 Utilizing multiclass Neural Network model standalone

Firstly, we evaluated the performance of our multiclass NN based ML model independently, without integrating any clinical expertise for predicting each diagnosis category as shown in Table 2. The best performance was obtained for MASH post-transplant complications with Area Under the Curve (AUC) of 0.929 calculated using the Receiver Operating Characteristic (ROC) curve with a sensitivity, and specificity of 0.89 and 0.92 respectively, followed by AIH and congestion, with AUC of 0.924 and 0.922 respectively. The overall AUC was obtained by averaging the AUC obtained for each individual category. In our case, the overall AUC for our neural network methodology on the test set was obtained to be 0.885 [95% Confidence Interval (C.I.): 0.864, 0.901].

View this table:

• View inline
• View popup
• Download powerpoint

Table 2: Performance metrics obtained through evaluation of the hybrid neural network algorithm ‘GraftIQ’ in predicting diagnosis categories.

Performance metrics obtained using only the multiclass NN as well as the hybrid neural network algorithm (multiclass NN + clinical insight) for each complication group. Comprehensive analysis in terms of Sensitivity, Specificity, Positive Predictive value (PPV), Negative Predictive Value (NPV) and Area Under the curve (AUC) is represented for each category. The higher each metric, the better the classification is. Note the improvement in AUC values as observed upon integrating clinician expertise into the neural network model for each diagnosis category.

4.1.2 Utilizing “GraftIQ”, hybrid model integrating NN prediction and clinical insight

As shown in Table 2, column 7, the incorporation of clinician-based probabilities (with α = 0.2 and β =0.8 for fusion after tuning as shown in Supplementary Table 2) into the final layer of our neural network model resulted in improvement in predictive performance for each diagnosis category. Specifically, the AUC values surpassed 0.8 for every category (notably high improvement for ACR prediction), representing a significant improvement compared to predictions made using the ML model alone. The overall AUC based on integrating clinical expertise improved from 0.885 to 0.902.

We then compared our neural network model to other conventional machine learning models (refer Table 3) and found that the neural network performs better in terms of overall AUC for classification with AUC of 0.902, 95% C.I.: [0.884, 0.919] as compared to the second-best approach Random Forest with an AUC of 0.823 [95% C.I.: [0.812, 0.839]. Regression approaches performed relatively less accurately in terms of multi-class classification with Logistic Regression with an AUC of 0.767 [95% C.I.: 0.626, 0.796], Lasso with an AUC of 0.783 [95% C.I.: 0.769, 0.802] and Ridge regression with an AUC of 0.781 [95% C.I.: 0.771, 0.811] justifying the prominence of Neural Networks in understanding the non-linear relationships in the data as well as in assigning subjects accurately to one of the multiple categories of diagnosis.

View this table:

• View inline
• View popup
• Download powerpoint

Table 3: Comparative analysis of proposed hybrid GraftIQ model vs. conventional Machine Learning algorithms for comparing overall AUC averaged across six diagnosis categories.

To make our neural network methodology more explainable and clinically relevant, we also computed the variable importance of each clinical feature in the classification task for individual categories. The higher the gradient obtained through the Integrated Gradient methodology detailed in subsection 2.4.3, the more important the feature is towards the classification task. As shown in Figure 3, ALT, ALP and hemoglobin were the top 3 features important to the classification of subjects in the ACR category. Similar plots for the rest of the five diagnosis categories are provided in the Supplementary document (Supplementary Figures 1-5).

• Download figure
• Open in new tab

Figure 3: Importance plot ranking features relevant to classification of subjects into the Acute Cellular Rejection (ACR) category.

The x-axis represents the gradient in neural network learning. The higher the gradient, the more important the feature is.

4.2 Results on external validation set

542 biopsies were reviewed and divided according to the 6 relevant categories along with other biopsies. 233 biopsies were consistent with ACR, 68 with biliary obstruction, 77 with MASH, 18 with congestion, 23 with HCV, 23 with AIH and 100 considered as others. We focused on the first six categories for clinical significance. The model performance in the external test set was in line with our main results with the best performance obtained for AIH with mean AUC of 0.962 followed by MASH and BO (Table 4). The overall AUC by averaging the AUCs obtained for each individual category was 0.934 [95% Confidence Interval (C.I.): 0.909, 0.959] showing the robustness of our methodology on a completely unseen external validation set.

View this table:

• View inline
• View popup
• Download powerpoint

Table 4: Evaluation results using mean Area Under the Curve (AUC) metric and 95% Confidence Intervals (CI) on each diagnosis category in the external validation dataset using the proposed GraftIQ model

5. Demonstration of Clinical Relevance

To demonstrate further clinical relevance of GraftIQ, we randomly chose one patient from each diagnostic category and applied the algorithm to obtain the probability of each possible diagnosis. We then manually reviewed the raw lab values for each patient to determine if the ML output made clinical sense or provided an expedited path to diagnosis that would otherwise have required further investigation.

Graphs demonstrating the probability of each diagnosis for each selected patient are displayed in Figure 4(a). For example, in patient 1147 with a liver biopsy demonstrating ACR, our algorithm determined an 81% probability of a diagnosis of ACR, followed by a 6% probability of MASH, 5% probability of HCV, 4% probability of BO, 3% probability of AIH and 1% probability of congestion. Subsequent review of the lab parameters for this patient that the algorithm used as part of its analysis demonstrated an ALP of 803, total bilirubin of 147, ALT 227, AST of 141.

• Download figure
• Open in new tab

Figure 4: Clinical relevance and integration of GraftIQ.

(a) Pie charts illustrating probability of a subject being classified into one of the six diagnosis categories using hybrid GraftIQ model. Example subjects selected based on their true diagnosis categories in the dataset, and higher probability in the pie chart represents their predicted class, e.g., for Subject 1501, the true diagnostic category in the test set was MASH and our hybrid model predicted subject being classified as MASH with a probability of 87%. (b) Illustration of a proposed interactive clinician-facing dashboard for clinical integration of our GraftIQ model.

For recurrent AIH, our hybrid model demonstrated an 88% probability of AIH. Review of the labs demonstrated ALP of 617, ALT of 377, AST of 377 and total bilirubin of 82. For post-transplant congestion, ALP was 331, ALT 66, AST 46, and bilirubin 25, whereas for HCV, ALP was 328, ALT 122, AST 66 and bilirubin was 15. Again, GraftIQ was able to identify these diagnoses with probabilities of 86% and 92%, respectively. Our patient with recurrent MASH had an ALP of 86, ALT 109, AST 35 and our algorithm identified this diagnosis with a probability of 87%.

As can be seen from these results, the pattern of liver tests between the different diagnoses is not particularly different, and many clinicians would have difficulty distinguishing between the separate diagnoses based on these lab values alone. They would normally request further tests such as imaging or biopsy to clarify the diagnosis. As demonstrated by the high probabilities above, our hybrid algorithm would be able to provide diagnostic confidence much earlier in the care pathway, possibly streamlining the path to appropriate management measures.

DISCUSSION

Liver transplant recipients often develop elevated liver enzymes post-transplant, indicating potential graft issues³. Upon detection, further diagnostic steps like imaging and biopsy are pursued, though they carry risks and may lead to delays in therapy. With no reliable noninvasive tools available, a probabilistic diagnostic ranking system could expedite treatment decisions and mitigate risks, as presented in our study. Our methodology using a multi-class neural network gave the best performance in predicting each individual category of diagnosis as well as in terms of overall AUC of 0.902 as compared to the conventional machine learning approaches. We also observed no overlap in terms of the 95% confidence interval of our neural network approach [95% C.I.: 0.884, 0.919] versus the second-best performing approach of Random Forest [95% C.I: 0.812, 0.839] validating the improvement provided by our proposed methodology.

The implementation analysis (refer to Section 3) underscored the potential superiority of our ML model over the clinical judgment of clinicians. However, it also revealed an opportunity to refine our misclassification outputs through this analysis. By integrating clinician-based probabilities into our ML model, we imposed logical constraints that reflect the underlying principles of medical diagnosis. These constraints serve as regularization mechanisms, guiding the model to focus on relevant features and preventing it from overfitting to noisy or irrelevant data. As a result, the model’s predictions become more robust and reliable, increasing our overall AUC from 0.885 (ML model only) to 0.902 (ML model + clinical expertise), as they are aligned with established medical knowledge. Ultimately, the synergy between ML models and clinician expertise holds tremendous potential to optimize patient outcomes. The external validation strengthens the model performance with an AUC of 0.934, showing robustness of our approach on a completely unseen dataset.

Reviews of existing studies show that various ML algorithms, including neural networks, have been used in the context of liver disease and transplantation¹². However, studies that focus on using ML to distinguish between various graft-related complications solely from demographic or biochemical parameters are limited¹². For example, a study by Hughes et al from 2001 found that an artificial neural network trained on data from 117 patients with biopsies could predict the presence of ACR with an AUC of 0.902¹⁸. This study is limited by its small sample size and the fact that the algorithm can only diagnose one disease state – ACR. Other such examples specific to post-transplant complications include predicting recurrence of primary disease, patient and graft survival, acute kidney injury and HCC recurrence²¹. Most studies examining ML in the context of liver disease are those that automate diagnosis via image analysis of histopathologic slides^22,23. This highlights our algorithm’s strength as the first demonstration of a neural network that can distinguish between multiple disease states based on demographic and laboratory data alone.

Neural Networks are usually perceived as black boxes wherein they improve predictive performance but are unable to provide the clinical variables driving the predictive ability. To enable interpretability of our ML modeling, we explored two avenues: Firstly, through integrated gradient methodology, we were able to identify the most important clinical variables relevant to the diagnosis of each disease state. For example, important clinical variables for ACR included elevation in ALT, AST and ALP elevation, which are well known to occur in the setting of ACR²⁴. ACR was also associated with recipient age, donor age and creatinine which are all known to be associated with ACR²⁵. Recurrent AIH was most associated with recipient age, consistent with a recent large study that implicated younger recipient age as a risk factor for recurrence²⁶. There was also a stronger association with cyclosporine than tacrolimus use, consistent with studies of the European liver transplant registry that found cyclosporine use after liver transplant for AIH predicted worse survival when compared to tacrolimus use²⁷. Expectedly, post-LT biliary complications were most associated with ALP and total bilirubin, commonly regarded as the most important biochemical parameters for diagnosis of biliary obstruction²⁸. Lastly, recurrent MASH was associated with most of the clinical variables used for analysis, including hemoglobin, ALP, CNI use, and creatinine. MASH is often associated with chronic kidney disease before and after transplant, explaining the importance of creatinine in our algorithm²⁹.

Secondly, through probability modeling, we were able to generate risk of graft etiology for each patient, for example, our randomly chosen patient with ACR had lab values which could be associated with various diagnoses other than ACR, including biliary obstruction or recurrent autoimmune hepatitis. Despite these lab values, our algorithm determined the probability of ACR to be 81%. This shows that our algorithm could allow prompt initiation of ACR treatment, potentially expediting management, reducing resource use, patient morbidity, and preserving graft and overall survival.

We acknowledge that the sample size for HCV-related graft injury was larger in our primary dataset, reflecting the overall cohort collected from 1992 onwards. However, recognizing that the clinical landscape has evolved and HCV is no longer a predominant cause of graft injury, we have retrained our neural network on stratified samples, differentiating between patients with HCV and non-HCV as the primary transplant indication. Our multiclass NN model demonstrates comparable performance to the original dataset, as presented in Supplementary Table 7. This finding suggests that the model’s predictive power remains robust, even as the prevalence of HCV in modern transplant populations declines.

We are also working towards creating a clinician-facing interactive dashboard for our proposed hybrid ML modeling (as shown in Figure 4(b)), where clinicians can load patient covariates such as laboratory data (blood work, liver enzymes, etc.), demographic data and clinical data (data on cholangitis, diabetes etc.) directly from the patient’s digital record, and run our ML model along with their feedback as input in the browser to return a probability of the patient being classified into one of the six post LT complications and furthermore, also get a list of the top clinical features instrumental in predicting the post LT complication in the patient. This dashboard will have the potential to inform the clinician to proactively monitor the most important clinical features in the patient making our ML approach more clinically relevant and useful.

We acknowledge some limitations of our study such as the exclusion of biopsies with dual diagnoses, potentially limiting generalizability at the current time. Additionally, we did not review the pathologies themselves and used solely the pathology report for diagnosis. Although, we might have missed some undiagnosed post-transplant complications, our sample was large and representative enough to offer sound observations. Many patients are treated empirically for mild rejection, without a liver biopsy having been performed. Regardless, we believe this work will serve as a strong basis for further development and future studies in this area. We also acknowledge the limited sample size in the external validation cohort. To address this, we applied bootstrapping and internal cross-validation to generate confidence intervals for our performance metrics and conducted sensitivity analyses, which demonstrated consistent trends across all six categories. Future validation using larger, independent datasets will be a key area for further exploration.

CONCLUSIONS

In conclusion, our hybrid multi-class neural network model, GraftIQ, demonstrates promising potential for non-invasive diagnosis of graft pathology in liver transplant recipients. By combining clinical expertise with efficient deep learning methodologies, we offer a robust framework for accurate diagnosis, potentially reducing the reliance on invasive procedures and improving patient outcomes. If validated and implemented clinically, we believe that this method has potential to decrease time to diagnosis, dependency on liver biopsy and lead to earlier therapeutic interventions that will improve graft and patient survival over time.