A hybrid-computer vision model to predict lung cancer in diverse patient populations

INTRODUCTION

Lung cancer treatment in the United States carries health disparities in minority populations and are especially pronounced in Black individuals¹. Low-dose CT (LDCT) lung cancer screening shifts diagnosis to early-stage disease and is associated with reduced mortality^{2, 3}, but stringent eligibility criteria from the United States Preventive Services Task Force (USPSTF)⁴ exacerbates disparities in Black populations^{4, 6}. These criteria, specifically regarding age and duration of tobacco usage, were derived from risk/benefit analysis of the ∼ 92% White National Lung Screening Trial (NLST) cohort. This results in exclusion of at-risk Black patients from screening. The eligibility to incidence ratio measures this disparity, and according to current USPSTF guidelines, White populations often have >100 eligible patients screened for one incidence of lung cancer, whereas in Black populations this is reduced to approximately 50 eligible patients for every diagnosed case of lung cancer⁴. We⁷ and others^5,8,9 have identified that individualized selection using clinical feature-based risk models improved lung cancer screening eligibility and sensitivity as compared to USPSTF guidelines. Moreover, individualized lung cancer risk prediction from LDCT has been achieved using Convolutional Neural Networks (CNNs). Taken together, risk prediction models have the potential to guide screening selection, to follow up, to determine eligibility, and to mitigate lung cancer disparities in early detection and survival. However, these models have been trained on patient cohorts which underrepresent minorities. To advance these risk models for implementation, their generalizability must first be assessed in minority populations including Black patients.

In this retrospective ever-smoker cohort analysis, we evaluated the utility of clinical and imaging based lung cancer risk models in both the NLST cohort, and a diverse urban and suburban real-world patient population across multiple sites in the Chicago metropolitan area (University of Illinois Health, UIH) representing 65,106 patients including 8,823 Black patients. The Chicago metropolitan area is racially, socioeconomically, and geographically diverse and has regions with higher rates of lung cancer, predominantly in areas populated by Black patients^10–12. We find that a hybrid deep learning model using ResNet and Support Vector Machines (SVMs) from LDCT and clinical features, respectively, outperformed individual models in the NLST test set. When evaluated in the UIH cohort, we observed a racially disparate performance deterioration in the hybrid model on Black – but not White subjects. Further analysis uncovered the model deterioration occurred within the ResNet imaging component of the Hybrid model. Efforts to regularize model performance via Artificial Neural Networks (ANNs) uncovered underlying disparities where improvement of accuracy in Black patients coincided with decreased accuracy in White patients. However, the residual bias still affected the Black population’s performance. This could potentially be due to a large imbalance of LDCTs in White versus Black patients in the ResNet training cohort. Overall, we show proof of concept that a hybrid model has the potential to identify individuals at risk of developing lung cancer with high accuracy in this retrospective cohort, but with disparate performance in Black patients. To mitigate against further lung cancer screening disparities, risk-based model performance must be benchmarked across all races and be trained in patient cohorts that have adequate representation of Black and other minoritized populations.

METHODS

Research analysis was carried out under approved UIC IRB protocols 2023-1321 and 2023-1377. We developed a hybrid AI predictive model to probe the biological basis of lung cancer, and thus assembled a real-world cohort of > 10,000 patients treated in urban and suburban settings in the city of Chicago as well as greater Cook County. This real-world cohort, including > 5,000 Black patients with history of smoking and/or lung cancer, was also filtered based on available chest imaging studies.

Collecting Data: UIH Lung Cancer Screening

To assess the hybrid model in the UIH patient population, we assembled a retrospective cohort of patients with a history of smoking under an IRB approved protocol. Inclusion criteria for this cohort is listed in Supplementary Table 1. Briefly, LDCT screening patients were placed into the cohort by CPT code 71271; and/or based on ICD codes for smoking cessation counseling and cigarette smoking, and through documented history of smoking. ICD-9 code 305.1 and ICD-10 code F17 were used to select LDCT patients and LDCT associated DICOMS were chosen; In addition to LDCT patients, we also expanded the real-world cohort to include patients with smoking histories and CT imaging. A diagnosis of lung cancer was determined based on ICD codes of neoplasm of bronchus or lung (ICD-9 162, ICD-10 C34). Those without a diagnosis of lung cancer were assumed to have no lung cancer at the time of the preparation of this manuscript. Median duration of follow-up between LDCT scans was 431 days and the median number of LDCT scans per patient with at least one LDCT was 1 (the median is 5 if each convolution kernel and reconstruction of a given LDCT exam is counted separately). The composition of races and ethnicities in both the NLST cohort and the UIH cohort are presented in (Fig. 1A). The UIH cohort had a diverse geographic distribution including urban, semiurban, and suburban and patients received healthcare across multiple sites (Fig. 1B).

View this table:

• View inline
• View popup
• Download powerpoint

Supplementary Table 1. The inclusion criteria for participants in the retrospective cohort of UIH

• Download figure
• Open in new tab

Figure 1.

Patient demographics and distribution A) Composition of races and ethnicities in the National Lung Screening Trial (NLST) cohort and the University of Illinois at Chicago (UIC) lung cancer screening cohort. B) Distribution of patient count by zip code in UIC cohort.

Prediction with PLCO_m2012

PLCO_m2012 is a logistic regression model for lung cancer risk inference and was evaluated on NLST data. The PLCO_m2012 model was published in 2012, and was created using a set of 11 chosen features to predict the probability of lung cancer diagnosis for a given patient within 6 years: age, race, education, Body-Mass Index (BMI), presence of Chronic Obstructive Pulmonary Disease (COPD), personal history of cancer, family history of lung cancer, smoking status, cigarettes per day, duration of smoking, and duration of quitting. All 11 features are directly represented or can be extracted from the data in the NLST. The PLCO_m2012 logistic regression model is represented in an R package created by the authors of PLCO_m2012¹⁶. This was translated to Python code to facilitate incorporation with the remainder of the code base. When using PLCO_m2012 for prediction, each sample represents an individual patient. The model was used to generate predictions for probability of lung cancer diagnosis within 6 years only (it was not used to evaluate performance within 1 to 5 years) as intended by its authors (Supp. Fig 3), and these predictions were compared with the ground truth at 6 years, and performance was evaluated across White versus Black patients.

• Download figure
• Open in new tab

Supplementary Figure 1.

Schematic of multi-year SVM training and evaluation

• Download figure
• Open in new tab

Supplementary Figure 2.

11 Feature SVM model and corresponding PR-AUC (left) and ROC-AUC (right) plots in the NLST test set.

• Download figure
• Open in new tab

Supplementary Figure 3.

PLCO_m2012 model prediction of lung cancer at year 6 within the NLST test set. PR-AUC curve (left) with AUC 0.154, ROC-AUC curve (right) with AUC of 0.671.

SVM Prediction

Clinical data elements from the NLST were evaluated for inclusion in a SVM using the Scikit-Learn Python library¹³. 11 features were retained after recursive feature elimination using linear kernel. We started with a large set of initial features from those available in NLST, as well as some generated features which were calculated from existing ones, such as BMI. Recursive feature elimination showed best results when using 11-13 features. We chose to optimize the SVM to 11 features since this is the fewest number of features which resulted in optimal performance. Six unique SVMs were then trained to make predictions 1-6 years post CT scan, with each sample being an individual patient. Each of the 6 SVM models was then evaluated on the test sets using the corresponding ground truths. Additional SVM models were created using a truncated set of features to allow comparison between UIH and NLST cohorts. This truncated set contains 7 features which are well-represented in both data sets; the 4 removed features were poorly represented in the UIH cohort. The training and prediction pipeline of SVM method is given in Fig. 2A.

• Download figure
• Open in new tab

Figure 2.

Pipelines of Machine Learning algorithms. A) Pipeline for implementation a Support Vector Machine (SVM) to predict lung cancer risk using NLST clinical and demographic data. B) Pipeline for the proposed Hybrid SVM model (SVM + ResNet18) combining LDCT scans with NLST clinical and demographic data to predict lung cancer risk. C) Pipeline for the proposed Hybrid model (ANN + ResNet18) utilizing NLST demographic data and LDCT scans to predict lung cancer risk.

RESULTS

NLST and UIH cohorts evaluated in this study and racial and ethnic breakdown initially comprised of 53,452 patients in NLST and 11,654 patients in UIH (Table 1). We sought to develop a hybrid predictive model to combine the long-term predictive potential of clinical features as well as near-term predictive potential of radiographic features from CT. To develop a hybrid lung cancer prediction model, we used the NLST as a training data set inclusive of CT scans and Chest X-Ray arms amounting to 48,628 total patients² for the training of the SVM. A SVM model was chosen owing to its interpretability of features and weights and allowed for multi-year risk prognostication from baseline characteristics of participants at entry (Supp Fig. 1). The SVM was also selected as it is effective at classifying two classes (i.e. patients with no lung cancer diagnosis versus patients with lung cancer diagnosis) where ground-truth was measured by the time from study entry to cancer incidence within the window of that year’s specific model (e.g. SVM₁ predicts the probability of lung cancer within one year or less, SVM₂ within two years etc.). We first focused on and initiated the SVM models with 65 features and utilized recursive feature elimination to optimize the model to 11 features (Supp Fig. 2, Supp. Table 2). These 11 features overlapped with many previous features identified in the PLCO_m2012 6-year risk model but were predictive of multi-year risk as opposed to just 6-year risk (Supp. Table 2)¹⁶. We then further optimized the model for real-world application by eliminating features not captured in electronic health records within routine clinical care since data was not available in > 50% of patients. The excluded features included former vs current smoker, whether patients have family history of lung cancer, patients’ education whether higher than bachelor’s or not, and the maximum number of years exposed to one of the following at work: asbestos, chemical, sand blasting, coal, foundry. We found that the SVM model could be reduced to 7 clinical features well represented in EHR of the UIH cohort while still retaining predictive performance (Fig. 3A, Supp Fig. 2). Performance of the 7-feature multiyear SVM model was first evaluated in a held-out test set from the NLST cohort with ROC-AUC values of 0.64-0.67 similar to the 6-year lung cancer risk when compared to the PLCO_m2012 model (Supp. Fig 2). We then applied the SVM to the UIH cohort of patients and found similar performance with ROC AUC 0.60-0.65 (Fig. 3B).

• Download figure
• Open in new tab

Figure 3.

PR and ROC-AUC curves for SVM model (A,B) and Hybrid models (C,D). A) SVM performance in NLST test set measured by PR-AUC curve (left) and ROC-AUC curve (right). B) SVM performance in UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right). C) Hybrid AI model performance in NLST test set measured by PR-AUC curve (left) and ROC-AUC curve (right). D) Hybrid AI model performance in the UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right). Numerical values in all panels are AUC values for each corresponding year.

Following initial model optimization of clinical risk features, we then incorporated an image-based lung risk predictor of the ResNet-18 CNN image classifier with additional training on LDCT images from NLST, Sybil¹⁴, which infers lung cancer risk probabilities at multiple years of follow up. The hybrid risk model was trained with the previously identified 7 clinical features in the SVM but with inclusion of the inference of the multiyear risk from the image based ResNet model from 65,161 LDCT study DICOM files. The Hybrid model demonstrated substantially improved accuracy with ROC-AUC values Y1-Y6 all significantly improved (p-values < .001, Supp. Table 4). We then benchmarked the hybrid model on the NLST test set compared to SVM models trained with clinical risk features alone which resulted in a substantial improvement of predictive potential (Fig 3C; Table 2; ROC-AUC ranging from 0.78-0.91; PR-AUC values 0.25-0.33). Positive predictive values and negative predictive values at a sensitivity of 80% (corresponding to a PLCO_m2012 6-year risk of 1.6%)¹⁶ and found an order of magnitude improvement in positive and negative predictive values when compared to the PLCO_m2012 risk predictive model (Supp. Table 3)¹⁶.

We then evaluated whether our hybrid model’s predictive ability could be reproduced in the diverse UIH cohort. We observed a substantial decrease in model performance (Fig 3D; ROC AUC 0.68-0.80). Given the similar performance of the clinical feature SVM model between the NLST and UIH cohorts, we focused on the ResNet component of the Hybrid model. To confirm the ResNet model performance was reproducible, we re-ran the model on individual CT scans from the NLST that were not used for the ResNet model training and confirmed findings as previously reported¹⁴ (Fig. 4A). We then evaluated baseline LDCT scans from the UIH cohort and identified reduced performance in the UIH cohort with Year 1 ROC-AUC values of 0.8 versus 0.92 in NLST, for example (Fig. 4B). We then asked whether model performance deterioration¹⁷ was due to a technical difference of the LDCT scans (e.g. modern CT scanners and image reconstruction in UIH versus NLST) or due to the demographic differences such as race. Evaluating performance of the ResNet model on White patients in the UIH showed no differences in performance (Fig. 4C) between the NLST and the UIH cohorts. However, when assessed in Black patients, there was marked deterioration in model performance with Y1-Y6 ROC AUC values 0.65-0.75 (Fig. 4D). We also evaluated other potential factors of model deterioration and found differences in BMI where model performance had significant deterioration in obese patients (BMI ≥ 30) (Supp. Table 4) where in the NLST cohort Y1 ROC AUC was 0.94 in BMI < 30 versus 0.88 in BMI ≥ 30 subjects. In the UIH cohort, we observed a more pronounced model deterioration where Y1 ROC AUC was 0.90 BMI < 30 versus 0.67 in BMI ≥ 30 subjects (Supp. Table 4). Notably, we found no correlation between race and BMI to suggest that these clinical features were linked to model deterioration.

• Download figure
• Open in new tab

Figure 4.

PR and ROC-AUC curves for the ResNet model alone in NLST and UIH Cohorts. A) ResNet performance in NLST test set measured by PR-AUC curve (left) and ROC-AUC curve (right). B) ResNet model performance in UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right). C) ResNet model performance in White subjects in UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right). D) ResNet model performance in the African American subjects in the UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right). Numerical values in all panels are AUC values for each corresponding year.

Because of the racial disparity in model performance, we subsequently inquired whether this could be mitigated by incorporating clinical information along with LDCT images. We utilized a simple two-layer ANN given ease of training to train a large number of unique models. We then used the NLST training cohort for ANN model training, 10% of the UIH cohort for testing, and completely held out 90% of the UIH cohort for validation. The ANN models were trained with the following three inputs: ResNet predicted lung cancer risk, subject’s age, and subject’s race. 100 ANN models were trained with Year 1 ROC AUC performance ranging from 0.898 to 0.908 in the NLST test set (Fig 5A). We then tested on 10% of the UIH cohort (UIH validation set) and utilized the top 10 performing ANNs to be evaluated in the held out UIH test cohort. The results were mixed, where in the overall UIH cohort Y1 ROC AUC values increased to 0.83 (Fig 5B) as compared to the ResNet model alone (Fig 4B). When evaluating this difference across races, we observed that ROC AUC values modestly increased by 0.01 to 0.02 across Years 1-6 (Fig 5D; Table 3) in Black subjects as compared to the ResNet model alone (Fig 4D). However, in White subjects, there was a substantial decrease in performance in Y2-6 ROC-AUC values by 0.10 (Fig 5C) as compared to the ResNet model alone (Fig 4C). While domain shift - which refers to the situation where the distribution of the training data differs from the distribution of the test data - is a significant factor in the performance discrepancies observed, the bias in the training data and model generalization also play critical roles. The observed results indicate that while the new model has improved overall performance, ongoing disparities reflect the lingering impact of the initial training data biases and the complexity of the population-specific factors influencing lung cancer risk predictions.

• Download figure
• Open in new tab

Figure 5.

PR and ROC-AUC curves for the hybrid models (ANN+ResNet) in NLST and UIH Cohorts. A) Hybrid ANN performance in NLST test set measured by PR-AUC curve (left) and ROC-AUC curve (right). B) Hybrid ANN model performance in UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right). C) Hybrid ANN model performance in White subjects in the UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right). D) Hybrid ANN model performance in African American subjects in UIH test set measured by PR-AUC curve (left) and ROC-AUC curve (right).

DISCUSSION

Lung cancer disparities are extensively documented in minority populations with respect to survival outcomes, clinical management, and screening. Recognizing these disparities as a potential means to uncover biological factors, we developed a hybrid AI model that integrates extracted clinical and lung parenchymal imaging features for risk inference. This novel approach not only addresses the limitations of traditional screening strategies but also enhances sensitivity and specificity for diverse populations. While LDCT screening has proven effective in reducing mortality through early detection and improved survival^2,18, existing guidelines have been shown to perpetuate inequities. The USPSTF LDCT criteria¹⁸, primarily based on the nearly 95% White NLST study population² has been shown to contribute to disparities where Black populations screening eligible population carries twice the risk of lung cancer as compared to White populations^4,5. Alternative strategies have been proposed including utilizing clinical risk models to determine screening eligibility^5,7. While these strategies demonstrate increased sensitivity as compared to USPSTF criteria, they have several limitations including patient selection only at long-term risk levels > 1-1.5% lung cancer risk, lack of risk adjustment from LDCT findings, and limited PPV and NPV values. When the hybrid model was evaluated in the diverse real-world cohort within the Chicago metropolitan area, we noted the hybrid model demonstrated higher performance among populations identifying as White as compared to Black, potentially reflecting bias from a training set of a >90% White population. Bias in AI models is an increasing concern, particularly among institutions serving diverse patient populations and may further exacerbate health disparities^19,20. However many initiatives are aiming to reduce bias in AI development including AIM-AHEAD^21,22 and specifically in lung cancer screening, the National Cancer Institute’s CANDID-4AI²³. Because of the existing disparities in lung cancer including the higher risk of lung cancer observed among Black patients who smoke, implementation of AI models will require representative data in training sets and inclusion of Black populations and other minorities in validation cohorts and studies. Furthermore, the domain shift we observed suggests that lung parenchymal features in White LDCT images have different features and latent representations than in Black individuals. The biological basis of this might be related to the lung parenchyma in Black individuals who smoke, which requires further research and radiomic, histologic and cellular characterizations. In addition, we observed model deterioration in individuals with a BMI ≥ 30. While the underlying mechanism for this deterioration in both the NLST and UIH cohorts was outside of the scope of this study, we speculate that latent features within lung parenchyma may be inadequately visualized due to a biological obesity paradox that confounds our hazard risk models^24,25. Alternatively, chest wall thickness from subcutaneous adipose tissue may interfere with ray penetration, or cause compression of lung parenchyma due to obesity hypoventilation.

This study has limitations, chiefly those of the retrospective nature of the UIH cohort which is subject to bias and data missingness. We are aiming to address these limitations with a prospective study on new LDCT participants across multiple sites. The hybrid model also utilizes a CNN architecture trained on the NLST cohort which has limited representation of Black and other minoritized populations. Current efforts such as the CANDID-4AI initiative²³ to develop more inclusive training sets may eliminate these biases and overcome domain shifts as we observed in our assessment of the ResNet model¹⁴ where performance in White subjects decreased when the model was optimized for Black subjects. Having an inclusive and representative training set will allow ResNet and more advanced deep learning architectures with race labeling to understand underlying structures of racial disparities in lung cancer incidence and incorporate these features and enhance model predictive ability and generalizability.

Collectively, we demonstrate an integrated approach combining imaging features from human lung parenchyma and clinical characteristics that elucidated biological contributors to lung cancer. The risk prediction model is capable of predicting lung cancer in two distinct patient cohorts, though it suffers from deterioration in Black populations and individuals with a BMI ≥ 30. Future studies should be conducted aimed at understanding the biological explanation underlying increased lung cancer incidence in Black populations through investigation of the lung parenchyma in Black individuals who smoke or in tumor adjacent normal tissue. Understanding these biological factors would increase our understanding of lung cancer, open up new avenues for prevention, and potentially enable biological based strategies that move beyond USPSTF criteria to create an inclusive and individualized paradigm for lung cancer screening.