Projecting COVID-19 disease severity in cancer patients using purposefully-designed machine learning

Study population and clinical variables collected

We analyzed patients admitted to Memorial Sloan Kettering Cancer Center with laboratory-confirmed SARS-CoV-2 (COVID-19) infection from March 10, 2020 (when testing first became available at our institution) to May 1, 2020. The Memorial Sloan Kettering Cancer Center Institutional Review Board granted a Health Insurance Portability and Accountability Act (HIPAA) waiver of authorization to conduct this study.

We aimed to study patients specifically hospitalized for COVID-19 illness by including all patients admitted between 5 days prior, to 14 days after, diagnosis of SARS-CoV-2 infection. COVID-19 patients who were not hospitalized, or who were admitted outside of this window were not included.

An overview of our analysis is shown in Figure 1. For each patient, we extracted and curated 267 clinical variables (Table S1). These included 26 cancer-related variables (e.g., the underlying cancer type, cancer-related medications); 6 demographics variables (e.g., age, sex, race, BMI); 195 variables for preexisting diagnoses (using ICD-9-CM and ICD-10-CM diagnostic code groups; e.g., I1: hypertensive diseases, J4: chronic lower respiratory diseases); 27 clinical laboratory variables (e.g., D-dimer, albumin, lactate dehydrogenase); and 13 radiology variables (e.g., patchy opacities, pleural effusions).

• Download figure
• Open in new tab

Figure 1: Overview of the study.

A) Data for 354 inpatients at Memorial Sloan Kettering Cancer Center were analyzed. For each patient, up to 267 clinical variables were collected, describing the patient’s cancer history, demographics, ICD medical history, laboratory work, and radiology work. Variables were only collected up to the patient’s COVID-19+ date (time zero). B) Variables are inputted into a machine learning algorithm (a random forest classifier), which learns to predict patient outcomes based on interactions between multiple variables. C) Three possible patient outcomes. Of the 354 patients, 212 did not require high levels of oxygen support, 71 required oxygen support within 3 days of being tested positive for COVID-19, and 71 patients required oxygen support after 3 days.

Importantly, we only used clinical variables that were collected on or before a patient’s COVID-19 diagnosis date. For clinical laboratory values, only the most recent value was used. To reduce redundancy, groups of highly correlated variables (Pearson r > 0.90) were removed, and one random variable from the group was kept. Variables could be either mutually exclusive (e.g., indicator variables for a patient having an abnormal vs. a normal X-ray), or overlapping (e.g., having a hematologic cancer vs. leukemia). Overlapping (hierarchical) variables were included to provide the algorithm with multiple resolutions to find discriminating risk factors.

Defining patient outcomes

Patients were grouped into three possible outcomes based on whether and when they required high levels of oxygenation support, which we defined as oxygen delivered via a nonrebreather mask, high flow nasal cannula (HFNC), bilevel positive airway pressure (BiPAP), or mechanical ventilator. Patients requiring high oxygen support within three days (0 to 3 days relative to COVID-19) were deemed “severe-early”. Patients requiring high oxygen support after three days (4 days after COVID-19 or later) were deemed “severe-late”. Patients not requiring high oxygen (i.e., patients who remained on room air and/or standard nasal cannula) for at least 30 days after COVID-19 were deemed “non-severe”.

Overall, our dataset included 354 inpatients: 212 Non-severe, 71 Severe-early, and 71 Severe-late (Table 1).

Machine learning algorithms and validation

To predict patient outcomes, we employed a random forest ensemble machine learning algorithm, consisting of multiple independent classifiers, each trained on different subsets of training variables [16]. These classifiers collectively estimate the patient’s most likely outcome. Our random forest model consisted of 500 decision trees, trained using the information gain criterion, and each with a maximum depth of 10 decision nodes and a minimum of 1 sample per leaf.

The model was evaluated using 10-fold stratified cross-validation, in which 90% of the dataset (approximately, 318 patients) were used to train the model, and the remaining 10% of the dataset (36 patients) were used to test the model. This process was repeated 10 times, such that each subject was assigned to the test set exactly once. This procedure also ensured that each fold had a class (outcome) distribution that approximately matched that of the complete dataset. We report area under the receiver operating characteristic (AUROC) and average precision scores for each class separately using a one-vs.-rest classification scheme [17].

The importance of each clinical variable towards performance was assessed using permutation testing [16], in which values for each variable (column) were randomly permuted over the observations and then model performance was re-assessed using cross-validation; the drop in performance was used as a measure of the variable’s importance.

Comparison of performance to prior work

Previous machine learning studies have reported impressive performance predicting COVID-19 outcomes for non-cancer patients using only a few clinical variables. For example, Yan et al. [7] (Nature Mach. Intell., 2020) report 90+% performance using just three variables (lactate dehydrogenase, C-reactive protein, and absolute lymphocyte count). Huang et al. (Lancet, 2020) reported statistical significance for 10 clinical variables (white blood cell count, absolute neutrophil count, absolute lymphocyte count, prothrombin time, D-dimer, albumin, total bilirubin, lactate dehydrogenase, troponin I, and procalcitonin). Other studies also used many of the same clinical variables [10,13,14]. For a fair comparison, and to test whether variables previously identified as important could also well-predict outcomes for cancer patients, we trained two random forest classifiers on our dataset using only the variables used by Yan et al. and Huang et al., respectively.

Experimental setup and rationale

Wynants et al. [8] recently reviewed 16 prognostic models for predicting COVID-19 severity and concluded that every study had a high or unclear risk of bias. To try and minimize bias in our analytic approach, we followed three guidelines suggested by the authors:

1. Practices to reduce model over-fitting. We used cross-validation, a standard practice in machine learning, to test how well a trained model can predict outcomes on patients it has never seen before. Evaluating models in this way helps to ensure that predictive patterns learned by the model can generalize to new patients whose outcomes are unknown.

2. Using a hybrid of expert clinical opinion and data-driven approaches to select variables. The authors of our study include both clinicians and computer scientists, who collaborated closely to home-in on a set of relevant clinical variables. As an example, using a completely data-driven approach, we found that a class of medications, atypical antipsychotics, correlated highly with disease severity; in fact, including these medications in our model would have increased our reported results by ∼4-5%. However, these medications are frequently given to elderly patients with dementia, and we felt these medications were very unlikely to directly cause severe COVID-19, and far more likely to be confounded by age and/or functional status. So, we removed this variable and instead decided “age” was the more appropriate and general risk factor. Thus, we began with a purely data-driven approach to identify candidate variables, and then iteratively eliminated those that seemed tenuous from a clinical perspective. Our final model was trained using only 55 of the 267 variables (Table S1).

3. Only including patients who had sufficient time to experience their outcome by the end of the study. We evaluated hospitalized patients diagnosed with COVID-19 from March 10 to May 1, 2020, and evaluated outcomes from March 10 until May 15, 2020, to ensure at least two weeks of follow-up for all patients.

Univariates and bivariates weakly correlate with COVID-19 patient outcomes

Figure 2A-F shows that neither of six clinical variables commonly associated with COVID-19 severity (age, C-reactive protein, D-dimer, albumin, lactate dehydrogenase, BMI) are by themselves able to discriminate among the three patient outcomes. Some laboratory variables can only stratify between non-severe and severe-early patients (e.g., Figure 2B, C-reactive protein), indicating that these labs may only be valuable for prognosing immediate risk as opposed to future risk. Others laboratory variables may be more discriminative but were only available for a fraction of patients at time zero (e.g., Figure 2C, D-dimer). Overall, none of the variables we tested were significantly different between all three outcome groups (non-severe, severe-early, severe-late).

• Download figure
• Open in new tab

Figure 2: Individual clinical variables weakly correlate with patient outcomes.

A-F) Each panel shows a variable (y-axis) grouped by patients in each of the three outcomes (x-axis). The number of patients (n) for which the variable was measured is shown for each group. For example, there were 212 non-severe patients and their average age was 59.9 years old. Each bar shows average; error bars show standard deviation. A) Age, B) C-reactive protein, C) D-dimer, D) Albumin, E) Lactate dehydrogenase, F) BMI. G-I) Each panel shows an interaction between two variables (x and y axes). Each patient is represented by a colored dot (red = non-severe, blue = severe-early, green = severe-late). * = P < 0.01, ** = P < 0.001, *** = P < 0.0001, Welch’s two-sample T-test.

We next looked at whether interactions between two variables could be used to increase prediction accuracy. While there are hundreds of pairs of variables to test, Figure 2G-I shows three representative plots using pairs of commonly used labs, none of which show any visually clear clustering of patients by outcome (i.e., clustering of the same colored dots together).

Improved prediction using machine learning

To test if a combinatorial approach, which takes interactions between numerous risk factors into account, may improve projections of COVID-19 severity, we trained an ensemble machine learning algorithm using a wide range of clinical variables (as described in Methods). Clinical variables included those related to the patient’s underlying cancer diagnosis and treatment, laboratory work, radiological work, pre-existing diagnoses (ICD code history), and demographics. We validated our model using cross-validation, in which a portion of the patients were used to train the model, and the model was then evaluated on the remaining or left-out patients, whose outcomes are known but are never provided to the model.

Our model accurately predicted outcomes for COVID-19 cancer patients who required high levels of oxygen support within 3 days of COVID-19 diagnosis (AUC = 0.828 for severe-early patients; Figure 3A). The model achieved fair accuracy in the more challenging instances of predicting severity that occurs after 3 days (AUC = 0.716 for severe-late patients) or that never occurs during the length of the patient’s disease (AUC = 0.715 for non-severe patients). The model maintains an AUC of greater than 0.8 if “severe-early” was defined as all patients that required oxygen support within 4 days of diagnosis (instead of 3 days), but performance then begins to drop at longer time horizons: AUC = 0.821 for ≤ 4 days (81 patients); AUC = 0.791 for ≤ 5 days (88 patients); and AUC = 0.727 for ≤ 6 days (99 patients). These results suggest that prediction is only reliable within a 3-4 day window from the time of diagnosis.

• Download figure
• Open in new tab

Figure 3: Machine learning algorithms improve COVID-19 outcome prediction in cancer patients.

AUROC plots for A) Our method, B) Yan et al. (2020), and C) Huang et al. (2020). AUROCs are reported for each class separately using a one-vs.-rest evaluation scheme. Diagonal dotted line shows random prediction (AUROC of 0.500). Perfect prediction lies at the upper left of the plot (black dot).

Prior work has reported that a small set of clinical variables can serve as a robust “signature” of COVID-19 disease severity [1,7] (Methods). However, we found significantly worse performance using these variables (Figure 3B-C). For example, for severe-early patients, Yan et al. (3 variables) and Huang et al. (10 variables) achieved AUCs of 0.638 and 0.628, compared to 0.828 for our method. Similarly, for non-severe patients, the two studies achieved AUCs of 0.511 and 0.596, compared to 0.715 for our method. We also computed average precision scores (a summary statistic of the precision-recall curve) and found similar gains for our method compared to prior works (Table 1).

Identifying variable and multi-variable interactions that are useful for predicting patient outcomes

Figure 4A shows the top 30 variables that were most discriminative in classifying patient outcomes. These were variables which, if effectively removed from the analysis, would result in a drop in performance (Methods). For example, platelets, ferritin, and AST (aspartate aminotransferase) were the three most important individual labs. Because we used dozens of variables, and many variable combinations may be correlated, we do not expect the loss of one or a few variables to make a significant difference in performance. Nonetheless, many of these variables have been previously identified in the COVID-19 literature (e.g., interleukin 6 [18,19], C-reactive protein [20], D-dimer). Interestingly, there are also variables the model used that are less discussed in the literature, including ferritin [12,21]. Our study also highlights the importance of variables related to cancer diagnoses and treatments on COVID-19 severity; for example, whether the patient had leukemia or lung cancer was particularly discriminative.

• Download figure
• Open in new tab

Figure 4: Important clinical variables identified by the model.

A) The top 30 variables (y-axis) and their importance (x-axis), defined using permutation testing. The category of each variable is listed next to its name: C = Cancer-related, D = Demographics, I = ICD codes, R = Radiology, L = Laboratory. B) The performance of the classifier (y-axis) when trained using variable from each category separately. For example, using only radiology variables, the random forest classifier achieved an AUROC, averaged over all three classes, of 0.605. “All” shows the combination of all variables, achieving an average AUROC of 0.753.

Variables from all five categories (cancer-related, demographics, ICD codes, laboratory work, radiological work) are represented in Figure 4A, highlighting how each clinical category contributes complementary information towards projecting COVID-19 severity. Indeed, classifying patient outcomes using variables from each category individually reduces accuracy compared to when using all variables together (Figure 4B). For example, training the model using only cancer variables produced an average AUROC of only 55.3%. On the other hand, using all variables except cancer-related variables dropped performance by 6.9%. The former means that the underlying cancer type, by itself, is not a very valuable predictor, but the latter suggests that when the cancer type is combined with clinical variables from other categories, its contribution becomes more pronounced and is unique. Similarly, using only radiology variables produced an average AUROC of 60.5%, and using all variables except for radiology variables dropped performance by 7.6%.