Abstract
Background While clinical characteristics and a range of mortality risk factors of COVID-19 patients have been reported, a practical early clinical survival calculator specialized for the unique cohort of patients has not yet been introduced. Such a tool would provide timely and valuable guidance in clinical care decision-making during this global pandemic.
Methods Demographic, laboratory, clinical, and treatment data (from 13 acute care facilities at Northwell Health) were extracted from electronic medical records and used to build and test the predictive accuracy of a survival probability calculator—the Northwell COVID-19 Survival (NOCOS) calculator—for hospitalized COVID-19 patients. The NOCOS calculator was constructed using multivariate regression with L1 regularization (LASSO). Model predictive performance was measured using Receiver Operating Characteristic (ROC) curves and the Area Under the Curve (AUC) of the calculators tested.
Results A total of 5,233 inpatients were included in the study. Patient age, serum blood urea nitrogen (BUN), Emergency Severity Index (ESI), red cell distribution width (RCDW), absolute neutrophil count, serum bicarbonate, and glucose were identified as the optimal early predictors of survival by multivariate LASSO regression. The predictive performance of the Northwell COVID-19 Survival (NOCOS) calculator was assessed for 14 consecutive days.
Conclusions We present a rapidly developed and deployed estimate of survival probability that outperforms other general risk models. The 7 early predictors of in-hospital survival can help clinicians identify patients with increased probabilities of survival and provide critical decision support.
Introduction
The World Health Organization designated coronavirus disease 2019 (COVID-19) a global pandemic on March 11th, 2020, with over 1 million confirmed worldwide cases.1 Estimates of severe disease range from 20-30% and case fatality rates from 2-7%.2,3 As healthcare facilities across the world struggle to provide care for increasing numbers of critically ill patients, many countries are reporting or anticipating significant ventilator and equipment shortages.4-6 The development of evidence-based resource allocation tools and processes will be necessary to ensure that we meet our ethical duty to provide the most benefit for the largest number of people.
In cities across the globe, physicians faced with resource limitations are independently deciding which patients to aggressively resuscitate and ventilate and for whom to withhold artificial respiratory support.6,7 Aiding healthcare workers with robust predictive survival models ensures more informed decision-making and efficient, just resource allocation while reducing physician stress and burnout. An early, simple, and clinically relevant model to predict survival in hospitalized COVID-19 patients brings objectivity to emotionally fraught decisions and conversations with patients and families. There have been no published multivariate models predicting survival in larger cohorts (>100) of patients with COVID-19 for at the time of this study, although reports from China have identified age, Sequential Organ Failure Assessment (SOFA) score, and d-dimer level as potential predictors.8
Our objectives were to use parameters available early to clinicians to characterize and predict survival for hospitalized COVID-19 patients within the largest health system in New York State, the current epicenter of the global COVID-19 pandemic. We consider significant variables reported from previous work and describe the demographics, baseline comorbidities, presenting clinical studies, and outcomes of hospitalized patients with COVID-19. We then present a simple, powerful, and clinically relevant predictive model of patient survival—the Northwell COVID-19 Survival (NOCOS) calculator—for all non-mechanically ventilated patients at the time of hospital admission with parameters available early in the care of all patients. The model utilizes routinely collected data typically available within 60 minutes of patient arrival in the emergency department and predicts hospital survival at a time that permits planning and proper decision-making around goals of care and resource allocation. This actionable model can be easily implemented and used to support providers during the current worldwide crisis.
Methods
This analysis of a COVID-19 survival calculator uses data from a retrospective cohort study that was approved by the Northwell Health Institutional Review Board. It includes all adult hospitalized patients (i.e., those aged 18 and up) with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection by positive result by polymerase chain reaction testing of a nasopharyngeal sample. Patients were excluded if they were placed on mechanical ventilation before presentation to or in the emergency department. These patients’ clinical characteristics and outcomes are described more completely in a prior publication on this cohort study.9,10 Patients were admitted to 1 of 13 Northwell Health acute care hospitals on or after March 1st, 2020, and were discharged or died before April 12th, 2020. Clinical outcomes (i.e., discharges, mortality, and length of stay) were monitored until April 12th, 2020, the final date of follow-up. With approximately 4,844 hospital beds and 672 intensive-care-unit (ICU) beds and serving approximately 11 million persons in Long Island, Westchester, and New York City, Northwell Health is the largest academic health system in New York. Notably, during the current pandemic, the number of general hospital beds and ICU beds has increased substantially and fluctuates daily.
Data
Data were collected from the enterprise electronic health record (EHR; Sunrise Clinical Manager, Allscripts, Chicago, IL) reporting database. Transfers from one in-system hospital to another were merged and considered as one visit. Data collected included patient demographic information, comorbidities, home medications, Emergency Severity Index (ESI; an objective marker of emergency department presenting acuity), initial laboratory values and studies, prescribed medications, treatments (including oxygen therapy and mechanical ventilation), and outcomes (including length of stay, discharge, and mortality). Initial laboratory testing was defined as having been obtained while the patient was in the emergency department. Continuous variables are presented as median and interquartile range (IQR), and categorical variables are expressed as number of patients (percentage). Acute kidney injury was identified according to the Kidney Disease: Improving Global Outcomes (KDIGO) definition.11 Acute hepatic injury was defined an elevation in aspartate aminotransferase (AST) or alanine aminotransferase (ALT) of greater than 15 times the upper limit of normal. Oxygen requirements were collected for the highest requirement level during the emergency department stay. We used the chi-squared test for categorical variables and Kruskal-Wallis for continuous variables across all groups to test for differences by survival status.
Predictive Modeling
LASSO regression was used to identify a small subset out of 85 EHR measurements that, when linearly combined, predict the survival of hospitalized COVID-19 patients (Table 1).12 By including an L1-norm regularization term that promotes sparsity, LASSO regression is well suited for determining the optimal subset of measurements. The magnitudes of the coefficients relate to the predictive values of the normalized measurements while coefficients of non-predictive measurements converge exactly to 0.
The data is normalized by taking the z-score so that all measurements are sampled from a distribution with 0 mean and a standard deviation of 1. The mean and standard deviation of the measurements with non-0 coefficients are stored as model hyperparameters during training and applied to test data. Missing measurements were imputed to the mean.
The regularization factor λ is another hyperparameter that is determined by sweeping λ over a range, evaluating the performance, and choosing the value that corresponds to the optimal tradeoff between maximizing performance and minimizing the number of predictors. After optimizing for λ, the number of predictors was fixed at 7 inputs. The performance is measured as the area under the Receiver Operating Characteristic (ROC) curve.
The training set is evaluated with the model using leave-one-out cross-validation to prevent overfitting in order to estimate the class conditional distributions (survived and expired) of LASSO predictions as Gaussian likelihood functions. The posterior probability that the patient will survive is where pc(x|µc, σc) is the Gaussian likelihood function estimated from the LASSO predictions that have outcomes for class c that is an element of the set containing survived and expired, P(c= Cc) is the prior probability of class c derived from the training set, and x is the LASSO prediction for a patient.
Two instances of the calculator were tested: one fixed, trained on data acquired until March 29th, 2020, and tested daily on new patients; and one retrained daily to incorporate new data. We also tested the predictive value of the SOFA and CURB-65 Score for Pneumonia Severity as well as a linear regression model termed SOFA+ that uses the SOFA score, age, and D-Dimer>1 μg/mL based on a recently published study.8 All models are tested across all days, from March 30th, 2020, to April 12th, 2020, using ROC curves and the Area Under the Curve (AUC) metrics with statistical differences in predictive performance tested using the nonparametric DeLong method.13,14 All analyses were performed in Matlab 2019b (Mathworks Inc.).
Results
Between March 1st, 2020, and April 12th, 2020, of the 5,233 patients admitted with COVID-19, 1,185 died while in the hospital (Table 1). As reported previously, 9 patients who died were more frequently older, white, and non-Hispanic males with a higher comorbidity burden, including coronary artery disease, diabetes mellitus, hypertension, heart failure, and kidney disease. With lower diastolic blood pressure, faster respiratory rate, and lower oxygen saturation, they were generally more acutely ill on emergency department arrival (based upon ESI score). The initial labs were almost all significantly different between survivors and non-survivors (Table 1), although many non-routine labs were not available for all patients. While the length of stay was not different between the groups, expired patients had been far more likely to require mechanical ventilation.
The proposed NOCOS calculator was built after optimizing for L1 regularization parameter lamda, based on out-of-sample AUCs, with multivariate logistic regression choosing 7 out of the 85 possible inputs available in the emergency department as the best predictors of survival upon hospitalization: patient age, serum blood urea nitrogen (BUN), ESI, red cell distribution width (RCDW), absolute neutrophil count, serum bicarbonate, and glucose. The fixed NOCOS calculator was trained using all cases hospitalized until March 30th, 2020. The NOCOS calculator was trained every day and tested using data only from the following day. Both fixed and daily retrained versions of NOCOS were compared to clinical benchmarks SOFA and CURB-65 as well as a variation of the SOFA score.8 Based on the ROC and the AUC values, the daily retrained NOCOS calculator—with an AUC of 0.832 while the fixed NOCOS and SOFA+ variation followed very closely (AUC of 0.825 and 0.830 respectively)—outperformed all other calculators (Figure 1). CURB-65 and SOFA score had significantly lower predictive performance than the three aforementioned calculators (AUC of 0.739 and 0.732 respectively, DeLong’s, p<0.05 when compared to the daily retrained NOCOS); they couldn’t always be calculated due to some missing values for the patients.
Operating points to determine performance of survival predictions for all calculators can be established by choosing thresholds on the probability scores. We chose three different operating points for each calculator and provide the numbers of true positives, true negatives, false positives and false negatives, as well as Positive Predictive Value (PPV) and Negative Predictive Value (NPV) for each case (Table 2). In all cases, daily retrained NOCOS outperformed all other calculators.
The NOCOS calculator also demonstrated stability both in its predictive ability and the selection of the predictors across multiple days. As shown in Figure 2, panel A, the NOCOS calculator maintains an AUC value roughly between 0.8 and 0.9 from March 30th, 2020, through April 12th, 2020, regardless of whether it was trained once or retrained daily. The daily trained NOCOS calculator was significantly more predictive than CURB-65 on 10 out of the 14 days, significantly more predictive than SOFA on 7 out of the 14 days, and significantly more predictive than the fixed NOCOS calculator on 5 out of the 14 days (DeLong’s method, p<0.05). It was not significantly more predictive than SOFA+ on any of the days.
The coefficients of the daily retrained NOCOS calculator, chosen by the LASSO regularization across 7 days, are shown with the counts of the times selected in “Figure 2”: “B.” The final 7 parameters were patient age, ESI, BUN, serum bicarbonate, absolute neutrophil count, RCDW, and serum glucose. Five of these 7 predictors were chosen on at least 13 of the 14 days with the exception of serum bicarbonate and serum glucose, which were both chosen on 6 out of 14 days. Other measurements such as platelet count, body temperature, serum albumin, oxygen saturation, and epidermal growth factor inhibitor (eGFRi) were also chosen on fewer days but were not included in the final build of the model. In the latest iteration of the daily retrained NOCOS calculator (trained with data up to April 11th, 2020), the negative predictors of survival in order of their contribution to the probability estimate are: patient age, BUN, RCDW, absolute neutrophil count and serum bicarbonate (Figure 2 panel C). The positive predictors of survival are ESI (lower scores are more acute) and serum glucose (Figure 2, panel C).
The performance of the NOCOS calculator was also tested when not limited only to the ED values of the 7 parameters of a patient, but also when the latest measurements are used as inputs. Figure 3 shows the performance of fixed NOCOS when tested using the up-to-date values of the seven measurements, with the AUC increasing steadily to values close to 0.91.
Discussion
In this study, we successfully developed a simple and practical survival calculator for hospitalized COVID-19 patients using only discrete and objective data values acquired during the patient’s initial time in the emergency department. Our Northwell COVID-19 Survival (NOCOS) calculator, modeled on over 5,200 COVID-19–positive patients, had an AUC of 0.83 and outperformed other well-established risk calculators, including CURB-65, SOFA, while it performed similarly to COVID-19–specific enhancements to SOFA.8 Developed to be parsimonious and easy to use, the predicted survival probability can be used to assist clinical decision-making and ease physician burden in this unprecedented situation. The output of this calculator (which is freely available at https://feinstein.northwell.edu/nocos) provides an easily comprehensible probability, which can be communicated to physicians and nurses, families, and other administrative teams.
The choice of variables included in our model, which were ascertained from the LASSO regularization, all have clinical face validity. It is well established with many diseases, and particularly with COVID-19, that older age confers an increased mortality risk.8 ESI, a well-established ED triage tool, is an early indicator of presenting severity of illness. Abnormal laboratory values included in our model have all been independently associated with negative outcomes in other populations,15,16 and an elevated BUN (as a maker of kidney dysfunction, in particular) was recently shown to increase mortality risk in COVID-19 patients. 17 Elevated values of RCDW, often suggesting chronic disease states and inflammation,18,19 can also be due to recently reported effects of COVID-19 on iron displacement of the heme molecule, leading to impaired red blood cells as well as free radical formation and toxic effect to the lungs.20 These findings suggest potential therapeutic approaches to reduce sudden decompensation, organ failure, and death of these patients.
A major strength of this work is the development of a powerful predictive model typically usable for clinicians within 60 minutes of a patient’s initial presentation. Although the calculator performs well with these very early measurements, it improves its predictive performance when these measurements are updated throughout the hospitalization of the patient (Figure 3), showing that, as expected, the most accurate prediction is given with the most up-to-date values of the seven measures. We also restricted inputs to commonly collected, discrete, and objective data. Its sheer simplicity and reliance on quantitative measurements makes it generalizable and easy to deploy to all interested stakeholders, including front-line providers and hospital administrators organizing distribution of scarce and limited resources. While we present the calculator output as a probability score, a specific operating point can also be chosen to provide a binary outcome prediction with significant accuracy. Choosing an operating point is left up to stakeholders; local clinical teams have flexibility to adjust thresholds toward a more stringent or risk-averse solution (Table 2), based on the rapidly changing needs during this pandemic.
Calculating estimates of survival or mortality using clinical measurements can extend from simple algorithmic rules and thresholds to linear regression models and more complex machine learning (ML) algorithms. Attempting to augment medical decision-making, studies ranging from modulating single parameters to advanced predictive modeling have been applied to forecast decompensation, mortality, and survival among other clinical outcomes.21-23 Early work with small patient cohorts of COVID-19 has led to models that identify some clinical characteristics that can be applied to predict severe cases (Yan et al., 2020, Jiang et al., 2020).24,25 However, these studies are limited to small numbers of patients as well as the inclusion of qualitative and subjective variables, are prone to mislabeling, and are not always readily available. Our approach benefits from a simple, straightforward formula of typical measurements acquired from ED patients; a patient base at least 20-fold larger than previous studies; and an approach of data-true feature selection based on their predictive value through the LASSO regularization.
Due to the challenging situation during the ongoing COVID-19 global health crisis, there is a need for robust tools to aid in complex clinical decision-making. Using well-known clinical calculators such as SOFA or CURB-65 shows ostensible promise; however, these calculators have limitations in both their accuracy and the ease of collecting necessary measurements to construct these scores. Input variables such as confusion (for the CURB-65 score) and Glasgow Coma Scale (for the SOFA score) are ambiguous, hard to measure, and frequently unavailable. Similar difficulties are encountered when trying a novel combination of SOFA score with age and D-dimer values.8 In our study, 78.3% of patients were missing the D-dimer measurement in the emergency department. In contrast, the NOCOS calculator is based on commonly collected laboratory results and a guideline based ESI triage acuity score. Moreover, the calculator is trained and tested on the patient cohort of interest and can account for the evolving nature of this pandemic by daily or more frequent updates and model retraining.26
The proposed calculator has some limitations. It was designed to be linear with only essential predictors included, and non-linear or convolutional/recurrent models may provide improved performance. Moreover, the model is not integrating additional, more complex information such as radiology X-ray or CT-scan reads. Due to the retrospective study design, not all laboratory tests—including lactate dehydrogenase, interleukin-6, and serum ferritin—were done on all patients, and the performance of these variables could not be adequately assessed. These data were automatically extracted from the EHR database, and some patient-level details could not be extracted. However, our NOCOS calculator aimed to leverage easily obtainable data, obviating the need for sifting through charts to obtain a predictive result.
Given the complexity of data acquisition and model development in the midst of a pandemic, we prioritized the creation and rapid dissemination of a more straightforward, clinically relevant implementation. While the model validation contained patients admitted to hospitals within the New York metropolitan area, we believe it will generalize well given the diverse demographic composition of the region and the Northwell Health patient population.
In an unprecedented way, the severity of the SARS-CoV-2 pandemic has strained hospitals’ resources, including space, materials, and front-line healthcare workers. Providers are often forced to take important clinical decisions under immense time pressure and limited information. Tools that could aid them in these circumstances are timely and important. The Northwell COVID-19 Survival calculator answers a clinical need and provides early information to physicians making a range of difficult-but-critical decisions every day.
Data Availability
The data that support the findings of this study are available on request from COVID19@northwell.edu. The data are not publicly available due to restrictions as it could compromise the privacy of research participants.
Financial Disclosures
The authors report no real or apparent conflicts of interest.
Funding Sources
This work was supported by grants R24AG064191 from the National Institute on Aging and R01LM012836 from the National Library of Medicine of the National Institutes of Health.
Role of the Funding Sources
The views expressed in this paper are those of the authors and do not represent the views of the National Institutes of Health, the United States Department of Health and Human Services, or any other government entity.
Other declarations
The investigators were independent from the funders; Todd J. Levy and Theodoros P. Zanos had full access to the data and can take responsibility for the integrity of the data and the accuracy of the data analysis; Theodoros P. Zanos affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Data Availability Statement
The data that support the findings of this study are available on request from COVID19{at}northwell.edu. The data are not publicly available due to restrictions as it could compromise the privacy of research participants.