ABSTRACT
Importance Critical care resources like ventilators, used to manage the current COVID-19 pandemic, are potentially inadequate. Established triage standards and guidelines may not contain the most appropriate severity assessment and outcome prediction models.
Objectives Develop a draft pandemic specific triage assessment score for the current COVID-19 pandemic. Design a website where initial Toward a COVID-19 Scores (TACS) can be quickly calculated and used to compare various treatment strategies. Create a TACS Registry where data and outcomes for suspected and confirmed COVID-19 patients can be recorded. Use the TACS Registry to develop an influenza epidemic specific database and score for use in future respiratory based epidemics.
Design, Setting, Participants Retrospective analysis of 3,301 ICU admissions with respiratory failure admitted to 41 U.S. Intensive Care Units from 2015-19. Independent external validation on 1,175 similar ICU Admissions using identical entry criteria from Barnes Jewish Hospital (BJH), Washington University from 2016-2019.
Main Outcomes TACS was created with 16 readily available predictive variables for risk assessment of hospital mortality 24 hours after ICU Admission and the need for prolonged assisted mechanical ventilation (PAMV) (> 96 hours) at 24- and 48-hours post ICU admission.
Results TACS achieved an Area Under the Curve (AUC) for hospital mortality after 24 hours of 0.80 in the development dataset; 0.81 in the internal validation dataset. At a probability of 50% hospital mortality, positive predictive value (PPV) was 0.55, negative predictive value (NPV) 0.89; sensitivity 22%, specificity 97%.
For PAMV after 24 hours, the AUC was 0.84 in the development dataset, 0.81 in the validation dataset. For PAMV after 48 hours, the AUC was 0.82 in the development dataset, 0.78 in the validation dataset.
In the external validation the AUC for TACS was 0.76 +/- 0.024. We launched a website that is scaled for mobile device use (https://covid19score.azurewebsites.net/) that provides open access to a user-friendly TACS Calculator for all predictions. We also designed a voluntary TACS Registry for collection of data and outcomes on ICU Admissions with COVID-19.
Conclusions and Relevance Toward a COVID-19 score is a starting point for an epidemic specific triage assessment that could be used to evaluate various approaches to treatment. The TACS Registry provides the ability to establish a respiratory specific outcomes database that can be used to create a triage approach for future such pandemics.
Question Can a rapid epidemic specific risk assessment severity score and data and outcome repository be constructed in the midst of a pandemic.
Findings Using development and validation datasets with ICU admissions similar to those developing COVID-19, developed an initial Toward a COVID-19 Score that could be used to compare various treatment approaches. Also launched an online facilitated data collection and outcome assessment registry for collection of a pandemic specific database so a new triage score could be created for use in the next pandemic.
Meaning In the midst of a pandemic rapid development of an epidemic specific triage score and a data registry for the creation of a new score for use in future pandemics appears feasible.
Introduction
Critical care resources like ventilators, used to manage the COVID-19 pandemic, are reported inadequate worldwide 1. In the US established triage standards, such as Ventilator Allocation Guidelines developed by New York State 2, by Johns Hopkins 3, and Pennsylvania4, have been recently endorsed5. These represent substantial efforts to design an ethical allocation system for a range of disaster scenarios.
One component of these may not be optimal in the current pandemic: sole reliance on the Sequential Organ Failure Assessment (SOFA) score for severity measurement6. SOFA was developed primarily for use in hospitalized sepsis. It has one respiratory measurement: the PaO2/FIO2 ratio. The other five measures document deterioration in other organ systems. Published data consistently report many COVID-19 patients did not develop multi organ failure during the Intensive Care Unit stay7–9. US and European colleagues treating COVID-19 patients confirm that many deaths are from primary respiratory, not multiple system, failure 10. SOFA scores for triage in a COVID-19 pandemic may then not discriminate who would benefit from intubation and only be useful late in a patient’s course, after they have received multiple days or weeks of mechanical ventilation. More so, since COVID-19 patients will presumably have lower SOFA score at the time of intubation compared to septic patients they will be allocated a ventilator even though current articles report a significantly higher mortality7,11.
Our objective was to begin a process whereby a primarily respiratory based severity assessment measure that also incorporated other known patient characteristics associated with this COVID-19 epidemic and patient outcomes could be established. More importantly, we sought to establish a COVID-19 Registry so a database that could be used to develop a new score for future pandemics.
Summary of Methods
We designed and validated an initial TACS based on 3,301 adult patients with a clinical diagnosis of respiratory distress admitted to 41 ICUs in the US from 2015-19.
We performed an additional external validation of the TACS in 1,175 ICU Admissions to Barnes Jewish Hospital (BJH), Washington University from 2016-2019.
Statistical Methods
The statistical models were initially developed and internally validated on admissions from 1/1/2015 through 9/30/2015 in Medical Decision Network’s, Charlottesville, Va. (MDN) Phoenix ICU database. Phoenix provides data critical to research efforts such as the development of TACS as well as providing insight and understanding to clinicians and administrators making front line decisions that impact patient care.
The dataset provided by MDN for the development of TACS consists of 3,301 de-identified admissions admitted to 41 adult ICUs. To qualify, an ICU had to send electronically captured vital signs to MDN. These ICUs were situated in the Southeast, Northeast, Mid-Atlantic and Pacific West regions. Patients had to be 18 years and above and have one of the following admitting diagnoses: asthma, COPD, pneumonia (bacterial, viral, or parasitic), pulmonary edema, respiratory arrest, restrictive lung disease (fibrosis, sarcoidosis), sleep apnea, or hemorrhage/hemoptysis.
Variables considered as predictors were organized as demographics, comorbidities and, pulmonary physiology parameters. Vital signs and labs were formatted dependent on methodology taken from the Acute Physiology, Age, and Chronic Health Evaluation (APACHE) system 12–14. The vital signs data were aggregated into minimum and maximum value within the first 24 hours after admission (e.g. max_hr24) and within 24 and 48 hours (e.g. max_temp48). The same was carried out for albumin and ph. Based on clinical knowledge the following vitals and labs were entered into the multivariable models:
Maximum heart rate
Maximum respiratory rate
Minimum mean arterial pressure
Maximum temperature
Minimum albumin
Minimum pH
Missing values for albumin were imputed as the median value of 2.80, while missing pH values were imputed with the median value of 7.39. None of the min/max vital signs had missing values.
Along with vital signs and the two lab values, a patient’s age, gender = female, square root of the length of time between hospital admission and ICU admission, (This was done to reduce the effect of extreme (long stay) patients),. minimum PaO2:FiO2 ratio during the first 24 hours after admission, maximum Glasgow Coma Score during the first 24 hours after admission, the presence or absence of the following comorbidities: COPD, congestive heart failure, chronic alcoholism, body mass index > 30, and chronic tobacco use. When arterial blood gases where not available, PaO2/FiO2 ratio was imputed by the APACHE methodology, = 80/.0.21. Another variable combined seven comorbidities indicative of immunosuppression was derived by a patient having one or more of the following comorbidities based on the APACHE methodology: AIDS, cirrhosis, leukemia, lymphoma, metastatic cancer, hepatic failure, and diabetes. Finally, interaction terms were created for age with COPD, congestive heart failure, body mass index, and male gender. All of these variables were chosen based on clinical expertise, the emerging literature describing COVID-19 cases, and descriptions provided by front-line practitioners and no addition or removal of variables was performed based on statistical significance. To account for the effects of when a patient was placed on mechanical ventilation, a variable called “initMV” was computed as the duration between ICU admission and hospital admission. If greater than 1.00 days, this value was set equal to zero for the 24-hour model (see below); for the 48-hour models (see below) initMV was truncated at 2.00 days.
The MDN data set was divided randomly 2:1 into development and validation data sets. A logistic regression procedure was carried on the development data set. Variables remained in the model regardless of statistical significance, as they were deemed clinically important. Model accuracy was determined by the area under the receiver operating characteristic curve 15and the ratio of observed to predicted outcomes. The coefficients from the development data set model were then fed into the validation data set, and the same statistics on this group of patients were obtained. Finally, a cut-point probability of 0.50 was used to calculate the positive predictive value, negative predictive value, sensitivity and specificity of the model. There were three outcomes for which the above procedures were carried out: hospital mortality given labs and vital signs at 24 hours, PAMV given labs and vital signs at 24 hours, and PAMV given labs and vital signs at 48 hours. For the outcome mortality at 24 hours, only a patient’s first ICU admission was included (to avoid counting one patient twice). When developing the models with PAMV as the outcome, all ICU admissions were included, except patients not in the ICU after 24 and 48 hours respectively were excluded.
For external validation, the 24h TACS mortality model was replicated in the Barnes Jewish Hospital dataset. The patients were selected with the same entry criteria as our developmental and validation datasets (see statistical appendix).
Results
Table 1 provides demographic and clinical characteristics of the MDN and Washington University -BJH datasets, respectively. All values given are the mean or percentage, depending on the variable’s distribution. The MDN data set had patients who were older, had less immunocompromised comorbidities, longer previous length of stay, ICU length of stay, and a higher percentage of patients with prolonged acute mechanical ventilation (“PAMV”; duration on ventilation > 96 hours) (14).
Demographic and clinical characteristics of the patients in the MDN and WU datasets. Continuous variables have mean and the Intra-quartile range.
Tables 2a, 2b, and 2c give each model’s coefficients, standard error, and p-value. Figures 1a, 1b, and 1c show each model’s calibration curve: observed vs. predicted outcome over deciles of predicted outcome within the MDN dataset.
In-hospital mortality based on variables collected during the first 24-hours within the MDN dataset
Duration of ventilation greater than 96h based on variables collected during the first 24 hours within the MDN dataset.
Duration of mechanical ventilation greater than 96-hours using data collected from 24 to 48 hours within the MDN dataset.
Table 3 displays the performance for predicting hospital mortality within the first 24 hours of ICU admission and prolonged ventilation within the first 24 and 48 hours. TACS achieved an Area Under the Curve (AUC) for predicting hospital mortality after 24 hours of ICU treatment of 0.80 in the development dataset; 0.81 in the internal validation dataset. At a probability of 50% for hospital mortality, the positive predictive value (PPV) is 0.55, negative predictive value (NPV) 0.89; sensitivity 22%, specificity 97%.
Results from the multivariable TACS models to predict hospital mortality at 24 hours after admission, and prolonged acute mechanical ventilation at 24 and 48 hours, respectively.
For prediction of the need for prolonged mechanical ventilation after 24 hours of ICU treatment, the TACS AUC was 0.84 in the development dataset, 0.81 in the validation dataset. For predictions of prolonged mechanical ventilation after 48 hours of ICU treatment, the TACS AUC was 0.82 in the development dataset, 0.78 in the validation dataset.
We also performed an external validation of the TACS 24-hour mortality model on 1,1175 ICU patients to Washington University/BJH treated between 2016-19. Mortality prediction at 24 hours the TACS AUC was 0.76 +/- 0.024.
DISCUSSION
We have developed an initial model of a respiratory oriented Toward a COVID 19 Score designed to be useful in possible triage decisions and to compare outcomes from various treatment approaches in the current pandemic. It was designed specifically with this pandemic in mind. We used the early reports from China and discussions with front line practitioners treating COVID-19 patients as the basis for selecting predictor variables. While it focuses on respiratory physiology parameters, it also has seven possible choices for immunosuppression among other pre-existing conditions, as these are well-documented and very important risk factors. The final model is broad enough to include all possible current and future COVID trajectories but also narrow enough to focus on death from respiratory causes such as Adult Respiratory Distress Syndrome (ARDS).
While previous attempts at developing an ARDS mortality prediction models were inferior to more general multi organ failure scores such as APACHE, those models combined a very heterogenous ARDS population 16The model relied on age, bilirubin, hematocrit and net volume balance over 24hrs. Pneumonia severity prediction tools including respiratory parameters have maintained their predictive performance (curb-65, psi). Since current reports describe COVID-19 pneumonia as predominantly a single organ failure we decide to focus on respiratory parameters.
A precursory study by Luo et al described that direct ARDS had fewer organ failures and different mortality predictors than indirect ARDS (e.g. non-pneumonia sepsis) despite similar mortality rates. The lung injury scores correlated with mortality in direct ARDS but not in indirect ARDS 17. In our study we also focused on lung physiology parameters along with variables that have been described in COVID reports.
In rationing situations, the main use for a risk stratification score would be to discriminate survivors and non-survivors among the critically ill so patients more likely to survive will be allocated a ventilator. A secondary use would be serial measurements to assess the trajectory of intubated patients and predict patients who will require prolonged mechanical ventilation. A third use would be to compare outcomes from various treatment strategies. We initially focused on these objectives. While earlier reports from China showed that higher SOFA scores were associated with higher mortality, all hospitalized patients were included 18. More than half of the patients were admitted to the general wards therefore comparing non-critically ill and critically ill patients. This comparison will not be relevant in crisis situations when the selection criteria will be applied solely to critically ill patients. This is why in our model, we included patients who had an ICU stay. Other studies have used SOFA in critically ill patients but with very small samples sizes, only 9 outcomes thus questioning the validity of the analysis 19. When analyzing 52 critically ill patients, Yang et al found the initial 24hr SOFA score was 4 in survivors compared to 6 in non-survivors 19. The traditional cutoff for SOFA is < 6 which correlates with a mortality of 10% 20. The findings by Yang et al reinforce the concept that early SOFA will not discriminate between survivors and non-survivors as most patients present to the ICU in single organ failure. In the study by Yang et al, the in-hospital mortality was 61.5% although the SOFA score in non-survivors predicted a mortality below 10%.
The final TACS 24 hour mortality model displayed good discrimination within the independent external Washington University dataset. We anticipate, however, that when the model is applied to a COVID-19 ICU cohort, the observed hospital death rates will be much higher than the 13% −14 % found in our derivation, validation, and external datasets 9. Under these circumstances, the model’s PPV at higher probabilities will be strengthened. In a pandemic, patients with the highest likelihood of survival should have priority access to therapy. It is our expectation that TACS should be able to identify early the patients more likely to benefit from life support when resources are limited.
LIMITATION
The major limitation of TACS is that it has not been applied to patients with a confirmed COVID 19 diagnosis. This is a major question. But, waiting for a substantial database of COVID-19 cases with confirmed outcomes in this country, meant there will have been many deaths and many decisions made with only a potentially adequate severity/outcome estimate, SOFA, as part of the decision making. We have therefore chosen to release it now with the intent of being able to acquire new COVID 19 cases as quickly as possible and improving upon this base. We will commit to providing new data on TACS’s performance with COVID-19 patients as soon as possible. We also acknowledge that we will need to create outcome predictions for later in the ICU stay. These will be a major focus of the TACS Registry along with development of a database in order to create a new revised score for use in the next such pandemic.
WEBSITE AND VOLUNTARY EPIDEMIC SPECIFIC REGISTRY
We have launched a website with mobile access (https://covid19score.azurewebsites.net/) that give open access to a user-friendly TACS Calculator for all predictions. Because TACS was developed on non-COVID cases, we believe it can be applied to other causes of respiratory failure as appropriate. A voluntary registry option reporting subsequent patient outcomes will enable us to further validate and make improvements in the score and our understanding of COVID-19. The TACS registry requests registration and the recording of patient outcomes (hospital mortality, hospital LOS, and duration of mechanical ventilation). It also enables data to be recorded over the course of the hospital stay. Collection of a pandemic specific database will allow use to create a new respiratory based resource for use in future such pandemics.
Conclusion
The Toward A COVID-19 score is designed to be used within an overall rationing approach when critical care resources are scarce (2-5). It provides an earlier and conceptually superior prediction of outcomes compared to SOFA. It also can be used to compare outcomes from various approaches to treatment of COVID-19 ICU admissions. Finally, we have established a COVID-19 Registry to collect and store data and outcomes on patients treated during this pandemic. TACS Registry data will hopefully be available to develop a new score for use in the next such pandemic (21).
Data Availability
The full equations are published in the manuscript. The accompanying website and registry are open to all users.
Footnotes
Development of Toward a COVID-19 Score and Registry was supported by The Gordon and Betty Moore Foundation and by Washington University. We thank Medical Decision Networks for access to portions of their database. We also acknowledge Yinjiao Ma, MS MPH and Kimberly A. Reske, MPH Division of Infectious Diseases, Washington University for support.
Approved under Washington University IRB Protocol. Number 202003158.