Abstract
Background Accurate methods of identifying patients with COVID-19 who are at high risk of poor outcomes has become especially important with the advent of limited-availability therapies such as monoclonal antibodies. Here we describe development and validation of a simple but accurate scoring tool to classify risk of hospitalization and mortality.
Methods All consecutive patients testing positive for SARS-CoV-2 from March 25-October 1, 2020 within the Intermountain Healthcare system were included. The cohort was randomly divided into 70% derivation and 30% validation cohorts. A multivariable logistic regression model was fitted for 14-day hospitalization. The optimal model was then adapted to a simple, probabilistic score and applied to the validation cohort and evaluated for prediction of hospitalization and 28-day mortality.
Results 22,816 patients were included; mean age was 40 years, 50.1% were female and 44% identified as non-white race or Hispanic/Latinx ethnicity. 6.2% required hospitalization and 0.4% died. Criteria in the simple model included: age (0.5 points per decade); high-risk comorbidities (2 points each): diabetes mellitus, severe immunocompromised status and obesity (body mass index≥30); non-white race/Hispanic or Latinx ethnicity (2 points), and 1 point each for: male sex, dyspnea, hypertension, coronary artery disease, cardiac arrythmia, congestive heart failure, chronic kidney disease, chronic pulmonary disease, chronic liver disease, cerebrovascular disease, and chronic neurologic disease. In the derivation cohort (n=16,030) area under the receiver-operator characteristic curve (AUROC) was 0.82 (95% CI 0.81-0.84) for hospitalization and 0.91 (0.83-0.94) for 28-day mortality; in the validation cohort (n=6,786) AUROC for hospitalization was 0.8 (CI 0.78-0.82) and for mortality 0.8 (CI 0.69-0.9).
Conclusion A prediction score based on widely available patient attributes accurately risk stratifies patients with COVID-19 at the time of testing. Applications include patient selection for therapies targeted at preventing disease progression in non-hospitalized patients, including monoclonal antibodies. External validation in independent healthcare environments is needed.
Introduction
COVID-19 is a systemic infection caused by a novel betacoronavirus, SARS-CoV-2.(1) A relatively conserved set of clinical and demographic factors are now recognized to correlate with an increased risk for severe disease requiring hospitalization, mechanical ventilation and death.(2–4) Accurate methods of risk stratifying ambulatory patients at the point of test positivity have many possible applications, including prioritizing patients at highest risk of hospitalization for early treatments aimed to prevent progression to severe disease, such as monoclonal antibodies, which are both limited in availability and also more likely to be effective in high-risk groups. Several risk classification models have been proposed.(3,5–9) We describe development and validation of a simple scoring model to predict hospitalization and mortality in a large cohort of ED and ambulatory patients with COVID-19.
Methods
Intermountain Healthcare is an integrated healthcare system that provides care to more than 1.5 million patients each year in Utah and bordering communities. As part of a systemwide COVID-19 response, Intermountain provides SARS-CoV-2 testing at 32 urgent care facilities, 23 emergency departments, and 16 community drive-up testing sites. During the study period, only polymerase chain reaction (PCR) assays were performed (Thermofisher, Waltham, MA; Cepheid, Sunnyvale, CA; Quidel, San Diego, CA, BioFire, Salt Lake City, UT; Roche, Basel, Switzerland). All testing required an order entered in the electronic health record (EHR) (Cerner, Kansas City, KS) by the ordering clinician through a structured form that requires the clinician to input the patient’s clinical symptoms and epidemiological features. These data are stored in the Intermountain Prospective Observational COVID-19 (IPOC) database, and the enterprise data warehouse. This analysis was approved by the Institutional Review Board at Intermountain Healthcare under #1051342.
We queried the IPOC database for consecutive adult patients with positive SARS-CoV-2 tests from March 25-October 1, 2020. Symptom data were extracted from the electronic test order form while demographic and co-morbidity data were obtained from the IPOC database and data warehouse using the Charlson and Elixhauser definitions (10,11). We defined immunosuppression as: recipient of a solid organ or hematopoietic stem cell transplant, on chemotherapy, biologic or other immunosuppressive agents targeting B or T cell activity, chronic corticosteroids at a prednisone-equivalent dose of 20mg per day or greater for more than 30 days, human immunodeficiency virus complicated by acquired immunodeficiency syndrome (AIDS), heritable immunodeficiency. We defined obesity as body mass index (BMI) of greater than or equal to 30 (12). Symptom and demographic data were complete; comorbidity data were complete insofar as patients had prior encounters in the integrated health system. Mortality data was captured via an existing linkage to state death records.
We used a random number generator to divide the cohort into a 70% derivation cohort and 30% validation cohort. In the derivation cohort data, we fitted a multivariable logistic regression model for hospitalization within 14 days of testing, using clinical and demographic features. Predictors were prespecified before model development based on: clinical features that would be available at the time of testing for all ambulatory and emergency department patients regardless of testing venue (a criterion that precludes, for instance, laboratory data), biological plausibility of association with severity, and reproducibility in other studies in existing COVID-19 literature. We intentionally did not fit a model for mortality, but instead planned a priori to validate the ultimate model against that outcome. Model discrimination was evaluated using the area under the receiver-operator characteristic curve (AUROC) and model fit by evaluating R2 using the Nagelkerke method (13,14).
We included patients who tested in the ambulatory setting as well as patients who tested positive in the emergency department to ensure that the score would be applicable in both environments. However, we recognized that some patients testing positive in the emergency department (ED) are then subsequently admitted. The decision to admit or not is not immediately known to emergency medicine providers who may still wish to use the score to stratify risk to aid in clinical decision making and selection of therapies. However, because patients who are admitted from the ED may have different characteristics than those tested in the ambulatory setting, we planned a priori to perform a sensitivity analysis by repeating the regression above after restricting the cohort to patients who were not admitted to the hospitalization at the time of their test.
We then adapted the original logistic regression model into a simple scoring tool by converting exponentiated β coefficients into weighted point assignments for each variable. We evaluated the test performance characteristics of this simplified clinical prediction tool in the derivation and validation cohorts using AUROC and by calculating the sensitivity, specificity, negative and positive predictive values across the range of scoring thresholds.
Results
From March 25 through October 1, 2020, 22,816 patients had a positive PCR test for SARS-CoV-2. The mean age was 40 years (see Table 1); 11,424 (50.1%) patients were female and 8753 (43.9%) identified as a member of a community of color (either non-white race or Hispanic or Latinx ethnicity). Patients had on average one significant medical comorbidity. 1419 (6.2%) of patients were admitted; of these, 799 (3.6%) tested positive in the emergency department during the encounter that culminated in admission. Overall 93 patients (0.4%) died within 28 days of their positive SARS-CoV-2 assay. Demographic and clinical features were very similar between derivation (n=16,030) and validation (n=6786) cohorts.
In the derivation cohort, the primary multivariable model (see Table 3) demonstrated adequate model diagnostics [AUROC 0.824 (95% CI 0.809-0.840), Nagelkerke R2 0.26]. Age, male sex, self-identification to a community of color, dyspnea and high-risk comorbidities including diabetes mellitus, obesity, immunosuppression and chronic neurologic disease were each associated with significantly greater odds of hospitalization. In an exploratory analysis in which individual comorbidities were replaced in the regression with a count of total comorbidities, the cumulative number of comorbidities was also significant (OR 1.4, 95% CI 1.3-1.5). In the planned sensitivity analysis excluding patients who were tested in the emergency department during their admission to the hospital, the multivariable model had slightly diminished performance [AUROC 0.789 (95% CI: 0.768-0.810), R2 0.164]. Overall, contributions of individual risk factors were similar in this model compared to the model including patients being admitted, (see Table 4) with the exception that the magnitude of risk of dyspnea was less in the ambulatory-only cohort (OR 2.1 vs 3.5), and the odds of immunosuppressed patients without palliative goals of care being admitted were greater (OR 7.0 vs 3.9).
Criteria included in the probabilistic, simplified clinical prediction score are displayed in Table 5. Because cumulative comorbidity count was significantly associated with poor outcomes, we included comorbidities in the simplified tool that were not statistically significant individually in the expanded logistic regression model. In the derivation cohort, the AUROC for the simplified clinical prediction score for 14-day hospitalization was 0.82 (95% CI: 0.81-0.84) and 0.8 (95% CI: 0.78-0.82) in the validation cohort. AUROC for 28-day all-cause mortality in the derivation cohort was 0.91 (95% CI: 0.83-0.94) and in the validation cohort 0.80 (95% CI: 0.69-0.9). The scoring threshold that optimized sensitivity and specificity (by Youden’s index) was 6 with test characteristics of 71.1% and 76.2% respectively (Table 6)(15).
Discussion
Given recent straining hospital volumes and the emergence of promising but limited-availability outpatient therapies for COVID-19, methods are needed to identify patients with COVID-19 at highest risk of progression to severe disease, hospitalization and death. Here we describe a simple scoring model capable of accurately risk stratifying ambulatory and emergency department patients for COVID-19 for subsequent hospitalization and mortality.
One of the primary strengths of this model is the simplified and easily calculable score using features that are widely accessible. In particular, our score does not require laboratory studies, which are unavailable in the majority of ambulatory patients testing positive for SARS-CoV-2. While preserving discriminative value, this simple scoring system has potential to facilitate more widespread clinical application in settings lacking robust integration of informatics. The model was derived and validated in a very large and diverse population in the western United States and is based on risk factors for severe disease that are largely conserved across global populations, including age (12), male sex, overall comorbid burden (13), and shortness of breath at the time of risk stratification. These factors align closely with those included in models derived in other locations and populations (3,5–9,16,17). As a result, we expect that this tool will be generalizable.
Although race and ethnicity are often omitted from clinical prediction models to prevent illegal or unethical profiling behavior, the National Quality Forum recommended that when applications of risk prediction include patient selection for preventive or therapeutic modalities, omission of race or ethnicity can actually cause inequity in healthcare access and worsen outcomes disparity by underestimating risk using other demographic and clinical features alone.(16) In COVID-19, it is now well-recognized that significant outcome differences among communities of color exist with respect to severe illness and hospitalization(19) despite adjustment for age, gender and underlying medical conditions.(5,6) This remains poorly understood and may be due to social determinants of health, inadequate access to healthcare, or poorly-controlled co-morbidities. Because we anticipated application of this risk stratification model to aid in allocating preventive therapies in COVID, we, like other published models(5,6), chose to include race and ethnicity in our score. In future work, more refined socioeconomic, cultural and healthcare access surrogates would be preferable alternatives.
When emergency use authorization (EUA) was granted by the United States Food and Drug Administration for monoclonal antibodies bamlanivimab and casirivimab/imdevimab for administration in non-hospitalized patients with early mild-moderate COVID-19, most states were experiencing peak community transmission, with thousands of new patients per day. It became clear that not only would the supply of drugs be inadequate initially to treat all patients qualifying under EUA criteria, but the capacity to administer infusions without compromising infection control in infusion sites would be even more limited. To address this limited resource situation, the Utah Crisis Standards of Care scarce medications committee was convened with the goal of equitably and efficiently matching available infusion capacity to patients at highest probability of hospitalization most likely to benefit. The simple scoring tool described herein was ultimately adopted because of the simplicity, widely accessible clinical features and validation in a large, representative local population (20). By regularly adjusting the eligibility criteria based on the risk score threshold that best calibrates current infusion capacity to the number of new cases in high-risk strata, this risk-targeted drug allocation strategy has provided an equitable and flexible means of drug delivery in the context of still-uncertain efficacy and limited resources.
Limitations of our study include the retrospective, observational design, and the possibility that comorbidity data may have been unavailable or out of date for some patients in the cohort who receive the majority of their medical care outside our integrated healthcare system. Although the large study population and inclusion of widely recognize features improves the likelihood of generalizability, this will need to be confirmed through external validation.
In this large retrospective cohort study, we identified simple risk factors that can easily be calculated at the bedside without laboratory values to risk stratify COVID-positive individuals for risk of hospitalization and death. Applications include guiding allocation of therapies that are limited in availability. External validation is needed to confirm generalizability in diverse and geographically independent population.
Data Availability
In order to protect patient privacy and comply with institutional data use policy, data used in this study are unavailable to upload to public servers. As required by the Intermountain Healthcare Institutional Review Board, data sharing agreement requests to access deidentified versions of the datasets generated and/or analyzed during the current study may be addressed to the Intermountain Office of Research (officeofresearch{at}imail.org)
Conflicts of Interest
IP reports salary support through a grant from the National Institutes of Health (U.S.A). SB reports salary support from the U.S. NIH, Centers for Disease Control and the Department of Defense; he also reports receiving support for chairing a data and safety monitoring board for a respiratory failure trial sponsored by Hamilton, effort paid to Intermountain for steering committee work for Faron Pharmaceuticals and Sedana Pharmaceuticals for ARDS work, support from Janssen for Influenza research, and royalties for books on religion and ethics from Oxford University Press/Brigham Young University. BW reports partial salary support from a U.S. Federal grant from AHRQ. ES receives partial salary support through grants from the Centers for Disease Control.
At the time of submission, Intermountain Healthcare and the University of Utah have participated in COVID-19 trials sponsored by: Abbvie, Genentech, Gilead, Regeneron, Roche, and the U.S. National Institutes of Health ACTIV and PETAL clinical trials networks; several authors (BW, IP, JB, SB, ES) were site investigators on these trials but received no direct or indirect remuneration for their effort. ES, BJW, SMB and MS are members of the Utah crisis standards of care scarce medication committee.
Author Contributions
Study concept: BJW, JB, IP, PJ, DH, BH, AS, NS, WB, EH, DM, RS, SMB
Study design: BJW, JB IP, SMB, GS
Data collection: BJW, NG
Statistical analysis: BJW, JB, IP, SMB
Interpretation of results: All authors
Manuscript preparation: All authors
Critical review of the manuscript: All authors
