Abstract
Title Initial Experience in Validation of a Telemedicine Risk Assessment Tool for Hospitalization of Patients with COVID-19
Importance Risk assessment of patients with acute coronavirus disease 2019 (COVID-19) in a telemedicine context is not well described. In the setting of large numbers of patients, a risk assessment tool may guide resource allocation not only for patient care but also for maximum healthcare and public health benefit.
Objective To determine whether a risk prediction tool developed and implemented in March 2020 (“COVID-19 Telemedicine Risk Tier Assessment”) accurately predicts subsequent hospitalizations.
Design Retrospective cohort study, enrollment from March 24 to May 26, 2020 with follow-up calls until hospitalization or clinical improvement (final call date range March 27 to June 19, 2020)
Setting Single center “Virtual Outpatient Management Clinic” (VOMC) assessing and managing outpatients from a large quaternary medical system in Atlanta, Georgia
Participants 496 patients with COVID-19 confirmed by nasopharyngeal sampling in isolation at home at the time of diagnosis. Exclusion criteria included: (1) hospitalization prior to VOMC enrollment, (2) immediate discharge from VOMC with no follow-up calls due to resolution.
Exposure Acute COVID-19 illness
Main Outcome and Measures Hospitalization was the outcome. Days to hospitalization was the metric. Survival analysis using Cox regression was used to determine factors associated with hospitalization.
Results The risk-assessment rubric assigned 496 outpatients to tiers as follows: Tier 1, 237 (47.8%); Tier 2, 185 (37.3%); Tier 3, 74 (14.9%). Subsequent hospitalizations numbered 3 (1%), 15 (7%), and 17 (23%) and for Tiers 1-3, respectively. From a Cox regression model with age ≥ 60, gender, and self-reported obesity as covariates, the adjusted hazard ratios using Tier 1 as reference were: Tier 2 HR=3.74 (95% CI, 1.06-13.27; P=0.041); Tier 3 HR=10.87 (95% CI, 3.09-38.27; P<0.001). Tier was the strongest predictor of time to hospitalization.
Conclusions and Relevance A telemedicine risk assessment tool prospectively applied to an outpatient population with COVID-19 identified both low-risk (Tier 1) and high-risk (Tier 3) patients with better performance than individual risk factors alone. This approach may be appropriate for optimum allocation of resources.
Introduction
The severity of coronavirus disease 2019 (COVID-19) varies from asymptomatic to life-threatening.1,2 At the time of diagnosis, most patients have mild illness and do not require hospitalization.3 For these patients, the recommendation is to isolate at home and monitor symptoms under the care of a medical provider.4,5 Many U.S. medical centers have employed telemedicine and remote monitoring programs to provide this care.6-8 Monitoring programs require investment and staffing;7 it may be appropriate to focus these resources on those at highest risk of hospitalization for severe COVID-19. While it is recognized that certain groups (e.g. older adults, patients with diabetes)9-12 have higher rates of hospitalization, there are no validated risk assessment tools that stratify risk for outpatients undergoing home monitoring.13 The tools in existence often require in-person criteria (e.g. vital signs, labs, imaging) not available by telemedicine.13,14
In order to better target care for outpatients with COVID-19, we created a risk assessment tool based on published data available in March 2020 (eAppendix 1). This tool incorporates age, comorbidities, symptom severity and course, and the ability to isolate – criteria highlighted in initial Centers for Disease Control (CDC) guidance for home monitoring of patients with COVID-19.4 Based on these factors, patients are assigned to a “risk tier 1-3” (indicating low, moderate, or high risk for hospitalization). For patients seen during acute illness, low-risk patients (Tier 1) must meet all of the following criteria: age <60, no comorbidities known to increase risk of severe COVID-19, no lower respiratory tract symptoms (except mild cough), able to self-isolate. High-risk patients (Tier 3) meet any of the following criteria: age ≥70, younger age with specific high-risk comorbidity or multiple comorbidities, new and/or worsening lower respiratory symptoms, persistent systemic symptoms (e.g. fevers >6 days), or uncertain ability to self-isolate. Exceptions to the criteria included: (1) provider discretion to override the risk assessment tool; (2) patients who appeared to be improving after the second week of illness could be assigned to a low tier even if older age or comorbidities were present.
We prospectively applied this risk assessment tool during the telemedicine assessment of outpatients diagnosed with COVID-19 in a large quaternary academic health system in Atlanta, GA. Patients were followed with regular phone calls (frequency and duration varied by risk tier) until clinical improvement or hospitalization. The clinical care pathway for outpatients with COVID-19 in our clinic, the Virtual Outpatient Monitoring Clinic (VOMC), is outlined in eAppendix 2. In this retrospective study, we analyzed patient data gathered systematically at VOMC intake visits, including patient characteristics and assigned risk tier, and used an outcome of hospitalization related to COVID-19. We hypothesized that the multifactorial tool would predict hospitalization rates. Below, we outline our findings and our initial experience using this risk assessment tool.
Methods
Ethics
The study was approved by the Emory University Institutional Review Board and the requirement for consent was waived as the study posed no more than minimal risk.
Study setting and population
The study is a retrospective cohort investigation of outpatients with confirmed COVID-19 at Emory Healthcare, the largest academic health system in Georgia (serving the greater Atlanta metropolitan area). During the study period, outpatient testing was performed by nasopharyngeal sampling for real-time reverse transcription–polymerase chain reaction (RT-PCR) detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Adult patients with positive RT-PCR results from the screening clinics or emergency departments were called by a result notification team to provide self-care advice and refer for enrollment in the VOMC.
The VOMC comprised an intake team of 14 physicians and 3 advanced practice providers (APPs) from two primary care clinics; and follow-up call teams included 19 redeployed registered nurses (RNs) and 20 APPs. All intake providers were trained in the use of the risk assessment tool in a one-hour webinar and conducted a median of 25 intake visits during the study period [range: 5-99].
Enrollment criteria for this study included: (1) completion of new patient VOMC visit during the period of March 24 to May 26, 2020, and (2) Documentation of positive RT-PCR for SARS-CoV-2. Exclusion criteria include: (1) hospitalization prior to VOMC enrollment, (2) immediate discharge from VOMC (no follow-up calls) due to no care needs.
Outcome
Hospitalization was the primary study outcome, consistent with stated purpose of the risk assessment tool. Emergency department visits and observation admissions were not included as events. Hospitalization at four Emory Healthcare acute care hospitals was determined by Emory Clinical Data Warehouse (CDW) queries, last performed July 6, 2020. External hospitalizations were identified by chart review in (1) clinical notes, (2) administrative messages; and (3) hospitalization documentation in the Emory electronic health record per data sharing agreements with other health systems. Loss to follow-up was minimal because VOMC patients were followed for minimum pre-specified periods with regular telephone calls and notifications of external hospitalizations were documented in the medical record. To validate this approach to identification of hospitalizations, we were able to compare our chart reviews of hospitalizations to patient-reported hospitalizations in a separate long-term follow-up telephone survey of 158 patients.
Covariates
Risk assessment data were obtained for all patients enrolling in the VOMC during a scheduled telemedicine visit utilizing synchronous two-way audio/video communication (with telephone call as backup option). VOMC intake providers completed a standard note template, including demographics, comorbidities (past medical history and specific conditions with elevated COVID-19 severity risk), symptom description (onset, severity, and course), social support and ability to isolate, and clinician-assigned risk tier using the risk assessment tool (eAppendix 1). These data were extracted from the clinical notes by CDW query. Missing data were included, when possible, by manual chart review by the authors (JO, GO) of provider documentation in the intake note. Actual body mass index (BMI) was obtained with a second CDW query. In practice, the patients could change tiers to receive more or less frequent contact based on change in illness severity, but we used data collected at the initial telemedicine visit for analysis.
Statistical methods
Survival analysis was used to determine factors associated with hospitalization to evaluate the risk tier model. Initial unadjusted hazard ratios (HRs) were calculated using a Cox proportional hazards model. A multivariable model was then constructed using a Cox proportional hazards model. Despite good assurance that we had good follow up in terms of hospitalizations we were conservative in our analysis and right censored patients at the last VOMC nurse phone call rather than extend the time variable to the end of the study. Time-varying covariates were identified by individual evaluation of covariates looking at Kaplan Meier curves and testing for a statistically significant time variable interaction. Covariates with a p value <0.05 for the interaction term were considered time-varying. Both forward stepwise likelihood ratio and backward likelihood ratio stepwise selection was performed. The models developed by backward and forward selection were then manually checked by adding and removing individual variables and assessing model fit. Pairwise deletion was used during the exploratory phase for any variables with missing data. The final model did not have any missing data. Logistic regression was also performed with the same variables to shadow the Cox regression analysis. All analyses were conducted using SPSS statistical software version 26 (IBM Corp).
Results
Participant characteristics
We identified 549 patients completing a new VOMC visit from March 24, 2020 through May 26, 2020 and included 496 patients in the analysis after excluding 7 patients without a positive RT-PCR, 26 patients hospitalized for COVID-19 prior to VOMC visit, and 20 patients who met criteria for discharge at the initial visit and were not placed into an initial Tier. Table 1 describes our patient population by tiers. At the initial visit, 237 patients (47.7%) were placed in Tier 1 (low-risk), 185 (37%) in Tier 2 (intermediate-risk), and 74 (14.9%) in Tier 3 (high-risk). The majority (330 patients, 66.5%) are female, 252 (50.8%) are black, and 383 (77.2%) were under age 60 years. Race was unknown or other for 147 (29.6%) of our patient population. Only 174 patients (35%) reported no high-risk comorbidities, with hypertension (175 patients, 35.3%) and reported BMI >30 (147 patients, 29.6%) as the most frequent comorbidities. Ability to self-isolate was documented as adequate for 409 patients (82.5%), inadequate for 9 patients (1.8%), and the rest were missing data. Most patients (316, 63.7%) had mild symptoms or no symptoms at the time of the visit. The mean age was 47.6 years; the mean days from first symptom to visit was 9.3, the mean days from RT-PCR test to visit was 3.7 days; the mean duration of follow-up telephone calls was 13.1 days.
Characteristics of Outpatients with COVID-19 by Assigned Risk Tier
Univariate analysis
Table 2 demonstrates unadjusted hazard ratios, 95% confidence interval, and p value for each factor comparing 35 VOMC patients requiring hospitalization and 461 patients who did not require hospitalization during the follow-up period. Statistically significant factors included risk tier, age, coronary artery disease, diabetes mellitus, heart failure, reported obesity (BMI >30), 2 comorbidities, 3 or more comorbidities, severe symptom rating, and worsening symptom course. Of the patients initially categorized as Tier 3, 17 of 74 (23%) were hospitalized in the course of their care, compared with 15 of 185 (7%) for Tier 2 and 3 of 237 (1.3%) of Tier 1. Among 35 hospitalized patients, the median days to admission from symptom onset was 8 in Tier 3, 11 in Tier 2, and 13 in Tier 1. If we combined Tiers 2 and 3, the Tier model has a sensitivity of 91% and specificity of 51%. Likewise, if we combine Tiers 1 and 2 the Tier model has a sensitivity of 49% and a specificity of 88%. Tier had the highest unadjusted hazard ratio of all factors with 5.29 for Tier 2 and 16.24 for Tier 3 in comparison to Tier 1.
Characteristics of Hospitalized and Non-hospitalized Patients with Outpatients with COVID-19
Multivariable analysis
The final model that predicts hospitalization among outpatients in VOMC as shown in table 3 includes (1) risk tier, (2) reported obesity, (3) age ≥ 60, and (4) gender as strata. This model had an overall fit that was statistically significant (p value <0.001). In checking the proportionality of hazards assumption, gender was found to be a time-varying covariate (eAppendix 3). As a result we analyzed gender as strata.15 The adjusted hazard ratio for Tiers 2 and 3 compared to Tier 1 were 3.74 (95% CI 1.06-13.27, p=0.041) and 10.87 (95% CI 3.09-38.27, p<0.001), respectively. Age ≥ 60 had an adjusted hazard ratio of 2.53 (95% CI 1.27-5.02, p=0.008) and reported obesity had an adjusted hazard ratio of 2.09 (95% CI 1.06-4.13, p=0.034). Survival curves (Figure 1) show days from symptom onset to hospitalization by Tier. Tier 3 patients were hospitalized earlier and at higher rates for both males and females. At 30 days, 24% of males in Tier 3 were hospitalized versus 9% in Tier 2. For females, 16% in Tier 3 were hospitalized while 6% in Tier 2 were hospitalized at 30 days. Males were hospitalized earlier and more often than females. Males reached the maximum number of admissions for males in our cohort occurred by day 23, while females in our cohort reached maximum hospitalization rates by day 41. Logistic regression performed with the same variables to shadow the Cox regression analysis found similar results with adjusted odds ratios (OR) of 4.87 for Tier 2 and 15.38 for Tier 3 compared to Tier 1. Age ≥ 60 and Reported Obesity both had adjusted ORs similar to their adjusted hazard ratios (table 3). Gender was not statistically significant but was kept in the logistic regression model for comparison to the survival analysis.
Hazard Ratios and Odds Ratios for Variables with Significant Predictive Value
Cox Regression Survival Curves Hospitalization by Risk Tier
Sensitivity Analysis
We performed sensitivity analysis for obesity to see if using the actual BMI (Corrected Obesity) would be more predictive that reported obesity. The adjusted hazard ratio for corrected obesity was 3.783 (95%CI 1.761-8.126, p< 0.001) with only minor changes in the hazard ratio and p values for Tier and Age ≥ 60 (eAppendix 4).
Proposed Simplified Tier Model
We looked at factors associated with hospitalization to propose a streamlined risk assessment model to predict patients in the VOMC setting that will not require hospitalization during COVID-19 illness. Table 4 demonstrates the strength of the model using Age<60, no high-risk comorbidities, able to self-isolate, symptom severity mild or none, symptom course stable or improving. This model has no hospitalizations for Proposed Tier 1 patients.
Proposed Simplified Risk Assessment for Low Risk Patients Tested in Study Cohort
Discussion
This study describes the outcomes of outpatients with confirmed COVID-19 who participated in a standardized telemedicine risk assessment and telephone monitoring program. The overall hospitalization rate in this outpatient cohort was 7%, which is lower than other populations reported in the literature.11,16 While the overall study design is retrospective, our program implemented the risk assessment tool prospectively for all new patients with COVID-19 and we were able to follow all patients until clinical improvement or hospitalization because of the availability of redeployed providers (RNs and APPs), minimizing gaps in data.
Improving identification of high-risk patients
We found that the risk tier assessment tool predicted hospitalization risk with highly significant results in multivariate analysis and time-to-hospitalization survival analysis. This supports our hypothesis that inclusion of multiple factors in patient assessment (age, risk factors, symptoms, social factors) would most effectively identify absolute hospitalization risk and time to hospital admission. In the same model, age ≥ 60, reported obesity, and male gender are significant predictors of risk which is consistent with previous studies.10-12 Age and obesity are notable because, despite inclusion in the risk assessment tool, they remain significant even after controlling for tier.
The identification of a small group of outpatients (Tier 3) at highest risk of hospitalization facilitates planning efforts for high-intensity outpatient monitoring with limited follow-up resources and may justify the expanded implementation of the risk assessment tool at the point of care. One question raised by a useful risk-tier rubric is whether it can be codified into a computer-resident algorithm, an artificial intelligence (AI) application. We attempted models with tier as an output rather than input and using the objective and subjective notes and clinical observations as inputs. We were unable to develop such a model, evidently because the tier assignment includes several points where clinical judgment is required and applied. Incorporating that clinical judgment is necessary and is beyond our AI ability at this point.
Identification of low risk patients
The risk assessment tool identified individuals in Tier 1 at low risk of hospitalization. In our VOMC cohort, this group represented a large volume of the follow-up care. Even with less frequent calls and shorter planned follow-up than other tiers (eAppendix 2), this group received 28.3% of the VOMC follow-up calls (internal data: 1741/6160 calls as of July 2). We were able to simplify criteria for Tier 1 for a proposed “new Tier 1” 4-item risk score that may simplify identification of low-risk individuals. With the end of emergency redeployment and staffing reduction, identification of low-risk individuals allows us to reduce the resources allocated appropriately. As additional remote monitoring tools become available (e.g. automated text message survey) this population may be appropriate to assign “as needed” follow-up instead of proactive monitoring calls.
Limitations
As a single-center study, the results of our risk assessment validation may not apply to other patient populations. Due to an initial screening strategy which prioritized healthcare workers, we had a high proportion of working-age individuals and relatively few older adults and socially disadvantaged individuals in the study population. This may explain the lower hospitalization rate than other published studies. The time to enrollment in the VOMC (9.3 days) reflects the real-world practice at our clinic, but limits generalizability to acute settings (e.g. urgent care); if same-day results are available, patients may present earlier in illness course.
While the prospective use of risk assessment, minimal gaps in data, and longitudinal assessment are strengths of this study, it is notable that the patients received different levels of observation (frequency of telephone calls, provider type for calls, and duration of follow-up calls) based on assigned tier, which may impact outcomes. A second limitation is the possibility of loss to follow-up: patients could end VOMC care on request and we do not have direct data for outside hospitalizations. However, no missed hospitalizations were identified in a separate quality improvement project involving long-term follow-up calls to 158 discharged VOMC patients.
The risk assessment tool itself has limitations. First, it was designed based on limited data available from early reports of COVID-19 in hospitalized patients and not derived from an outpatient cohort, since none existed at the time. Second, due to the differential risk posed by age ranges and specific comorbidities, the risk tool is relatively complex. The best use was achieved by training a dedicated provider group familiar with its use, but this limits external validity. Even in the optimal setting, we encountered missing data fields (e.g. symptom severity) and underreporting issues (e.g. self-reported obesity vs. BMI). As a related issue, further refinement of Tiers 2 and 3 will require ongoing analysis of risk factors in outpatients.
Conclusions
Our findings suggest that patients at low risk and high risk for hospitalization may be identified with a telemedicine risk assessment tool incorporating age, medical history, symptom severity, and social factors. The Tier 1 patients in our cohort had low hospitalization rates. We observed increasing odds of hospitalization in Tiers 2 and 3, respectively. External validation of these findings is necessary, but we also recognize that care delivery decisions need to be made immediately in the context of recently escalating cases in the COVID-19 pandemic. It is possible to use these data to create care models targeting highest-risk patients during the highest-risk time periods, but further study of the safety and outcomes of this risk-based approach is needed. This study represents our initial experience with an outpatient telemedicine COVID-19 risk assessment tool. In the absence of clear guidelines on the risk stratification and duration of monitoring of outpatient COVID-19, these data may help guide resource allocation, planning current care structures, and future research.
Data Availability
Data is not available in de-identified form for distribution at this time.
Author Contributions
Dr. David Tong and Mr. Taylor had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: D. Tong, E. Tong, J. O’Keefe
Acquisition, analysis, or interpretation of data: D. Tong, E. Tong, G. O’Keefe, Taylor, J. O’Keefe
Drafting of the manuscript: E. Tong, J. O’Keefe
Critical revision of the manuscript for important intellectual content: D. Tong, E. Tong, G. O’Keefe, Taylor, J. O’Keefe
Statistical analysis: D. Tong, Taylor
Obtained funding: None
Administrative, technical, or material support: All authors
Supervision: D. Tong, E. Tong, J. O’Keefe
Conflict of Interest Disclosures: Dr. G. O’Keefe served on an advisory board of Eyepoint Pharmaceuticals in 2019. It is unrelated to the current work.
Acknowledgements
We would like to acknowledge Dr. Sharon Bergquist, MD and Tina-Ann Thompson, MD for contributions to the risk assessment tool and Dr. David Roberts, MD for the design of the structured intake assessment note. We would also like to acknowledge the members of the Virtual Outpatient Management Clinic including faculty, staff and administrative members of the Paul W. Seavey Comprehensive Internal Medicine Clinic and Emory at Rockbridge Primary Care clinic as well as the physicians, nurses, and advanced practice providers who volunteered from other sites.
Footnotes
Phone: 404-778-5373, Email: jbokeef{at}emory.edu, Address: Emory Clinic A, 1365 Clifton Road NE Atlanta, GA 30322, USA.
Phone: 404-778-1209, Email: etong{at}emory.edu
Phone: 404-315-8889, Email: ttaylor47{at}mac.com
Phone: 404-778-3056, Email: g.d.okeefe{at}emory.edu
Phone: 404-778-7334, Email: dtong{at}emory.edu