Abstract
Importance The Covid-19 pandemic has placed unprecedented stress on health systems across the world, and reliable estimates of risk for adverse outcomes are needed.
Objective To quantify admission laboratory features associated with mechanical ventilation and mortality risk across 5 Eastern Massachusetts hospitals.
Design Retrospective cohort study using narrative clinical notes and laboratory values through April 28, 2020.
Setting Emergency department and inpatient settings from 2 academic medical centers and 3 community hospitals.
Participants All individuals with hospital admission and documented coronavirus testing across these 5 hospitals.
Main Outcome or Measure Mechanical ventilation, as documented in narrative notes.
Results Among 1,898 hospitalized Covid-19-positive individuals through April 28, 2020, 481 (25.3%) required mechanical ventilation and 241 (12.7%) died. L1-regression models developed in 3 of these hospitals yielded area under ROC curve (AUC) of 0.837 for in-hospital mortality and 0.839 for mechanical ventilation in the other 2 hospitals. In total, 66.7% of deaths occurred in the highest-risk mortality quintile.
Conclusions and Relevance Specific admission laboratory studies in concert with sociodemographic features facilitate risk stratification among individuals hospitalized for COVID-19.
Funding none.
Trial Registration None
Question What are the characteristics and outcomes of individuals hospitalized with Covid-19 in Eastern Massachusetts through April 28, 2020, as determined from electronic health records surveillance?
Findings Across 2 academic medical centers and 3 affiliated community hospitals, among 1,898 hospitalized individuals who tested positive for Covid-19, 481 (25.3%) required mechanical ventilation and 241 (12.7%) died. Markers of red cell immaturity and poor renal function, as well as older age, were associated with mortality. Markers of tissue injury, and male sex, were associated with risk for mechanical ventilation.
Meaning Simple prediction models incorporating admission laboratory studies may assist in risk stratification among hospitalized COVID-19 patients.
Introduction
With the rapid spread of the severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), efforts to predict clinical outcomes and stratify risk have taken on greater urgency as a means of allocating resources and targeting interventions. A recent report of 1099 admitted individuals from China found that 5.0% required intensive care unit (ICU) transfer, and 2.3% required mechanical ventilation1. In Lombardy, Italy, around 16% of test-positive individuals required ICU admission2. In the United States, characteristics of admitted patients may differ somewhat. A recent case series from the Seattle area described 24 ICU-admitted patients, of whom 75% required mechanical ventilation3. In one of the largest U.S. studies to date, among a series of 2634 hospitalized patients in New York who died or were discharged, 12.2% had required mechanical ventilation4.
Given the constrained resources for treatment of COVID-19, particularly with regard to mechanical ventilation, simple approaches to stratifying morbidity and mortality risk at time of hospitalization are needed. In cohorts ranging from 100–200 patients, multiple laboratory studies have been associated with mortality risk, including elevated ferritin, troponin, and C-reactive peptide (CRP)5, elevated d-dimer6, and low eosinophil count.7 A recent large cohort study across 169 hospitals identified cardiovascular disease as a major predictor of in-hospital mortality8.
Electronic health records may facilitate rapid and efficient investigation of clinical cohorts, and form the basis of consortia efforts to address COVID-19 at scale9. Here, we examined records from 2 academic medical centers and 3 affiliated community hospitals in Eastern Massachusetts. As coded clinical data lags by 4-6 weeks or more, and may not be complete until after discharge, we relied on narrative clinical notes and laboratory studies, available with less than 24-hour lag, to determine features of admitted patients. We applied data from 3 hospitals to generate simple base-case models to estimate risk of need for mechanical ventilation, and risk for death, and validated these results in another academic medical center and a community hospital, as a starting point for generalizable efforts at clinical risk stratification10.
Method
Subjects
The full cohort included all individuals age 18 or older hospitalized at any of the 2 academic medical centers and 3 community affiliate hospitals between March 1, 2020 and April 28, 2020, with documented COVID+ test result at any point in this period. For all of these individuals, any available narrative clinical notes from the emergency department or inpatient setting were extracted from the Partners Research Patient Data Registry (RPDR)11 and used to generate an i2b2 datamart.12 Data were augmented with age, sex, race, and ethnicity from the same source. The enterprise laboratory feed was used to extract coronavirus test order and results (LC)INC:94309-2), as well as additional laboratory values (Supplemental Table 1). Laboratory values available in at least 80% of individuals were included in subsequent analysis as continuous measures, after Winsorization at the 99th percentile but otherwise without transformation, along with laboratory-specific high and low flags. As an aggregate measure of comorbidity, age-adjusted Charlson comorbidity index was calculated using coded ICD9 and 10 diagnostic codes drawn from the EHR, but by necessity lagging admissions by 4-6 weeks, as previously described13.
The study protocol was approved by the Partners Healthcare Human Research Committee. No participant contact was required in this study which relied on secondary use of data produced by routine clinical care, allowing waiver of requirement for informed consent as detailed by 45 CFR 46.116.
Hospital Course Characterization from Narrative Clinical Notes
As coded clinical data (e.g., ICD-10 diagnoses or CPT codes) is not available in real time in the RPDR, we developed and validated simple string-based classifiers to identify mechanical ventilation based on narrative clinical notes. A preliminary list of tokens was curated by one of us (RP) based on manual chart review; the strings resulting from these queries were then further reviewed in context to determine specificity, and iteratively revised. We identified date of emergency department evaluation, inpatient hospitalization, ventilation if present, and discharge if present on the basis of index note of that type.
Study Design and Analysis
We included all newly-hospitalized individuals undergoing Covid-19 testing during the study period. The two primary outcomes of interest were mechanical ventilation and overall mortality. (In the former case, mortality was also considered an outcome, because it would otherwise represent a competing risk.) Beyond descriptive analysis, we report appropriate univariate comparisons (i.e., chi square test for binary variables, Student’s t-test for continuous measures) followed by L1-penalized regression, or the least absolute shrinkage and selection operator (Lasso)14, to identify a parsimonious model with sociodemographic features and laboratory values as candidate predictors. Lasso was applied to all participants with complete laboratory studies in 1 academic medical center and 2 community hospitals, and the performance characterized in an additional academic medical center and community hospital. Model fitting used all individuals in the training set, with median imputation of missing data; testing utilized all participants with complete data. Model performance was characterized using standard metrics of discrimination and calibration, focusing on the 5 quintiles of risk determined in the training data set.
Regression models offer advantages in interpretability but fail to consider censoring or competing risk. Therefore, to better characterize model performance in the testing set, we utilized survival analysis, right-censoring at time of hospital discharge or end of available data (4/28/20), presenting Kaplan-Meier curves comparing risk quintile groups. All analyses utilized R 3.6.215. STROBE reporting guidelines for cohort studies were applied.
Results
The 1,898 individuals hospitalized with Covid-19 through April 28, 2020, including 1137 (59.9%) at academic medical centers and 761 (40.1%) at community hospitals, were 54.5% male, 47.4% white, and 6.7% Hispanic; mean age was 62.8 years (Table 1). In all, 481 (25.3%) required mechanical ventilation and 241 (12.7%) died. Median time of follow-up was 6 days (IQR, 3-9). Laboratory values are summarized in Supplemental Table 2.
We utilized L1-penalized regression to train a prediction model based on admission characteristics and laboratory values in one academic medical center and 3 community hospitals (Table 2). For overall mortality notable features associated with elevated risk included red blood cell (RBC) indices (presence of nucleated RBC’s, and RBC distribution width (RDW)) as well as markers of diminished renal function (Table 2A). For mechanical ventilation or mortality (vent+mortality), beyond male gender and elements of the chemistry-20 panel, notable laboratory flags associated with risk include elevation in high-sensitivity cardiac troponin, creatine kinase, and procalcitonin (Table 2B).
In an independent testing set including a second academic medical center and a second community hospital, the mortality model yielded AUC of 0.827 (Figure 1A), with sensitivity of 82.2% and specificity of 66.7% at the top risk quintile (positive predictive value is 92.6%, while negative predictive value is 42.4%). For the vent+mortality model, AUC is 0.823 (Figure 1B); 89.3% and specificity of 56.0% (positive predictive value is 77.9%, while negative predictive value is 75.0%). Both models exhibit substantial lift, with the highest-risk quintile markedly enriched for adverse outcomes (Figure 2A and B) - as specificity results indicate, 66.7% of mortality is observed in the highest-risk quintile, and 56% of vent+mortality outcomes occur in the highest risk quintile.
For illustrative purposes, we also examined risk quintile in Kaplan-Meier survival curves, with curves censored at time of hospital discharge, end of follow-up, or 15 days, whichever came first. Figure 3 illustrates mortality (A), or vent+mortality (B), by predicted risk quintile. Quintiles were significantly associated with predicted outcome by log-rank test (p<0.0001 for each).
Discussion
In this study of 1,898 individuals with positive COVID-19 status hospitalized at academic medical centers and community hospitals in Eastern Massachusetts through April 28, 2020, 25.3% required mechanical ventilation and 12.7% died. Measures of renal function and red cell immaturity, along with age, were associated with greater mortality risk, confirming two non-pulmonary markers associated with COVID-19 severity. Measures of cardiac or other tissue injury (creatine kinase, troponin T) and systemic infection (procalcitonin), along with male sex, were associated with greater likelihood of requiring mechanical ventilation. Notably, even though the latter model overlaps with the former by design in order to avoid the problem of competing risk, inclusion of mechanical ventilation still yields a substantially different model.
In general, discrimination of both models appears promising, identifying a high-risk quintile with reasonable sensitivity and specificity. Survival curves demonstrate the limitations of these models, with deteriorating discrimination beyond day 10. As such, predictions may be most useful during the initial week of hospitalization; a useful next-step study could examine whether re-running models with additional laboratory studies, or incorporating other biomarkers, can improve subsequent prediction.
Our results are consistent with a recently-reported cohort of 8910 COVID-19 patients, in which cardiovascular disease was associated with marked elevation in mortality8, as well as a smaller study associating renal involvement with mortality16. Multiple smaller cohorts have also reported laboratory features associated with morbidity and mortality among hospitalized COVID-19 patients. For example, a retrospective cohort study from Wuhan in 191 hospitalized patients found older age and greater d-dimer value at admission were associated with risk of death6. Among 95 fatal cases of COVID-19, low eosinophil count at admission was also common.7 Ferritin also associated with mortality in a retrospective cohort study of 120 patients from Wuhan5, along with troponin and CRP.
In developing these simple regression-based prediction models, we were mindful of the recent frameworks for10 and criticisms of17 such models - particularly the recognition that poorly validated or calibrated models may cause more harm than good. Initial models are likely optimistic (i.e., overfit to data) and biased (i.e., by nonrepresentative samples), with a lack of transparency17. On the other hand, strategies to allow risk stratification are particularly necessary in an environment of constrained resources. As such, we report these results in the hope they will provide simple base-case models for others to improve upon. Undoubtedly application of artificial intelligence pixie dust will yield further improvement in model fit, but whether the degree of improvement balances a diminution of interpretability will merit careful consideration.
We note multiple limitations that likely diminish model performance. First, the lag in availability of coded clinical data precludes inclusion of clinical features useful for prediction. Reliance on a manually-curated definition of ventilator use likely diminishes model performance. Our modeling methodology (i.e., reliance on logistic regression) does not fully account for censoring; while extensions of Lasso to Cox regression exist, inpatient clinical data typically violates proportional hazards assumptions, prompting our hybrid approach (i.e., model development using regression, and characterization using survival curves.) Survival analyses do suggest that, while this approach may not yield optimal models for lower-risk individuals, incorporation of censoring still results in discriminativeness for the highest-risk quintile. Finally, many laboratory values are highly non-normal; we elected to incorporate standard high/low flags plus continuous measures, rather than adopting specific transformations for each value which would risk overfitting or diminish generalizability but likely extract additional information. Efforts to aggregate laboratory data across international health systems will provide an opportunity to explore such transformations if individual-level data becomes accessible9.
Despite these limitations, our analyses suggest the utility of laboratory values in combination with sociodemographic features in identifying individuals at particularly high risk for more severe hospital course. These admission models provide an opportunity for comparison as more sophisticated models are developed, particularly those incorporating additional physiologic measures or other biomarkers. As hospital resources remain constrained, the ability to target resources to highest-risk individuals is likely to be valuable, and expansion and refinement of risk models may represent a useful approach to optimizing care.
Data Availability
IRB approval does not allow for sharing of data outside of health system
Role of the Funding Source
No funding source contributed to any aspect of study design, data collection, data analysis, or data interpretation. The corresponding author (RHP) had full access to all the data in the study. All authors shared the final responsibility for the decision to submit for publication.
Acknowledgements
The authors would like to thank the entire RPDR team for their support in making up-to-date EHR data available for this research.
Dr. Perlis had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
No funding was received for this study. Dr. Perlis has received consulting fees from Burrage Capital, Genomind, RID Ventures, and Takeda. He holds equity in Outermost Therapeutics and Psy Therapeutics. Mr. Castro reports no conflict of interest.