Identifying High and Low Performing Emergency General Surgery Hospitals Using Direct Standardization

Introduction

Emergency general surgery (EGS) hospitalizations comprise a significant proportion of total admissions in the United States with both operative and non-operative treatment options available to EGS patients.¹ Given variation across both patient-level and hospital-level risk means, certain patients may do better being directed to particular hospitals.² As such, measuring quality amongst EGS hospitals can provide insights into where patients would have the best outcome.

One method to capture variability across EGS hospitals performance includes ranking hospitals according to both mortality and non-mortality measures.³ Given the heterogeneity in both patient and hospital characteristics of EGS-treated populations, such an analysis provides meaningful insight into consistencies or deviations in both treatment pathways and resource utilization.

We sought to use a novel patient risk-adjustment method to rank EGS hospitals according to mortality, readmission, length of stay, and adverse events. We looked to identify both high and low performers amongst hospitals to understand unique attributes of those hospitals to serve as signals for hospital selection. We then sought to determine if patient outcomes were correlated among operative and non-operative patient populations.

Methods

Patients with a principal diagnosis of an EGS conditions who were Medicare (fee-for-service) beneficiaries 65.5 years of age and older, and admitted through the emergency department to a non-federal, acute care hospital between July 1, 2015 and June 30, 2018 were included. EGS conditions were identified using International Classification of Disease, Ninth or Tenth Revision, Clinical Modification codes.⁴ Patient comorbidities were classified and analyzed according to previously published methods.^4–6 The main outcome was a composite of 30-day adverse events defined by death, prolonged length of stay or readmission within 30-days of hospital discharge.⁷ Death, prolonged length of stay and readmission within 30-days were examined as secondary outcomes. Hospital characteristics were extracted from the American Hospital Association and mean star rating from the Medicare Hospital Compare tool.^8,9

To control for differences in patient populations across hospitals that could account for estimated differences in hospital outcomes, we performed direct standardization using balancing weights on 71 indicators. Specifically, we standardize each hospitals’ patient population to the overall patient population.¹⁰ Balancing weights are a computational generalization of inverse propensity score weight that we use to create hospitals with patient populations with similar covariate distributions. Balancing weights assign a scalar weight to each patient such that we can take hospital level weighted averages that account for differences across hospital populations. We then generated hospital level rankings using an Empirical Bayes estimator that corrects for variation in the size of each hospital’s patient population. Prior research has shown that a balancing weights approach to direct standardization reduces bias due to observed differences between the comparison in patient populations which increasing precision by reducing the loss of sample size.¹⁰ We also identified the set of hospitals with a high probability of having a high or low rank relative to the overall distribution. Standard and rank correlations of hospitals were examined between performance for operative and nonoperative treatments. All statistical analyses were conducted in R version 4.1.1.

Results

There were 536,284 total patients with a mean age of 74.4 ± 12.2 years and frailty index of 0.16 ± 0.06. Most patients were female (55%), white (84%), and had 3+ comorbid conditions (61%). Ten percent of the population was Black and 2% Hispanic.

Among the 1866 hospitals identified, 3 best performing hospitals were identified for all patients. The mean hospital compare star rating was 3.7 ± 0.9. There were 11 identified worst performing hospitals with a mean hospital star rating of 2.4 ± 1.1. For operative treatment, there were no distinctly best performing hospitals and 3 worst performing hospitals identified. Of these 3 hospitals, the average hospital stars rating was 2.9 ± 0.8. For non-operative treatment, there were 6 identified best performing and 2 worst performing hospitals. The best performing non-operative hospitals had a mean star rating of 2.8 ± 0.5. Amongst the hospitals that were worst performing at non-operative treatment (n=2), there was an average star rating of 3.0 ± 1.4. There were no apparent differences in hospital characteristics across hospitals by performance. (Table 1)

Unadjusted results are presented in Table 2. A caterpillar plot (Figure 1) demonstrates individual hospital performance as measured by risk standardized adverse event rates relative to the remainder of the hospitals. There are only a select few hospitals that either performed statistically better (at the right tail) or worse than (at the left tail) the remainder of the hospital dataset. After adjustment, there was a significantly higher rate of adverse events at the worst hospitals (57% ± 4%)) when compared with the best hospitals (25% ± 5%). There were no best performing hospitals for operative treatment. The worst hospitals for operative treatment displayed adjusted rates of adverse events that were similar to those seen for the hospitals performing worst in non-operative treatment (57% ± 10%). For non-operative treatment, there was a higher rate of adverse events at the worst hospitals (60% ± 2%) when compared with the best hospitals (29% ± 6%). Secondary outcomes demonstrated similar findings. (Table 2)

• Download figure
• Open in new tab

Figure 1: Caterpillar Plot Indicating Individual Hospital Risk-Standardized Adverse Event Rates.

The standard correlation between operative and non-operative outcomes at the hospital level was 0.10 for overall mortality, 0.27 for prolonged length of stay, 0.17 for readmission and 0.20 for adverse event. The Spearman rank correlation between the two subsets of patients was 0.10 for overall mortality, 0.32 for prolonged length of stay, 0.18 for readmission and 0.21 for adverse events (all p<0.001).

Discussion

Using direct standardization to rank hospitals on EGS performance in this nationwide study of older adults, we demonstrate a more than two-fold difference in adverse event rates between the best and worst performing hospitals. Hospital performance differed similarly on 30-day death, prolonged length of stay and readmissions. There was also significant variation in hospital performance in the non-operative treatment subgroup. The worst hospitals for operative treatment achieved outlier status while there were no hospitals that distinguished themselves as the best in operative treatment.

Ingraham et al. found that EGS performance across conditions was only weakly correlated within hospitals. ¹¹ Further, Zogg et al. demonstrated that within EGS conditions, hospital performance was consistent across multiple outcome types.³ We build on these findings to demonstrate that hospital performance on several metrics in operative and nonoperative treatment is only weakly correlated, and that there are a limited number of hospitals within the United States that are distinguished in EGS care of older adults.

Limitations

This study has several limitations. First, variation in coding practices across hospitals may impact the capture of adverse events. However, for this study, we captured death, prolonged length of stay, and readmissions which are all known to be valid and reliable in Medicare claims. Second, direct standardization can only adjust for observable covariates which may explain the paucity of outlier hospitals. That is, if there are key patient covariates that are unobserved which contribute to hospital quality, we cannot account for these covariates under direct standardization. However, the number of best and worst performing hospitals in our study is similar to other published reports.^11,12

Conclusion

We have confirmed variation in EGS performance among hospitals treating older adults with the best hospitals significantly outperforming the worst hospitals on all outcome measures examined. These data suggest that EGS outcomes may be optimized with targeted hospital selection. Weak correlation between hospital performance on operative and non-operative care suggests that finding the right hospital for each patient is more complicated than defining a Center of Excellence designation. Further, over 14 years, there has only been a 13% increase in adoption of acute care surgery models that could improve EGS outcomes within hospitals.¹³ As such, alternative approaches to supplement quality and safety initiatives are needed until best practices become universal.