Validation of Utstein-Based score to predict Return Of Spontaneous Circulation (UB-ROSC) in patients with out-of-hospital cardiac arrest

Participants

All consecutive patients who suffered an OHCA of any etiology between 1st of January 2019 and 31st of December of 2021 in Lombardia, northern Italy, and in Switzerland were included in the study. Patients declared dead before ambulance arrival, with a “do not resuscitate” order, with incomplete or unknown data were excluded from further analysis.

Study design and setting

This is a prospectively designed study to validate a multiparametric operative score (UB-ROSC score) capable to predict the probability of survival to hospital admission (sustained ROSC) by using consecutive OHCA cases already available in 2 large national registries. UB-ROSC score development was described elsewhere (7). Lombardia region and Switzerland both have a prospectively designed registry of cardiac arrest. Both registries follow the Utstein recommendations for data collection (12,13). The two registries are periodically reviewed for quality assessment by an internal commission, and were approved by the local ethical committee. The Lombardia CARe enrolls all the OHCA cases occurring in the Province of Pavia since January 1, 2015 and in the provinces of Pavia, Lodi, Cremona, and Mantua since January 1, 2019. All the data are collected following Utstein 2014 recommendations (11). The registry was approved by the Ethical Committee of the Fondazione IRCSS Policlinico San Matteo (proc. 20140028219) and by all others who were territorially involved. An informed consent form was signed by all the patients discharged alive. SWISSRECA is a national cardiac arrest registry set up by the Interassociation for Rescue Services (IVR-IAS) at the end of 2018, which collects OHCAs of every etiology occurring in the whole of the Switzerland. SWISSRECA is approved by the national Ethical committee (Swissethics-ID-2016-01844) and IVR-IAS is responsible for its maintenance.

EMS and resuscitation network in Lombardia Region, Northern Italy

The total area covered by the Lombardia CARe registry was 7,863 km² in 2019 and increased to 15,125 km² in 2021. Details of registry development and corresponding changes over time in provinces included in the registry are reported in table 1 of supplemental materials. Each province has several rural regions and a few urban areas for a total population covered of 1,547,333 inhabitants in 2019, increased to 4,274,296 inhabitants in 2021.

The EMS dispatch center coordinates ambulances staffed with basic life support and defibrillation (BLS-D)-trained personnel, and advanced life support (ALS)-trained staffed vehicles (a physician and a specialized nurse or a specialized nurse only). The specialized nurse, if alone in the ALS-staffed vehicle, applies the same ALS protocol, using supraglottic devices instead of tracheal intubation. Five helicopters with a physician and a specialized nurse on board also serve the entire region of Lombardy and another three can intervene from other neighboring regions. In the case of suspected OHCA, the EMS dispatcher activates one to three emergency vehicles (which may include a helicopter) with at least one physician and assists the calling bystander during chest compressions (telephone CPR). The decisions about the attempt and the duration of resuscitation are left to the physician whilst BLS-D-trained personnel are instructed to start resuscitation unless clear signs of death are present (rigor mortis, hypostasis, and injuries not compatible with life).

EMS and resuscitation network in Switzerland

In Switzerland (8.57-million inhabitants over a territory of 41,285 Km², divided in 26 Cantons), OHCAs are managed by local EMS with a two-tiered response system, coordinated via a regional (cantonal) dispatch center. The first tier consists of paramedics who can provide advanced life support, the second tier is made up of teams (ambulance or helicopter) with an emergency physician, alerted if required. Paramedics are instructed to initiate resuscitation unless clear signs of death are present or in case of a Do Not Attempt Resuscitation order. The decision to stop resuscitation and the death declaration are based on the physician’s clinical judgment. The Swiss territory encompasses several rural areas (mountains and valleys) and few relatively small urban areas. In all cantons, there is a network of first responders (FR), made up of off-duty EMS personnel, fire-fighters, police and laypeople trained in CPR, who are alerted via a mobile application and can provide basic life support and use an automatic external defibrillator (AED).

UB-ROSC score variables and outcome of the model

UB-ROSC score model development was described before (7). Briefly, variables that may be determinants of survival at hospital admission were identified among Utstein variables immediately available at EMS arrival on the scene. Age, sex, aetiology, location, witnessed OHCA, bystander CPR, time of EMS arrival and shockable rhythm were included in the model (7). The coefficients estimated from the model, multiplied by 10 and rounded to the closest integer were used to compute a prediction score. All variables included in the model and corresponding scores for UB-ROSC calculation were reported in table 1. Survival to hospital admission was defined as patient with ROSC sustained until arrival at the emergency department and transfer of care to medical staff at the receiving hospital. This definition corresponds to the Utstein recommendations’ core outcome of “Survived event” (13). Probability of survival at hospital admission corresponding to different values of UB-ROSC score were reported in table 2 of Supplemental Material.

Statistical analysis

All analyses were performed using the Stata software (release 17, College Station, TX, USA). A 2-sided p<0.05 was considered statistically significant. Continuous data are reported as mean and standard deviation or median and quartiles if skewed. Categorical data are reported as counts and percent. Data were compared between groups of patients (by national territory and by ROSC groups) with the Mann Whitney U test and the Fisher exact test, respectively. The UB-ROSC score was computed for each patient according to the original paper (7) and categorized in the 3 subgroups of low, medium and high likelihood of ROSC according to the published cut-offs of UB-ROSC. (≤-19; −18 to 12; ≥13). The latter variable was included in a logistic model for ROSC as the independent variable; Huber-White robust standard errors were computed to account for intra-region correlation. We derived the odds ratios (OR) and 95% confidence intervals (95%CI). To assess the performance of the UB-ROSC score in this new cohort, we assessed bot discrimination and calibration. For discrimination we computed the cross-validated model area under the Receiver Operating Characteristic-ROC curve and 95%CI. Sensitivity and specificity of the model were also computed. For this purpose, patients were classified as ROSC if the predicted probability was equal to or above 0.5. To assess calibration, we plotted the observed and predicted probabilities of ROSC as a calibration belt. Model goodness of fit was assessed with the Pearson test and was always satisfied.

Observed ROSC and UB-ROSC validation

A ROSC was obtained in 3,151 patients (21%). The overall observed ROSC rate was higher in SWISSRECA than in Lombardia CARe (31% versus 15%, p<0.001, table 2). UB-ROSC model had a similar discrimination power with an area under the ROC curve (AUC) of 0.68 in SWISSRECA and of 0.73 in Lombardia CARe (figure 1). Model validation and calibration in the entire population is reported in figure 2. Overall, UB-ROSC model showed a good discrimination (AUC 0.72, 95% CI 0.71-0.73, figure 2, panel A). Model had 10% sensitivity and 99% specificity to predict ROSC, with a positive predictive value of 85% and a negative predictive value of 80%. Events correctly classified by the model were 80%. The shape of risk predicted by the model was almost linear and the model was very well calibrated, with a 10-fold cross validation AUC of ROC of 0.71, 95%CI 0.70-0-72, P 1.0. Curve of the shape of risk and internal calibration test is presented in figure 2, panel B and C. The probability to obtain ROSC, according to the 3 subgroups and corresponding ORs are reported in table 3 and Figure 3. For UB-ROSC score values of less than or equal to −19, only in 9% of patients a ROSC was achieved, increasing to 35% for UB-ROSC score between −-18 and 12 (OR 5.3, 95% CI 2.9-9.4, P<0.001). In 85% of patients with UB-ROSC values of more than or equal to 13 a ROSC was obtained (OR 52.1, 95%CI 15.6-173.0, P <0.001). Characteristics of OHCA presentation according to UB-ROSC score subgroups are reported in table 3. Patients with the lowest UB-ROSC values (≦-19) were significantly older (median age 80 years old, IQR 68-87), 52% unwitnessed, 83% occurred in private location and 99% of them had a no-shockable rhythm. In contrast, patients in the high likelihood subgroup of UB-ROSC (UB-ROSC score ≧ 13) were younger (median age 62 years old, IQR 53-71), 58% occurred in public location, and 99% of them were witnessed and with a first shockable rhythm, respectively.

• Download figure
• Open in new tab

Figure 1.

Receiver Operating Characteristic (ROC) curve of UB-ROSC score in Lombardia CARE (panel A) and in SWISSRECA population (panel B).

• Download figure
• Open in new tab

Figure 2.

UB-ROSC model discrimination, shape of risk and calibration. Panel A: Receiver Operating Characteristic (ROC) curve of UB-ROSC score; Panel B: Shape of risk of the model. Panel C: Calibration curve of the model. The bisecting line corresponds to perfect calibration of the model (perfect agreement between observed ROSC and predicted ROSC. The line is entirely included in the shaded area corresponding to the 80% and 95% confidence intervals for the observed-predicted relationship, denoting that the model is well calibrated (there is neither over nor underestimation of mortality).

• Download figure
• Open in new tab

Figure 3.

Cut-offs of UB-ROSC score and corresponding probability of ROSC.

Limitations

This study has some limitations. First, the validation cohort is an European population and the two regions shared similarities in geographical context and in EMS organization. Moreover, the two population of the original validation cohort of UB-ROSC score are included in this validation cohort. Further studies are needed to assess UB-ROSC application in different geographical areas and with substantial differences in EMS organization (i.e. Asian countries). Last, UB-ROSC score is calculated using Utstein variables and categories. Potential application of the score in non-Utstein registries requires adjustment of variables and may determine unpredictable differences in the performance of the model that needs further assessment.