Prognostic accuracy of emergency department triage tools for adults with suspected COVID-19: The PRIEST observational cohort study

Design

We undertook an observational study to collect standardised predictor variables recorded in the ED, which we then used to evaluate triage tools for predicting adverse outcome up to 30 days after initial hospital presentation. The study did not involve any change to patient care. Hospital admission and discharge decisions were made according to usual practice, informed by local and national guidance.

Setting and population

We identified consecutive patients presenting to the ED of participating hospitals with suspected COVID-19 infection. Patients were eligible if they met the clinical diagnostic criteria [10] of fever (≥ 37.8°C) and acute onset of persistent cough (with or without sputum), hoarseness, nasal discharge or congestion, shortness of breath, sore throat, wheezing, sneezing. This was determined on the basis of the assessing clinician recording that the patient had suspected COVID-19 or completing a standardised assessment form designed for suspected pandemic respiratory infection [11].

Interventions

We planned to evaluate triage tools recommended for use in the COVID-19 pandemic or the 2009 H1N1 influenza pandemic, as outlined in the introduction: the WHO algorithm, NEWS2, CURB-65, CRB-65, PMEWS and the swine flu adult hospital pathway (SFAHP). The triage tools are described in Appendix 1. NEWS2 can be used as a score, with thresholds between zero and 20 on the total score, or a rule, with a single threshold of a total score greater than four or a score of three on any parameter. We therefore evaluated the performance of NEWS2 as both a score and a rule. The SFAHP has a criterion (G) that is positive if there is any clinical concern. This is difficult to judge objectively or identify from clinical records, so we evaluated the pathway in two ways: (1) A 6-point rule that did not include parameter G; (2) A 7-point rule in which parameter G was positive if the NEWS2 rule was positive. NEWS2 is widely used in the UK health service to identify clinical concern.

Data collection

Data collection was both prospective and retrospective. We provided participating EDs with a standardised data collection form that included the predictor variables used in the triage tools [11]. Participating sites could adapt the form to their local circumstances, including integrating it into electronic or paper clinical records to facilitate prospective data collection, or using it as a template for research staff to retrospectively extract data from clinical records. We did not seek consent to collect data but information about the study was provided in the ED and patients could withdraw their data at their request. Patients with multiple presentations to hospital were only included once, using data from the first presentation identified by research staff.

Outcome measurement

Research staff at participating hospitals reviewed patient records at 30 days after initial attendance to identify any adverse outcomes. Patients who died or required respiratory, cardiovascular or renal support were classified as having an adverse outcome. Patients who survived to 30 days without requiring respiratory, cardiovascular or renal support were classified as having no adverse outcome. Respiratory support was defined as any intervention to protect the patient’s airway or assist their ventilation, including non-invasive ventilation or acute administration of continuous positive airway pressure. It did not include supplemental oxygen alone or nebulised bronchodilators. Cardiovascular support was defined as any intervention to maintain organ perfusion, such as inotropic drugs, or invasively monitor cardiovascular, status, such as central venous pressure or pulmonary artery pressure monitoring, or arterial blood pressure monitoring. It did not include peripheral intravenous cannulation or fluid administration. Renal support was defined as any intervention to assist renal function, such as haemofiltration, haemodialysis or peritoneal dialysis. It did not include intravenous fluid administration.

Analysis

The primary outcome was death, or respiratory, cardiovascular or renal support, as defined above. We also planned secondary analyses using the following outcomes: (1) Respiratory, cardiovascular, or renal support, to predict need for life-saving treatment; (2) Death without respiratory, cardiovascular, or renal support, to predict poor prognosis. If triage tools are used to determine treatment decisions, such as referral to critical care, then it is helpful to know how well they predict need for treatment rather than a potentially irremediable poor prognosis.

We retrospectively applied each triage tools to the data, excluding pregnant women from analysis of NEWS2. Appendix 1 provides details of scoring and handling missing data for the triage tools. For each tool we plotted the receiver-operating characteristic (ROC) curve and calculated the area under the ROC curve (c-statistic) for discriminating between cases with and without adverse outcome. We calculated sensitivity, specificity, positive predictive value and negative predictive value at the following pre-specified decision making thresholds, based on recommended or usual use: 0-1 versus 2-5 for CURB-65; 0-2 versus 3+ for PMEWS; 0-4 versus 5-20 for the NEWS2 score. The WHO algorithm and Swine Flu Hospital Pathway are positive if any criterion is positive. We used STATA (version 16) for analyses [12].

The sample size was dependent on the size and severity of the pandemic, but based on a previous study in the 2009 H1N1 influenza pandemic we estimated we would need to collect data from 20,000 patients across 40-50 hospitals to identify 200 with an adverse outcome. In the event, the adverse outcome rate in adults was much higher in the COVID-19 pandemic, giving us adequate power to undertake primary and secondary analyses.

Patient and public involvement

The Sheffield Emergency Care Forum (SECF) is a public representative group interested in emergency care research. [13] Members of SECF advised on the development of the PRIEST study and two members joined the Study Steering Committee. Patients were not involved in the recruitment to and conduct of the study. We are unable to disseminate the findings to study participants directly.

Summary of main findings

ED clinicians usually use triage tools to support decisions, such as admission to hospital, where sensitivity needs to be optimised at the expense of specificity to avoid missed opportunities to predict and prevent adverse outcome. Our analysis suggests that the WHO algorithm or PMEWS greater than two provide good sensitivity at the expense of specificity, and could be used to support decision-making where sensitivity needs to be optimised. NEWS2 needs to use a lower threshold (any score above one) than currently recommended to achieve a comparable balance of sensitivity and specificity.

The triage tools predicted death without organ support better than they predicted receipt of organ support. Only NEWS2 predicted receipt of organ support with good accuracy. This reflects NEWS2 using only physiological measures, while other triage tools include age, performance status or comorbidities that are more likely to predict death without organ support.

Previous research

Studies undertaken during the H1N1 influenza 2009 pandemic suggested that existing triage tools have suboptimal accuracy for predicting adverse outcome in acute respiratory infections, with c-statistics below 0.8 [14,-16]. Recent studies have evaluated NEWS2, CURB-65 and CRB-65 in adult inpatients with confirmed COVID-19. Fan et al (N=654) [17] reported c-statistics of 0.81, 0.85 and 0.80 respectively for NEWS2, CURB-65 and CRB-65 as predictors of in-hospital death. The conventional thresholds for positivity of scores above four, one and zero offered suboptimal sensitivity (0.79, 0.63 and 0.83), with corresponding specificities of 0.69, 0.91 and 0.69. Bradley et al (N=830) [18] reported c-statistics of 0.67 for NEWS2 and 0.74 for CURB-65 as predictors of 30-day mortality, with sensitivities and specificities at conventional thresholds of 0.83 and 0.37 for NEWS2, and 0.80 and 0.59 for CURB-65. Ma et al (N=305) [19] reported c-statistics of 0.79 for NEWS2 and 0.85 for CURB-65 for predicting death. Satici et al (N=681) [20] reported a c-statistic of 0.79 for predicting 30-day mortality with CURB-65, with sensitivity 0.73 and specificity 0.85 at the conventional threshold. Nguyen et al [21] reported that 36/171 (21%) patients with CURB-65 scores of zero or one died or received intensive care admission. Gidari et al (N=68) [22] evaluated NEWS2 as a predictor of intensive care admission and Myrstad et al (N=66) [23] evaluated NEWS2 and CRB65 as predictors of death or intensive care admission, but the small sizes produced imprecise estimates of prognostic parameters.

These studies concur with our findings that the conventional thresholds for NEWS2 and CURB-65 offer inadequate sensitivity to support discharge decisions after ED assessment. The larger studies used 30-day or in-hospital mortality as their outcome. Our analysis suggests that this may overestimate prognostic accuracy if the tools are used to predict need for life-saving treatment rather than simply predicting mortality.

Strengths and limitations

We collected data from a clinically relevant population of patients presenting with suspected COVID19 across a large and varied range of EDs. The large sample size and high rate of adverse outcome allowed us to estimate parameters with a high degree of precision in primary and secondary analyses. The main limitation is that the triage tools applied to data collected from clinical record review or a standardised data collection form, rather than being applied directly to the patient by the assessing clinician. This may have led to under-estimation of the performance of the triage tool, especially when relevant data were missing. Table 1 shows that data were relatively complete for age, physiological variables and performance status, but the recording of other parameters (respiratory distress, respiratory exhaustion, dehydration) was limited by inability to determine whether the feature was not present or not recorded. This is most salient for the Swine Flu Hospital Pathway and may have led to underestimation of the sensitivity of this triage tool. Another potential limitation is that we may have missed adverse outcomes if patients attended a different hospital after initial hospital discharge. This is arguably less likely in the context of a pandemic, in which movements between regions were curtailed, but cannot be discounted. Finally, although some triage tools can be used in the prehospital or community setting, we recommend caution in extrapolating our findings to other settings, where there may be a lower prevalence of adverse outcome.

Implications for practice

Our findings suggest that the WHO algorithm or PMEWS greater than two could be used to support hospital admission decisions, providing good sensitivity at the expense of specificity. NEWS2 would need to use a threshold greater than one to achieve a similar balance of sensitivity and specificity. If a triage tool is used to select patients for higher levels of treatment, rather than simply predict risk of adverse outcome, then NEWS2 offers better discrimination than other triage tools. Use of triage tools for this purpose may also require a different balance of sensitivity and specificity, with a higher threshold being used to ensure higher levels of care are reserved for those most likely to benefit.

In general, however, the accuracy of the triage tools evaluated was far from optimal, especially for predicting receipt of organ support. This is arguably unsurprising since they were developed for a variety of purposes and none were derived using data from patients presenting to the ED with suspected COVID-19. Research to derive and validate triage tools specific for COVID-19 is therefore an urgent priority.

Ethical approval

The North West - Haydock Research Ethics Committee gave a favourable opinion on the PAINTED study on 25 June 2012 (reference 12/NW/0303) and on the updated PRIEST study on 23rd March 2020. The Confidentiality Advisory Group of the Health Research Authority granted approval to collect data without patient consent in line with Section 251 of the National Health Service Act 2006.

Competing interests

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: grant funding to their employing institutions from the National Institute for Health Research; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.