Abstract
Background Uneven vaccination and less resilient health care systems mean hospitals in LMICs are at risk of being overwhelmed during periods of increased COVID-19 infection. Risk-scores proposed for rapid triage of need for admission from the emergency department (ED) have been developed in higher-income settings during initial waves of the pandemic.
Methods Routinely collected data for public hospitals in the Western Cape, South Africa from the 27th August 2020 to 11th March 2022 were used to derive a cohort of 446,084 ED patients with suspected COVID-19. The primary outcome was death or ICU admission at 30 days. The cohort was divided into derivation and Omicron variant validation sets. We developed the LMIC-PRIEST score based on the coefficients from multivariable analysis in the derivation cohort and existing triage practices. We externally validated accuracy in the Omicron period and a UK cohort.
Results We analysed 305,564, derivation 140,520 Omicron and 12,610 UK validation cases. Over 100 events per predictor parameter were modelled. Multivariable analyses identified eight predictor variables retained across models. We used these findings and clinical judgement to develop a score based on South African Triage Early Warning Scores and also included age, sex, oxygen saturation, inspired oxygen, diabetes and heart disease. The LMIC-PRIEST score achieved C-statistics: 0.82 (95% CI: 0.82 to 0.83) development cohort; 0.79 (95% CI: 0.78 to 0.80) Omicron cohort; and 0.79 (95% CI: 0.79 to 0.80) UK cohort. Differences in prevalence of outcomes led to imperfect calibration in external validation. However, use of the score at thresholds of three or less would allow identification of very low-risk patients (NPV ≥0.99) who could be rapidly discharged using information collected at initial assessment.
Conclusion The LMIC-PRIEST score shows good discrimination and high sensitivity at lower thresholds and can be used to rapidly identify low-risk patients in LMIC ED settings.
What is already known on this subject
Uneven vaccination in low- and middle-income countries (LMICs) coupled with less resilient health care provision mean that emergency health care systems in LMICs may still be at risk of being overwhelmed during periods of increased COVID-19 infection.
Risk-stratification scores may help rapidly triage need for hospitalisation. However, those proposed for use in the ED for patients with suspected COVID-19 have been developed and validated in high-income settings.
What this study adds
The LMIC-PRIEST score has been robustly developed using a large routine dataset from the Western Cape, South Africa and is directly applicable to existing triage practices in LMICs.
External validation across both income settings and COVID-19 variants showed good discrimination and high sensitivity (at lower thresholds) to a composite outcome indicating need for inpatient admission from the ED
How this study might affect research, practice or policy
Use of the LMIC-PRIEST score at thresholds of three or less would allow identification of very low-risk patients (negative predictive value ≥0.99) across all settings assessed
During periods of increased demand, this could allow the rapid identification and discharge of patients from the ED using information collected at initial assessment.
Background
The severity of illness associated with COVID-19 has been reduced by mass vaccination and emergence of less severe variants. However, emergency health care systems in low- and middle-income countries (LMIC) may still be at risk of being overwhelmed during periods of increased infection, due to uneven vaccination and less resilient health care systems.1,2 Risk-stratification scores including the UK Royal College of Physicians National Early Warning Score, version 2 (NEWS2) and the COVID-specific Pandemic Respiratory Infection Emergency System Triage (PRIEST) score have been proposed to aid clinical decision-making around need for inpatient admission in the Emergency Department (ED) in patients with suspected COVID-19.3–5 Such risk-stratification scores were developed in high-income settings during initial waves of the pandemic.3,4 Validation of these scores in the middle-income setting of the Western Cape, South Africa, demonstrated good discrimination.6 However, the scores did not outperform existing clinical decision-making and used predictors and physiological cut-offs that are not part of routine clinical practice in this setting.
In LMICs, disposition decision-making is based on clinician experience and gestalt.7 Use of risk-stratification score to allow rapid triage of need for hospitalisation can help prevent hospitals being overwhelmed and assist less-experienced clinicians during periods of increased COVID infection. To be applicable and easily useable in LMICs, a risk-stratification score must be based upon existing clinical triage practice. In South Africa patient acuity on arrival to the ED is triaged using the South African Triage Scale (SATS).8 The Western Cape of South Africa presented a unique opportunity to use routinely collected linked electronic health-care data, in a setting with a high degree of COVID case ascertainment compared to similar settings,9 to develop a risk-stratification score applicable to LMICs.
Our study aimed to:
Develop a contextually appropriate clinical severity score for patients with suspected COVID-19 in an Emergency Department setting in the Western Cape.
Externally validate the developed score.
Methods
Study Design
This observational cohort study used routinely collected clinical data from EDs across the Western Cape, from the Hospital Emergency Centre Triage and Information System (HECTIS)10 data repository, to develop a clinical risk-stratification score for ED patients with suspected COVID.
The performance of the risk-stratification score was externally validated in patients who presented during the Omicron wave and in a cohort of patients from the UK Pandemic Respiratory Infection Emergency System Triage (PRIEST) study (collected during the first wave).11
The study was conducted and reported in accordance with Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) and Reporting of studies Conducted using Observational Routinely-collected Data (RECORD) guidelines.12,13
Setting
For the development and Omicron period validation cohorts, data were collected from patients with suspected COVID-19 infection who attended public-sector EDs in the Western Cape Province. This is one of nine provinces in South Africa, and has almost 7 million inhabitants, of whom three quarters use public sector services.14 A convenience sample of seven hospital EDs (based on use of the HECTIS system) was selected, representing predominantly urban, Cape Town metropole district and, a large peri-rural hospital ED. Clinical decision-making was largely based on clinician experience, contextualised to the local status: at times hospitals were overwhelmed and admission thresholds were raised.15,16 No specific prognostic score were applied in the ED beyond routine triage with SATS.
The external validation population was derived from the PRIEST mixed prospective and retrospective cohort study that collected data from 70 EDs across 53 sites in the UK between 26th March and 28th May 2020.3
Data Sources and linkage
In the Western Cape, data on ED clinical presentation are routinely collected by the HECTIS system, including presenting complaint, triage variables and outcome of consultation. Through deterministic matching, based on unique patient hospital numbers (performed by the Western Cape Provincial Health Data Centre (PHDC)),14 linked data were obtained which included COVID test results from the National Health Laboratory Services (NHLS), comorbidities (based on prior health system encounters), data around admissions and movements within the health care system during the index COVID encounter, and death (if within, or reported to, the health care system). For patients with multiple ED attendances, data were extracted for the first ED attendance and outcomes were assessed up to 30 days from index attendance.
Data collection for the UK PRIEST cohort study has been described in previous publications.3,17 An anonymised version of the study data was used to derive the external validation cohort.18
Inclusion Criteria
Our Western Cape study sample consisted of all adults (aged 16 years and over) at the time of first (index) ED attendance between 27th August 2020 and 11th March 2022, where a clinical impression of suspected, or confirmed, COVID-19 infection had been recorded. For those with multiple presentations, analysis was limited to the index presentation. Patients who presented after the emergence of the Omicron variant (November 2021) were included in a validation cohort.19
Patients for whom age or sex were not recorded were excluded from analyses.
Outcome
The primary composite outcome, in the Western Cape population, was intubation or non-invasive ventilation in the ED on index attendance, Intensive Care Unit (ICU) admission or inpatient death up to 30 days from index attendance. This was comparable to the PRIEST study primary outcome of death or organ support (respiratory, cardiovascular, or renal) by record review at 30 days.
The secondary outcomes were: 1) death and 2) ICU admission (organ support in UK cohort), up to 30 days.
Patient characteristics and candidate predictor variables
Physiological parameters and presenting complaints at triage at index ED presentation were extracted from the HECTIS database. Where no comorbidities were recorded, they were assumed not to be present. Implausible physiological variables were set as missing, including systolic blood pressure <50 mm HG, temperature >42 or <25 degrees, heart rate < 10/minute, oxygen saturation < 10% and respiratory rate = 0/minute.
Candidate predictor variables were selected on the basis of a previous systematic review of factors suitable for use in LMICs, previous research and availability at ED triage in the Western Cape.1,3,20 Variables included: age, sex, presenting symptoms (cough or fever), co-morbidities (heart disease, diabetes, HIV, chronic lung disease, hypertension or pregnancy), physiological parameters and supplemental oxygen. Asthma/COPD was excluded from analysis due to an implausible protective relationship identified in preliminary modelling.
Prognostic model development
The model development cohort was randomly split into derivation and internal validation cohorts. Candidate predictors were combined in a multivariable regression with Least Absolute Shrinkage and Selection Operator (LASSO) using ten-sample cross-validation to select models. The LASSO began with a full model of candidate predictors and simultaneously performed predictor selection and penalisation during model development to avoid overfitting. The LASSO was performed twice: first, when the number of predictors were unrestricted, and a second time, with restriction to ten predictors. Estimates of selected model discrimination and calibration were performed in the split internal validation cohort.
Continuous variables were modelled using fractional polynomials to account for non-linear forms and using categories based on TEWS (Supplementary Material 3).8 As TEWS categories are used as part of existing triage, unless alternative modelling methods demonstrated significant increases in accuracy, these categories were planned a priori to be used in the clinical severity score. Three multivariable analyses were completed using different approaches to missing predictor variable data in the derivation cohort for comparison: (1) Complete case; (2) Multiple imputation using chained equations (10 imputations); (3) Deterministic imputation with missing variable assumed to be in the normal range using TEWS categorisation.
Clinical severity score derivation and validation
Clinical members of the research team reviewed the models and selected variables for inclusion in the triage score, based on the prognostic value and consistency of selection across models, the clinical credibility of their association with the primary outcome, and their availability in the South African ED setting. Selected variables were categorised and assigned integer values using TEWS and the PRIEST score, if present in these clinical scoring systems, whilst checking that categorisation reflected the relationship between the variable and adverse outcome in the derived models.
Additionally, selected variables were assigned integer values to each category of predictor variable, based on the coefficient derived from a multivariable logistic regression model using categorised continuous predictors. This generated a composite clinical score in which risk of adverse outcome increased with the total score.
We applied the clinical score to the model development cohort, the Western Cape Omicron period and UK PRIEST external validation cohorts, calculating diagnostic parameters at each threshold of the score, constructing a receiver-operating characteristic (ROC) curve, calculating the area under the ROC curve (C-statistic) and calculating the proportion with an adverse outcome at each level of the score. Calibration plots for the risk-score were estimated in the external validation cohorts. We used deterministic imputation to handle missing data in the validation cohort, assuming missing predictor variable data were within normal physiological categories but excluding cases with fewer than three predictor variables.
All analyses were completed in STATA version 17.21
Sample Size
The sample size was fixed based on a census sample of patients in the Western Cape recorded on the HECTIS during the study period. In the smallest prognostic model development cohort, there were 102, 503 patients with over 100 outcomes per predictor parameter.
Ethics
The study was approved by the University of Cape Town Human Research Ethics Committee (HREC 594/2021), and the Western Cape Health Research Committee (WC_202111_034). All data were de-identified at source before being provided to the research team.
Patient and Public Involvement (PPI)
A community advisory board (CAB) comprising eight community members affected by COVID (infected themselves or immediate family infected/ hospitalised). PPI members were recruited by an experienced community liaison officer with links to key community groups. Members were intentionally sought to be representative of the various population groups and demographics of the population. Through several meetings, the CAB were kept abreast of the study, and given the opportunity to input on the outcomes, particularly the acceptability of the risk-stratification score.
Results
Study populations
Figures 1 and 2 summarise population selection for the study cohorts. Table 1 summarises the characteristics of the 305,564 patients used for model development. Supplementary Materials 1 and 2 present the characteristics of the 140,520 patients in the Omicron, and 20,698 patients in the UK, validation cohorts. In total, 12,610 patients (4.13%, 95% CI:4.06% to 4.2%) experienced the primary outcome in the development cohort. This compared to, 2,787 patients (1.98%, 95% CI: 1.91% to 2.06%) in the Omicron period, and 4,579 patients (22.12%, 95% CI: 21.56% to 22.69%) in the UK, validation cohorts. In total, 74,580 patients used for model development (24.41%, 95% CI: 24.26 to 24.56%) had a diagnosis of COVID confirmed by PCR testing.
Prognostic models
Tables 2, 3 and 4 show the results of multivariable analysis restricted to inclusion of 10 predictor variables using complete case analysis, multiple imputation and deterministic imputation. Supplementary Materials 4, 6 and 8 show the results of the unrestricted multivariable analyses. Supplementary Material 5 and 7 present the corresponding calibration plots from split internal validation. Unrestricted LASSO on multiply imputed data with modelling of continuous variables using multifractional polynomials produced the model with the highest C-statistic (0.87, 95% CI 0.866 to 0.874) and calibration in the large (CITL) of −0.017 (95%CI −0.043 to 0.009). However, restriction of modelling to 10 predictors and categorisation of continuous variables using TEWS only marginally reduced measures of accuracy in internal validation (worst performing restricted models; C-statistic 0.85 (95%CI 0.845 to 0.855) and CITL 0.126 (95% CI: 0.098 to 0.155)).
When restricted, there were eight predictors that were retained in all analyses (age, use of supplemental oxygen, oxygen saturation, diabetes, consciousness level, heart disease, respiratory rate, heart rate). Sex was retained in all but one complete case analysis.
Clinical severity score derivation and validation
Clinical review judged that the eight predictors retained in all models and sex (included in all but one model) were clinically credible and should be included in the clinical severity score. As TEWS categories for physiological parameters are used clinically in the Western Cape and categorisation did not materially reduce measures of model accuracy, these were used in the risk-score. TEWS also routinely includes measurement of systolic blood pressure and temperature, which were retained in unrestricted models and therefore also included. The co-morbidities diabetes and heart disease were assigned scores based on the relative size of their coefficients across models. As age had a similar modelled form and effect size to the original UK PRIEST study, it was assigned categories and scores based on the PRIEST score. The developed score is shown in Table 5.
The LMIC-PRIEST score was applied to the model development cohort, Omicron and UK PRIEST validation cohorts. The estimated ROC curves for the primary outcome of these analyses are presented in Figure 3. Supplementary Material 9 and 10 show the estimated ROC curves when estimating the secondary outcome of 1) death or 2) admission to ICU/organ support. The score achieved better estimated discrimination when predicting death (C-statistic range: 0.79 UK cohort to 0.83 development cohort) compared to organ support/ICU admission (C-statistic range: 0.68 Omicron cohort to 0.74 development cohort). Figure 4 shows the calibration plots for performance of the score in the external validation cohorts. The score overestimated risk in the Omicron cohort as risk increased and systematically underestimated risk in the UK cohort.
Existing clinical decision-making to admit patients to hospital from the ED in the South African setting had a sensitivity of 0.77 (95% CI 0.76 to 0.78) and specificity of 0.88 (95% CI 0.87 to 0.88) for the primary outcome (prevalence primary outcome 3.45%). The positive predictive value (PPV) was 0.18 (95% CI 0.18 to 0.18) and the negative predictive value was (NPV) 0.99 (95% CI 0.99 to 0.99). Clinicians discharged 85.28% of patients on first presentation. Table 6 presents the estimated sensitivity, specificity, negative and positive predictive values for levels of the score for the primary outcome in the three study populations that could be used clinically to inform admission decisions. Supplementary Materials 11, 12 and 13 shows these values for each level of the score and Supplemental Material 14 shows the risk of the primary outcome for at each level in the 3 study cohorts.
Discussion
Summary
The LMIC-PRIEST score has been developed using a large cohort of patients with suspected COVID and a study period that encompasses Beta, Delta and Omicron waves in the Western Cape. Alongside use of an external validation cohort, we were able to assess accuracy in both different income settings and variants. The LMIC-PRIEST score has shown consistent discrimination across different settings, C-statistics: 0.82 (95% CI: 0.82 to 0.83) development cohort, 0.79 (95% CI: 0.78 to 0.80) Omicron cohort and 0.79 (95% CI: 0.79 to 0.80) UK validation cohort. However, differences in prevalence of adverse outcomes resulted in over- and underestimation of risk in the Omicron and UK external validation cohorts (Figure 4).
The LMIC-PRIEST score builds on existing clinical triage practices in South Africa, and in addition to parameters used to calculate SATS, the score includes other routinely collected variables. The score is therefore clinically applicable to the intended setting of use. In existing practice, clinicians admitted 14.7% of patients, and discharged patients had a 1% risk of the primary outcome (NPV 0.99, 95% CI 0.99 to 0.99). Use of the score at score thresholds up to <5 could offer gains in sensitivity, but, in the Western Cape, this would increase the proportion of admitted patients with a very small associated reduced risk of false negative triage. Lower thresholds could be used to rapidly and transparently identify a proportion of very low-risk patients who could be discharged from the ED based on information routinely collected at initial assessment.
Comparison to previous literature
A systematic review found that no risk-stratification scores for patients with suspected COVID had been developed and validated in LMICs.20 The Nutri-CoV score was subsequently developed and validated in Mexico using data from the first wave of the pandemic and has been proposed for use to triage patient acuity in hospital settings.22 In internal validation, the score achieved a C-statistic of 0.797 (95% CI 0.765 to 0.829), sensitivity 0.93 (95% CI 0.88 to 0.98), specificity 0.11 (95% CI 0.02 to 0.21) and NPV 0.6 (95% CI 0.34 to 0.87) when estimating death or ICU admission at a recommended threshold. Although not explicitly requiring inpatient investigations, a key predictor in the Nutri-CoV score is diagnosis of pneumonia, which requires either radiological or clinical diagnosis. Consequently, the Nutri-CoV score may not be suitable for rapid identification of low-risk patients suitable for discharge using information available at triage. Unlike the LMIC-PRIEST score score, the Nutri-COV score has not undergone external validation in different COVID variants or income settings.
The variables consistently selected by LASSO modelling and used to inform the LMIC-PRIEST score are consistent with other studies.23 Age, inspired oxygen and oxygen saturations have been found, as in our models, to be highly predictive of adverse outcomes in the ED setting.3,24 Although diabetes and heart disease, amongst other comorbidities, have been found to be prognostic in COVID infection, they have not been found to be as highly predictive of adverse outcomes in the ED setting as in our study.3 However, diabetes has previously been identified as a strong predictor (OR 1.84, 95% CI 1.24 to 2.73) of death in patients with COVID in studies conducted in South Africa.15
Strengths and limitations
We followed robust statistical model development techniques that have been rated as low risk of bias and have an adequate sample size for model development.3,23 Restricted and unrestricted models derived using different methods for handling missing data, alongside clinical judgement, were used for score development. These measures all reduce the risk of over-optimistic prediction and inclusion of variables by chance that are not truly predictive. The use of cohorts of patients with suspected infection means that the score has been developed and validated in the population of intended use, the population whom ED staff must clinically triage.1 Our available datasets also comprise multiple COVID waves and different income settings allowing external validation and assessment of generalisability.19
The cohort used for model development was collected from selected government hospitals using the recently implemented HECTIS system. Use of the HECTIS system, electronic records and linking of data from various sources is in its infancy in this context and is dependent on many data entry points across facilities and institutions. Data collection is not primarily intended for research purposes and may be subject error and missingness. However, the HECTIS system is used clinically to collect and record the physiological and other variables used to calculate SATS in the ED as part of clinical practice.25 Deaths are recorded if they occurred in, or were notified to, a health facility. Our cohort is formed by patients who were tested and diagnosed with COVID and clinical staff performing the initial assessment in the ED had a strong clinical impression of likely infection. This was partly determined by prevalence of infection and clinical guidance, which varied during the study period. Although use of PRIEST study data allowed external validation of our developed score, the intended setting of use is in other LMICs during current waves of the pandemic.
Implications
During periods of increased COVID prevalence, patients in South Africa with suspected infection were found to bypass primary care and self-present to hospitals.15 This was associated with excess attendances for patients who required no specific treatment. This partly explains the lower prevalence of the primary outcome in the Western Cape. In the UK, telephone triage was used effectively to reduce ED attendances of lower-risk patients.26 Disposition decision-making in LMICs is based on clinician experience and gestalt.7 Existing clinical decision-making was found to perform well with only 14.7% of patients admitted as inpatients and a risk of false negative triage of around 1%. Although clinical-decision rules have been found to rarely out-perform clinician gestalt,27 exercising clinical judgement requires time and experience, which may be limited during periods of increased demand.
Despite, imperfect calibration in external validation, use of the LMIC-PRIEST score at thresholds of three or less would allow identification of very low-risk patients (NPV ≥0.99) across different settings using information routinely collected during initial triage. During periods of increased COVID prevalence and corresponding ED attendances, the score could potentially be used by practitioners with basic training to identify very low-risk patients for discharge without full clinical assessment, thereby reducing the risk of hospitals being overwhelmed. A conservative, high sensitivity (0.96) threshold of >2, in the Omicron validation cohort would allow the theoretical discharge of 30% of assessed patients.
The wide variation in the prevalence of the primary outcome between settings resulted in miscalibration in external validation. The LMIC-score will need calibration in settings of intended use. This may simply involve selecting the most appropriate threshold based on the population risk and clinical context. However, the emergence of new variants may require different weightings of predictor variables.
A primary outcome of death and ICU admission/organ support was used to encompass need for hospital admission.3 However, across all development and validation settings, the score predicted death better than ICU admission/organ support (Supplementary 9 and 10). The accuracy of the LMIC-PRIEST score for the composite outcome should not be used to guide treatment decisions beyond need for admission, such as potential benefit from invasive treatments, as differences in the prediction of death and interventions are likely to mean that the estimation of benefit is inaccurate.28
Conclusion
The LMIC-PRIEST score has been developed using robust methods and the score shows generalisable discrimination across a range of COVID variants and income settings. It is specifically designed to be used as part of existing triage practices in South Africa and other LMICs. The score could be used to identify very low-risk patients with suspected COVID infection rapidly and transparently during periods in which health care systems experience increased demand due to a high prevalence of infection. Further external validation may be necessary if the score is used in different settings or novel COVID variants.
Data Availability
The data used for this study are subject to a data sharing agreement with the Western Cape Government Department of Health and Wellness, which prohibits further sharing of patient- level data. Access to these and related data should be requested directly from this organisation and is subject to the necessary ethical and organisational approval processes.
Author Disclosure Statement
No competing financial interests exist.
Funding
CM is a National Institute for Health Research (NIHR) Clinical Lecturer in Emergency Medicine (Grant Number Not Applicable/NA). This work is part of the Grand Challenges ICODA pilot initiative, delivered by Health Data Research UK and funded by the Bill & Melinda Gates Foundation and the Minderoo Foundation. The Provincial Health Data Centre (PHDC), Health Intelligence Directorate, Western Cape Government Health and Wellness acknowledges funding from the United States National Institutes of Health (R01HD080465, U01AI069911), Bill and Melinda Gates Foundation (1164272; 1191327; INV-004657, INV-017293), the Wellcome Trust (203135/Z/16/Z), the United States Agency for International Development (72067418CA00023)
Authors’ contributions
The idea for the study was conceived by SG, LW, PH, PB, GF and CM. Data processing and linkage was supervised and completed DM. The analyses were completed by CM with additional expert advice from SG, and PB. All authors contributed to interpretation of results, read, and approved the final manuscript.
Data sharing
The data used for this study are subject to a data sharing agreement with the Western Cape Government Department of Health and Wellness, which prohibits further sharing of patient-level data. Access to these and related data should be requested directly from this organisation and is subject to the necessary ethical and organisational approval processes.
Acknowledgements
This work uses data provided by patients as part of their care and support and the authors wish to recognise the Western Cape Government Health and Wellness (WCGHW) for their contribution of the data that made this research possible, specifically Nesbert Zinyakatira and the team from the Provincial Health Data Centre, Health Impact Assessment Directorate, Western Cape Government Health; and Dr Moosa Parak and the HECTIS team. We further acknowledge and thank the National Health Laboratory Service of South Africa, for their contribution to the study through the provision of the digitised laboratory results accessed through the Provincial Health Data Centre. Dr Laura Sutton (University of Sheffield) conducted the statistical analysis for the UK PRIEST study and provided a template for the statistical analyses.
Footnotes
Fax: +44 (0)114 222 0749 Tel: (+44) (0)114 222 4345, Email: C.Marincowitz{at}Sheffield.ac.uk