Abstract
Objectives To evaluate the National Early Warning Score (NEWS2), currently recommended in the UK for risk-stratification of severe COVID-19 outcomes, and subsequently identify and validate a minimal set of common parameters taken at hospital admission that improve the score.
Design Retrospective observational cohort with internal and multi-hospital external validation.
Setting Secondary care.
Interventions Not applicable.
Participants Main outcome measures
Results Training and temporal external validation cohorts comprised 1464 patients admitted to King’s College Hospital NHS Foundation Trust (KCH) with COVID-19 disease from 1st March to 30th April 2020. External validation cohorts included 3869 patients from two UK NHS Trusts (Guys and St Thomas’ Hospitals, GSTT and University Hospitals Southampton, UHS) and two hospitals in Wuhan, China (Wuhan Sixth Hospital and Taikang Tongji Hospital).
The primary outcome was patient status at 14 days after symptom onset categorised as severe disease (transferred to intensive care unit or death). Age, physiological measures, blood biomarkers, sex, ethnicity and comorbidities (hypertension, diabetes, cardiovascular, respiratory and kidney diseases) were included.
Conclusions NEWS2 score on admission was a weak predictor of severe COVID-19 infection (AUC = 0.628). Adding age and common blood tests (CRP, neutrophil count, estimated GFR and albumin) provided substantial improvements to a risk stratification model, particularly in relation to sensitivity, but performance was only moderate (AUC = 0.753). Improvement over NEWS2 remained robust and generalisable in GSTT (AUC = 0.817), UHS (AUC = 0.835) and Wuhan hospitals (AUC = 0.918).
Adding age and a minimal set of blood parameters to NEWS2 improves the detection of patients likely to develop severe COVID-19 outcomes. This finding was replicated across NHS and non-UK hospitals. Adding a few common parameters to a pre-existing acuity score allows rapid and easy implementation of this risk-scoring system.
Key Messages
The National Early Warning Score (NEWS2), currently recommended for severe COVID-19 disease in the UK shows overall poor discrimination for severe outcomes (transfer to ICU or death). It can be improved by the addition of a small number of blood and physiological parameters routinely measured at hospital admission.
The addition of age and a minimal set of common blood tests (C-reactive protein, neutrophil count, estimated GFR and albumin) provided substantial improvements in a risk stratification model.
Although predictive performance varied from hospital to hospital, the improvement over NEWS2 alone was consistent across different patient cohorts.
The proposed addition of a limited number of dichotomised parameters is easily derived from a pre-existing acuity score would be substantially easier to implement in a short-time scale compared to novel high-dimensional risk-scoring systems.
Introduction
As of 7th June 2020, there have been >6.7 million confirmed cases of COVID-19 disease worldwide[1]. While approximately 80% of infected individuals have mild or no symptoms[2], some develop severe COVID-19 disease requiring hospital admission. Within the subset of those requiring hospitalisation, early identification of those who deteriorate and require transfer to an intensive care unit (ICU) for organ support or may die is vital.
Currently available risk scores for deterioration of acutely-ill patients include (1) widely-used generic ward-based risk indices such as the National Early Warning Score (NEWS2)[3] or modified sequential organ failure assessment (mSOFA)[4]; and (2) the pneumonia-specific risk index, CURB-65[5] which usefully captures a combination of physiological observations with limited blood markers and comorbidities.
NEWS2 is a summary score of six physiological parameters or ‘vital signs’ (respiratory rate, oxygen saturation, systolic blood pressure, heart rate, level of consciousness, temperature and supplemental oxygen dependency), used to identify patients at risk of early clinical deterioration in the United Kingdom (UK) NHS hospitals[6,7] and primary care. The physiological parameters assessed in the NEWS2 score – particularly patient temperature, oxygen saturations and the supplemental oxygen dependency – have been associated with COVID-19 outcomes[2]; however, little is known about their predictive value for the severity of COVID-19 disease[8]. Additionally, a number of COVID-19-specific risk indices are being developed[9-11] as well as unvalidated online calculators[12] but generalisability is unknown[11]. A Chinese study has suggested a modified version of NEWS2 with addition of age only[13] but without any data on performance. With near universal usage ofNEWS2 in UK NHS Trusts since March 2019[14], minor adaptation to NEWS2 is relatively easy to implement.
As the SARS-Cov2 pandemic has progressed, evidence has emerged regarding potentially useful blood biomarkers[1,15–18]. Although most of these early reports contain data from small numbers of patients, several markers have been found to be associated with severity. These include neutrophilia and lymphopenia, particularly in older adults[10,17,19,20], neutrophil-to-lymphocyte ratio[21], C-reactive Protein (CRP) and lymphocyte-to-CRP ratio[21], markers of liver and cardiac injury such as alanine aminotransferase (ALT), aspartate aminotransferase (AST) and cardiac troponin[22] and elevated D-dimers, ferritin and fibrinogen[2,5,7].
Our aim was to evaluate the NEWS2 score and then identify a minimal combination of clinical and blood biomarkers routinely measured in hospitals to supplement the acuity score to allow medium-term stratification for risk of a severe disease outcome at 14 days from symptom onset. Our specific objectives were:
To explore independent associations of routinely measured physiological and blood parameters (including NEWS2 parameters) at or near hospital admission with disease severity (ICU admission or death at 14 days since symptom onset), adjusting for demographics and comorbidities;
To examine which minimal combination of these potential determinants of disease severity (physiological and blood parameters, sociodemographics and comorbidities) are the best predictors of disease severity; and
To compare the predictive value of the resulting model with a model based on the NEWS2 total score alone.
Methods
Study cohorts
The KCH training cohort (n=439) was defined as all adult inpatients testing positive for SARS-Cov2 by reverse transcription polymerase chain reaction (RT-PCR) between 1st March to 31st March 2020 at two acute hospitals (King’s College Hospital and Princess Royal University Hospital) in South East London (UK) of Kings College Hospital NHS Foundation Trust (KCH). All patients included in the study had symptoms consistent with COVID-19 (e.g. cough, fever, dyspnoea, myalgia, delirium, diarrhoea). For validation purposes, the following cohorts were used:
External temporal hold-out at KCH of 1025 cases (1st April to 30th April 2020)
External validation cohort at Guys and St Thomas’ Hospital NHS Foundation Trust (GSTT) of 1417 cases (3rd March 2020 to 21st May 2020)
External validation cohort at University Hospitals Southampton NHS Foundation Trust (UHS) of 564 cases (7th March to 20th May 2020)
External validation cohort at Wuhan Sixth Hospital and Taikang Tongji Hospital totalling 1888 cases (4th February to 30th March 2020)
The KCH component of the project operated under London South East Research Ethics Committee (reference 18/LO/2048) approval granted to the King’s Electronic Records Research Interface (KERRI): specific work on COVID-19 research was reviewed with expert patient input on a virtual committee with Caldicott Guardian oversight. The UHS validation was performed as part of a service evaluation agreed with approval from trust research leads and the Caldicott Guardian. Ethical approval for GSTT was granted by The London Bromley Research Ethics Committee (reference 20/HRA/1871) to the King’s Health Partners Data Analytics and Modelling COVID-19 Group to collect clinically relevant data points from patient’s electronic health records. The Wuhan validation was approved by the Research Ethics Committee of Shanghai Dongfang Hospital and Taikang Tongji Hospital.
Data Processing
King’s College Hospital
Data was extracted from the structured and unstructured components of the electronic health record (EHR) using natural language processing (NLP) tools belonging to the CogStack ecosystem[23], namely MedCAT[24] and MedCATTrainer[25]. The CogStack NLP pipeline captures negation, synonyms, and acronyms for medical SNOMED-CT concepts as well as surrounding linguistic context using deep learning and long short-term memory networks. MedCAT produces unsupervised annotations for all SNOMED-CT concepts (Supplementary Table 1) under parent terms Clinical Finding, Disorder, Organism, and Event with disambiguation, pre-trained on MIMIC-III[26].
Starting from our previous model[27], further supervised training improved detection of annotations and meta-annotations such as experiencer (is the concept annotated experienced by the patient or other), negation (is the concept annotated negated or not) and temporality (is the concept annotated in the past or present) with MedCATTrainer. Meta-annotations for hypothetical, historical and experiencer were merged into “Irrelevant” allowing us to exclude any mentions of a concept that do not directly relate to the patient currently. Performance of the NLP pipeline for comorbidities mentioned in the text was evaluated on 4343 annotations in 146 clinical documents by a clinician (JT). F1 scores, precision, and recall are presented in Supplementary Table 2.
Guy’s and St Thomas NHS Foundation Trust (GSTT)
Electronic health records from all patients admitted to Guy’s and St Thomas NHS Foundation Trust who had a positive COVID-19 test result between the 3rd of March and 21st of May 2020, inclusive, were identified. Data were extracted using structured queries from six complementary platforms and linked using unique patient identifiers. Data processing was performed using Python 3.7. The process and outputs were reviewed by a study clinician.
University Hospitals Southampton (UHS)
Data were extracted from the structured components of the UHS CHARTS EHR system and data warehouse. Data was transformed to the required format for validation purposes using Python 3.7. Diagnosis and comorbidity data of interest were gathered from ICD-10 coded data. No unstructured data extraction was required for validation purposes. The process and outputs were reviewed by an experienced clinician prior to analysis.
Wuhan cohort
Demographic, premorbid conditions, clinical symptoms or signs at presentation, laboratory data, treatment and outcome data were extracted from electronic medical records using a standardised data collection form by a team of experienced respiratory clinicians, with double data checking and involvement of a third reviewer where there was disagreement. Anonymised data was entered into a password-protected computerised database.
Measures
Outcome
For all sites, the primary outcome was patient status at 14 days after the index date, categorised as transfer to ICU/death (WHO-COVID-19 Outcomes Scales 6-8) vs. not ICU/death (Scales 3-5). For KCH, the index date was symptom onset (or hospital admission where symptom onset was missing) and dates of symptom onset, ICU transfer, and death were ascertained and verified manually by a clinician. For GSTT and UHS, the index date was the date of diagnosis (positive SARS-CoV-2 RT-PCR) minus 4 days because symptom onset was not available. For the Wuhan cohort, the index date was the date of admission minus 4 days.
Blood parameters
We focused on biomarkers that were routinely obtained at or shortly after admission and were therefore available for the vast majority of patients. These comprised: albumin (g/L), alanine aminotransferase (ALT; IU/L), creatinine (μmol/L), C-reactive protein (CRP; mg/L), estimated Glomerular Filtration Rate (eGFR; mL/min), Haemoglobin (g/L), lymphocyte count (x 109/L), neutrophil count (x 109/L), and platelet count (PLT; x| 109/L). We also derived the neutrophil-to-lymphocyte ratio (NLR) and the lymphocyte-to-CRP ratio[21]. Troponin-T (ng/L) and Ferritin (ug/L) were included, although these measures were only available for a subset of participants. D-dimers and HbA1c were excluded since they were measured in very few patients at admission and insufficient samples were available for analysis.
Physiological parameters
We included the NEWS2 total score as well as the following parameters measured separately: respiratory rate (breaths per minute), oxygen saturation (%), systolic blood pressure (mmHg), heart rate (beats/min), temperature (°C), and consciousness (Glasgow Coma Scale; GCS). All were measured at or shortly after admission. Diastolic blood pressure was also included.
Demographics and comorbidities
Age, sex, ethnicity and comorbidities were considered. Self-defined ethnicity was categorised as caucasian vs. BAME (Black, Asian and minority ethnic) and patients with ethnicity reported as ‘unknown/mixed/other’ were excluded. Binary variables were derived for comorbidities: hypertension, diabetes, heart disease (heart failure and ischemic heart disease), respiratory disease (asthma and chronic obstructive pulmonary disease, COPD) and chronic kidney disease.
Statistical analyses
All parameters were scaled (mean 0; standard deviation = 1) to facilitate interpretability and logarithmic or square-root transformations were applied to skewed parameters. Outlying high values for some blood parameters were retained after clinical examination to ascertain plausibility. To explore independent associations of physiological and blood parameters with 14-day death/ICU (Objective 1) we used logistic regression with Firth’s bias reduction method[28]. Each parameter was tested independently, adjusted for age and sex (Model 1) and then additionally adjusted for comorbidities (Model 2). P-values were adjusted using the Benjamini-Hochberg procedure to keep the False Discovery Rate (FDR) at 5%[29]. These models were conducted with R 4.0[30] using the logistf[31] package.
To investigate which panel of parameters performed best in predicting the 14-day outcome over and above NEWS2 we used a regularized logistic regression with a LASSO (Least Absolute Shrinkage and Selection Operator) estimator which shrinks parameters according to their variance, reduces overfitting and enables automatic variable selection[32]. The optimal degree of regularization was determined by identifying a tuning parameter X using cross-validation[33]. From an initial model with NEWS2 total score only, sets of features were added in order of (i) age and sex; (ii) blood/physiological parameters; and (iii) comorbidities. To estimate the predictive performance of our model on new cases of the same underlying population, we performed internal nested cross-validation (10 folds/20 repeats for inner loop; 10 folds/100 repeats for outer loop). Overall discrimination was assessed based on the area under the curve (AUC). To impute missing features we used k-Nearest Neighbours imputation (k=5). Scaling and kNN imputation were incorporated within the model development and selection process to avoid data leakage which would otherwise result in optimistic performance measures[34].
The predictive performance of the derived regularized logistic regression model was then evaluated using temporal external validation[35] on a hold-out sample of 1025 patients admitted to KCH after the training sample (Supplementary Figure 1) as well as external validation samples from GSTT (n=1417), UHS (n=564), and Wuhan (n=1888). Validation used logistic regression models trained on the KCH training sample, with code and pre-trained models shared via GitHub1. All procedures were identical to those used during training, including scaling and imputation. Performance was assessed using AUC, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Model calibration was assessed using a calibration plot (model predicted probability vs. true probability). These models were estimated in Python 3.8[36] using Scikit-Learn[37].
For ease-of-use in a clinical setting, two further models were estimated: (1) A model (NEWS2 + CoV) that supplemented NEWS2 with five of the most important features chosen based on internal validation and data completeness; (2) A threshold model, derived by dichotomising each feature from the NEWS2 + CoV model. Thresholds were defined using a decision tree model tuned on the training sample.
To ensure robust results, we explored whether predictive performance could be improved using a more complex approach such as gradient boosted trees as implemented in the XGBoost library[38]. Procedures were identical to those described above, except the imputation step was not required. Additional sensitivity analyses were performed to account for potential demographic variability. Following recent studies suggesting sex differences in COVID outcome[17] we tested interactions between each physiological and blood parameter and sex using likelihood-ratio tests. We also repeated all models with adjustment for ethnicity in the subset of individuals with available data for ethnicity (n=276 in KCH training sample).
Results
The KCH training cohort comprised 439 inpatients testing positive for COVID-19 (from 1st March to 31st March 2020) of whom 155 (35%) were transferred to ICU or died (COVID-19 WHO Score 6-8) within 14 days of symptom onset. Table 1 describes the clinical characteristics of the cohort. Patients experiencing more severe outcomes were significantly older but there was no evidence of differences by sex or ethnicity. There were some differences between groups in the prevalence of comorbidities but these did not reach statistical significance after correction for multiple testing. At admission, patients who had transferred to ICU or died within 14 days had lower levels of Albumin and estimated GFR; and elevated levels of CRP, creatinine, ferritin, neutrophils, and neutrophil/lymphocyte ratio. Median NEWS2 total scores were significantly higher in patients who transferred to ICU or died (3 vs 2; p < 0.001), compared to patients experiencing less severe outcomes. Characteristics of validation cohorts from GSTT, UHS, and Wuhan are described in Supplementary Table 3. There were fewer BAME minorities in UHS (compared to KCH) and a much lower proportion of severe cases in Wuhan hospitals (3% vs. 25-35% in UK sites).
Logistic regression models were used to assess independent associations between each physiological and blood parameter and disease severity (Supplementary Table 4). Transfer to ICU/death was associated with higher NEWS2 total score, CRP, heart rate, neutrophil count, neutrophil/lymphocyte ratio, respiration rate; and lower lymphocyte/CRP ratios, Albumin, estimated GFR, creatinine, and oxygen saturation. These associations remained after adjustment for age, sex and comorbidities. Sensitivity analyses showed no evidence of differences by sex and findings were consistent when additionally adjusting for ethnicity in the subset of individuals with ethnicity data (Supplementary Table 5).
Supplementing NEWS2 with a minimal set of routinely collected blood and physiological parameters to improve NEWS2 to improve prediction of 14-day outcome
We included all predictors in a combined logistic regression model using LASSO regularisation. Internally validated predictive performance is presented in Table 2. The baseline model with only NEWS2 showed poor discrimination with an AUC of 0.628. Adding age, sex, and all blood and physiological parameters increased the AUC by 0.114, to 0.742 (+/-2SD: 0.735, 0.750). Performance was not improved when comorbidities were included in this model. A final model (NEWS2 + CoV) was estimated including NEWS2 and the top five most important features.
This simpler model resulted in a slightly larger AUC of 0.753 (+/-2SD: 0.739, 0.768) which may indicate some overfitting due to the pre-selection of variables from previous analyses. Results were consistent when repeating these models in the subset of patients with information available on ethnicity (Supplementary Table 6).
Figure 1 summarises feature importances from LASSO logistic regression models fitted to the KCH training sample. When adding demographic, blood, and physiological parameters to NEWS2 (NEWS2 + DBP) eight features were retained, in order of effect size: NEWS2 total score, CRP, estimated GFR, neutrophils, albumin, age, Troponin T, and oxygen saturation. Notably, comorbid conditions were not retained when added in subsequent models (NEWS2 + DBPC), suggesting most of the variance was already captured by the top five parameters.
When these models were repeated using a XGBoost[38] the pattern of results was consistent with those from regularized logistic regression (Supplementary Table 7). The internally validated AUC improved from 0.647 (+/-2SD: 0.631, 0.663) for a model with NEWS2 alone, to 0.731 (+/-2SD: 0.706, 0.756) for the NEWS2 + CoV model. Importantly, while the pattern of results was consistent, a more complex machine learning estimator did not improve predictive performance.
Temporal external validation was conducted on a hold-out sample of 1025 KCH patients. This sample was similar to the training sample (Supplementary Table 3) except that the proportion who transferred to ICU or died was lower (35% vs 27%). For the hold-out sample, the AUC for NEWS2 alone was 0.665, and this improved to 0.761 for the simplified MEWS2 + CoV model comprising NEWS2 and the top five features (CRP, neutrophils, estimated GFR, albumin and age) (sensitivity = 0.624; specificity = 0.764) (Table 3). Calibration for these models (Supplementary Figure 2) was acceptable but suggested some overestimation of risk probabilities. Overall, results from the hold-out sample were consistent with those from internal validation.
Multi-site replication
When we replicated our trained models in GSTT (n=1417), UHS (n=564) and Wuhan (n=1888) we found our results to be robust despite changes in cohort characteristics (Table 3). In all sites, the supplemented NEWS2 model (NEWS2 + CoV) showed improved predictive performance over NEWS2 alone, with an average increase in AUC of 0.112. Furthermore, the threshold model showed similar performance across sites (Table 3). Details of the validation cohorts can be found on Supplementary Table 3. The code and pre-trained models used for replications is available at https://github.com/ewancarr/NEWS2-COVID-19 and can be tested in other datasets.
Discussion
Principal findings
To our knowledge, this study is the first to systematically evaluate the UK NEWS2 acuity score in COVID-19 and attempt to improve performance of NEWS2 in multiple NHS Trusts. We found that the NEWS2 score shows overall poor discrimination with high specificity but poor sensitivity for severe outcomes (transfer to ICU or death). However, its value for risk stratification (especially sensitivity) can be improved by adding age and a small number of additional blood parameters (CRP, neutrophils, estimated GFR, and albumin). Several blood measures previously linked with severe outcomes – such as lymphocyte and ALT[13], or transformations of inflammatory markers such as CRP/lymphocyte or neutrophil/lymphocyte ratios - did not provide additional value. Pre-existing comorbidities did not improve risk prediction and were not retained in the final model. This was unexpected but may indicate that the effect of pre-existing health conditions is manifest through some of the included blood or physiological markers.
Comparison with other studies
A systematic review of 10 prediction models for mortality in COVID-19 infection[9] found broad similarities with the features retained in our models, particularly regarding CRP and neutrophil levels. However, existing prediction models suffer several methodological weaknesses including over-fitting, selection bias, and reliance on cross-sectional data without accounting for censoring. Additionally, many existing studies have relied on single centre studies or in ethnically homogenous Chinese cohorts, whereas this study shows validation in multiple organisations and diverse populations indicative of generalisability.
NEWS2 is a summary score derived from six physiological parameters, including oxygen supplementation (yes/no). Lack of evidence for NEWS2 use in COVID-19 especially in primary care has been highlighted[8]. The oxygen saturation component of physiological measurements added value beyond NEWS2 total score and was retained following regularisation (i.e. model NEWS2 + DBP). This suggests some residual association over and above what is captured by the NEWS2 score, and reinforces Royal College of Physicians guidance that the NEWS2 score ceilings with respect to respiratory function[39].
Cardiac disease and myocardial injury have been described in severe COVID-19 cases in China[1,22]. In our model, blood Troponin-T, a marker of myocardial injury, had additional salient signal but was only measured in a subset of our cohort at admission, so it was excluded from our final model. This could be explored further in larger datasets.
Strengths and limitations
Our study provides a risk stratification model for which we obtained generalisable and robust results across national and non-UK sites (total patients=5333) with differing geographical catchment and population characteristics. However, some limitations should be acknowledged. First, there are likely to be other parameters not measured in this study that could substantially improve the risk stratification model (e.g. radiological features or comorbidity load). This could be addressed by future work, but these parameters were not considered in the present study to avoid limiting the real-world implementation of the risk stratification model. Second, we used a 14-day time window from symptom onset to provide a balance between medium-term prognostication and actionable risk stratification at the usual period of deterioration. Longer timeframes may be useful for prognostication but are likely to be harder to generalise due to the greater number of factors affecting outcomes (e.g. institutional, regional or national policies). Since NEWS2 score is optimised for very near-term deterioration at 24 hours[7], a 14-day window was used as a compromise. Thirdly, our models showed better performance in UK secondary care settings with populations with higher rates of severe disease after COVID-19 infection. Further research should be performed to investigate the suitability of our model for primary care settings with higher prevalence mild disease severities[8].
Conclusion
The simple addition of age and a limited number of blood parameters to the existing and widely implemented UK NEWS2 system can contribute to improved risk stratification among COVID-19 patients. This type of model can be easily implemented in clinical practice and predicted risk score probabilities of individual patients are easy to communicate. The additional parameters are routinely collected on patients at hospital admission, and with near universal usage of NEWS2 in NHS Trusts since March 2019[14], a minor adaptation to NEWS2 is substantially easier to implement in a variety of health settings than a bespoke risk score.
Contributor and guarantor information
The corresponding author, Dr Ewan Carr, is guarantor of the manuscript.
JT, AS, RD, EC and RB conceived the study design and developed the study objectives. JT, RD, AF, LR, DB, ZK, TS and AS were the leads to develop CogStack platform. DB, ZK, TS, AS were responsible for the data extraction and preparation. EC, RB, AP, DS contributed to the statistical analyses. All authors contributed to the interpretation of the data. AS, JT, KO, RZ provided clinical input. All authors contributed to interpret the data, draft the article and provided final approval of the manuscript. DMB, ZK, AS, TS, JTHT, LR, KN performed data processing and software development; KOG, RZ, JTHT performed data validation.
At GSTT, WW and WM were responsible for the data extraction and preparation. WW performed the model validation. AD and VC contributed to the interpretation of the data.
At UHS, MS and FB were responsible for the data extraction and preparation. MS, HP and AS contributed to the statistical analysis. All authors contributed to the interpretation of the data. MS and AP provided clinical input. MS and HP performed data/model validation.
For the Wuhan cohort, XZ, XW and JS extracted the data from the EHR system. HW and HZ preprocessed the raw data and conducted the prediction model validations. BG, HW, HZ, TS and JS interpreted the data and results.
Disclaimer: The views expressed are those of the authors and not necessarily those of the MRC, NHS, the NIHR or the Department of Health and Social Care. The funders of the study had no role in the study design, data collection, data analysis, data interpretation, writing of the report or the decision to submit the article for publication.
Copyright/license for publication
The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to i) publish, reproduce, distribute, display and store the Contribution, ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, iv) to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, vi) licence any third party to do any or all of the above.
Data sharing
Code and pre-trained models are available at https://github.com/ewancarr/NEWS2-COVID-19 and openly shared for testing in other COVID datasets.
Source text from patient records used at all sites in the study will not be available due to inability to safely fully anonymise up to the Information Commissioner Office (ICO) standards and would be likely to contain strong identifiers (e.g. names, postcodes) and highly sensitive data (e.g. diagnoses).
A subset of the KCH dataset limited to anonymisable information (e.g. only SNOMED codes and aggregated demographics) is available on request to researchers with suitable training in information governance and human confidentiality protocols subject to approval by the King’s College Hospital Information Governance committee; applications for research access should be sent to kch-tr.cogstackrequests@nhs.net. This dataset cannot be released publicly due to the risk of re-identification of such granular individual level data, as determined by the King’s College Hospital Caldicott Guardian.
The GSTT dataset cannot be released publicly due to the risk of re-identification of such granular individual level data, as determined by the Guy’s and St. Thomas’s Trust Caldicott Guardian.
The UHS dataset cannot be released publicly due to the risk of re-identification of such granular individual level data, as determined by the University Hospital Southampton Caldicott Guardian.
The Wuhan dataset used in the study will not be available due to inability to fully anonymise in line with ethical requirements. Applications for research access should be sent to TS and details will be made available via https://covid.datahelps.life/prediction/.
Public and Patient Involvement statement and Ethics Statement
This project operated under London South East Research Ethics Committee approval (reference 18/LO/2048) granted to the King’s Electronic Records Research Interface (KERRI). Given the current context, specific work on COVID19 research was reviewed with a virtually convened 4-expert patient panel input on a virtual committee with Caldicott Guardian oversight. In addition, the KERRI database used in this study was developed according to KCH use of data for research guidelines (https://www.kch.nhs.uk/research/use-of-data-for-research) and patient engagement groups for this dataset have been conducted in 2018-2019.
Dissemination declaration
Dissemination to study individual participants is not possible to protect anonymity. Dissemination to patient organisations and stakeholder groups is planned.
Transparency declaration
The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Role of the funding source
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests
JTHT received research support and funding from InnovateUK, Bristol-Myers-Squibb, iRhythm Technologies, and holds shares <£5,000 in Glaxo Smithkline and Biogen.
Acknowledgments
DMB is funded by a UKRI Innovation Fellowship as part of Health Data Research UK MR/S00310X/1 (https://www.hdruk.ac.uk).
RB is funded in part by grant MR/R016372/1 for the King’s College London MRC Skills Development Fellowship programme funded by the UK Medical Research Council (MRC, https://mrc.ukri.org) and by grant IS-BRC-1215-20018 for the National Institute for Health Research (NIHR, https://www.nihr.ac.uk) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London.
RJBD is supported by: (1) NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London, London, U.K. (2) Health Data Research UK, which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and Wellcome Trust. (3) The BigData@Heart Consortium, funded by the Innovative Medicines Initiative-2 Joint Undertaking under grant agreement No. 116074. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA; it is chaired by DE Grobbee and SD Anker, partnering with 20 academic and industry partners and ESC. (4) The National Institute for Health Research University College London Hospitals Biomedical Research Centre. (5) National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. (5) The UK Research and Innovation London Medical Imaging & Artificial Intelligence Centre for Value Based Healthcare (6) the National Institute for Health Research (NIHR) Applied Research
Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust.
KO’G is supported by an MRC Clinical Training Fellowship (MR/R017751/1).
WW is supported by a Health Foundation grant.
AD and VC acknowledge support from National Institute for Health Research (NIHR) Applied Research Collaboration (ARC) South London at King’s College Hospital NHS Foundation Trust and the Royal College of Physicians, as well as the support from the NIHR Biomedical Research Centre based at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. VC is additionally supported by Health Data Research UK, which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and Wellcome Trust.
RZ is supported by a King’s Prize Fellowship.
AS is supported by a King’s Medical Research Trust studentship.
JTHT is supported by London AI Medical Imaging Centre for Value-Based Healthcare (AI4VBH) and the National Institute for Health Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust.
FB and PTTH are funded by National Institute for Health Research (NIHR) Biomedical Research Centre, Data Sciences at University Hospital Southampton NHS Foundation Trust and the Clinical Informatics Research Unit, University of Southampton.
JB is funded by the Clinical Informatics Research Unit, University of Southampton, and part funded by the Global Alliance for Chronic Disease (GDAC).
AP is part funded by UHS Digital, University Hospital Southampton, Tremona Road, Southampton.
AJS is supported by a Digital Health Fellowship through Health Education England (Wessex).
HW and HZ are supported by Medical Research Council and Health Data Research UK Grant (MR/S004149/1), Industrial Strategy Challenge Grant (MC_PC_18029) and Wellcome Institutional Translation Partnership Award (PIII054). XW is supported by National Natural Science Foundation of China (grant number 81700006).
AMS is supported by the British Heart Foundation (CH/1999001/11735), the National Institute for Health Research (NIHR) Biomedical Research Centre at Guy’s & St Thomas’ NHS Foundation Trust and King’s College London (IS-BRC-1215-20006), and the Fondation Leducq. AP is partially supported by NIHR NF-SI-0617-10120. This work was supported by the National Institute for Health Research (NIHR) University College London Hospitals (UCLH) Biomedical Research Centre (BRC) Clinical and Research Informatics Unit (CRIU), NIHR Health Informatics Collaborative (HIC), and by awards establishing the Institute of Health Informatics at University College London (UCL). This work was also supported by Health Data Research UK, which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care
Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and the Wellcome Trust.
This paper represents independent research part-funded by the National Institute for Health Research (NIHR) Biomedical Research Centres at South London and Maudsley NHS Foundation Trust, London AI Medical Imaging Centre for Value-Based Healthcare, and Guy’s & St Thomas’ NHS Foundation Trust, both with King’s College London. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We would also like to thank all the clinicians managing the patients, the patient experts of the KERRI committee, Professor Irene Higginson, Professor Alastair Baker, Professor Jules Wendon, Dan Persson and Damian Lewsley for their support.
The authors acknowledge use of the research computing facility at King’s College London, Rosalind (https://rosalind.kcl.ac.uk), which is delivered in partnership with the National Institute for Health Research (NIHR) Biomedical Research Centres at South London & Maudsley and Guy’s & St. Thomas’ NHS Foundation Trusts, and part-funded by capital equipment grants from the Maudsley Charity (award 980) and Guy’s & St. Thomas’ Charity (TR130505). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, King’s College London, or the Department of Health and Social Care.