DERIVATION AND VALIDATION OF A CLINICAL SCORE TO PREDICT DEATH AMONG NON-PALLIATIVE COVID-19 PATIENTS PRESENTING TO EMERGENCY DEPARTMENTS: THE CCEDRRN COVID MORTALITY SCORE ============================================================================================================================================================================ * Corinne M. Hohl * Rhonda J. Rosychuk * Patrick M. Archambault * Fiona O’Sullivan * Murdoch Leeies * Éric Mercier * Gregory Clark * Grant D. Innes * Steven C. Brooks * Jake Hayward * Vi Ho * Tomislav Jelic * Michelle Welsford * Marco L.A. Sivilotti * Laurie J. Morrison * Jeffrey J. Perry * on behalf of the CCEDRRN investigators for the Network of Canadian Emergency Researchers and the Canadian Critical Care Trials Group ## ABSTRACT **Background** Predicting mortality from coronavirus disease 2019 (COVID-19) using information available when patients present to the Emergency Department (ED) can inform goals-of-care decisions and assist with ethical allocation of critical care resources. **Methods** We conducted an observational study to develop and validate a clinical score to predict ED and in-hospital mortality among consecutive non-palliative COVID-19 patients. We recruited from 44 hospitals participating in the Canadian COVID-19 ED Rapid Response Network (CCEDRRN) between March 1, 2020 and January 31, 2021. We randomly assigned hospitals to derivation or validation, and pre-specified clinical variables as candidate predictors. We used logistic regression to develop the score in a derivation cohort, and examined its performance in predicting ED and in-hospital mortality in a validation cohort. **Results** Of 8,761 eligible patients, 618 (7·01%) died. The score included age, sex, type of residence, arrival mode, chest pain, severe liver disease, respiratory rate, and level of respiratory support. The area under the curve was 0·92 (95% confidence intervals [CI] 0·91–0·93) in derivation and 0·92 (95%CI 0·89–0·93) in validation. The score had excellent calibration. Above a score of 15, the observed mortality was 81·0% (81/100) with a specificity of 98·8% (95%CI 99·5–99·9%). **Interpretation** The CCEDRRN COVID Mortality Score is a simple score that accurately predicts mortality with variables that are available on patient arrival without the need for diagnostic tests. **Trial registration** Clinicaltrials.gov, [NCT04702945](http://medrxiv.org/lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT04702945&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) Keywords * coronavirus disease * COVID-19 * mortality * clinical prediction score * prediction model * clinical decision instrument ## INTRODUCTION Throughout the coronavirus disease 2019 (COVID-19) pandemic, health systems around the world have been confronted with, and at times overwhelmed by high numbers of critically ill patients.(1,2) For every critically ill patient in the Emergency Department (ED), a number of less severely ill patients present for care, some of whom deteriorate later, placing additional pressure on resources. Accurate, disease-specific mortality prediction is needed to inform shared decision-making with patients and their families around the patients’ goals of care, and can allow healthcare systems to allocate resources in the most transparent, objective, and fair manner possible to save as many lives as possible, and facilitate timely access to palliative care, if needed.(3–5) Numerous models have been developed to predict mortality from COVID-19, but most were at high-risk of bias.(6–8) Many were developed in small or non-representative patient samples, and enrolled patients from the early pandemic before evidence-based treatments had been identified, included palliative patients, censored outcomes, and had moderate predictive performance.(6) The ISARIC 4C Mortality score is the strongest of those developed.(9) However, it included palliative patients, limiting its utility in risk-stratifying non-palliative COVID-19 patients. In addition, the rule was developed using data from the early pandemic. Most other published rules use imaging or laboratory tests, which precludes their use as a first-line triage tool in the ED where decisions on the appropriateness of intubation and mechanical ventilation may have to be made on arrival.(9–14) Our objective was to develop and validate a clinical score that accurately predicts mortality among non-palliative COVID-19 patients, using clinical variables that are readily available on ED arrival. ## METHODS ### Study design and setting The Canadian COVID-19 ED Rapid Response Network (CCEDRRN, pronounced “SEDrin”) is an ongoing multicentre pan-Canadian registry that enrolls consecutive eligible COVID-19 patients presenting to EDs in hospitals located in eight Canadian provinces, including the four most populous.(15) This study was approved by the research ethics boards of all participating institutions with a waiver of informed consent for enrolment. Model development and reporting followed TRIPOD standards.(16) Funders had no role in the collection, analysis, or interpretation of the data, the writing of the manuscript, or the decision to submit for publication. All authors vouch for the accuracy and completeness of the data, and for adherence to the protocol. CCEDRRN’s patient engagement committee reviewed and provided input into the development of the research question, the choice of outcomes and the study protocol, and reviewed the submitted manuscript. Patient partners were involved in developing CCEDRRN’s website and knowledge translation tools to disseminate study results. ### Study patients Participating sites needed to demonstrate ≥99% compliance in enrolling consecutive eligible patients for their data to be included in this study. We included data from 44 of 50 CCEDRRN sites that met this criterion by the time of the data cut (Appendix Table 1). View this table: [Table 1.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/T1) Table 1. Characteristics and outcomes of patients in the derivation and validation cohorts We included patients with confirmed COVID-19 who presented to the ED of a participating site between March 1, 2020 and January 31, 2021. We defined confirmed COVID-19 as patients presenting with ongoing COVID-19 symptoms and a positive nucleic acid amplification test (NAAT) for severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) obtained within 14 days prior to, or after their arrival in the ED. This allowed us to capture patients who were diagnosed in the community and subsequently presented to the ED, and those with early false negative tests that became positive. We also included patients presenting with COVID-19 symptoms and diagnosed with “confirmed COVID-19” to capture patients who were transferred into a CCEDRRN hospital whose NAAT at the sending site could not be confirmed, and patients who were presumed by treating clinicians to have COVID-19 despite persistently negative NAATs. We excluded patients under 18 years of age, those whose goals of care precluded invasive mechanical ventilation, and patients transferred to a hospital outside of CCEDRRN, as we would have been unable to ascertain their outcomes (Figure 1). We followed patients for 30 days if they were discharged from the ED, or until hospital discharge if their admission lasted longer than 30 days. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/31/2021.07.28.21261283/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/F1) Figure 1. Flow diagram of included and excluded Emergency Department visits ### Data collection Trained research assistants abstracted data from electronic and paper-based medical records into a central, web-based REDCap database (Vanderbilt University; Nashville, TN, USA), and captured demographics, vital signs, symptoms, and comorbidities, COVID-19 exposure risk, diagnostic test results, and patient outcomes. We evaluated the inter-rater agreement of key predictor variables by comparing data collected retrospectively with prospective data.(15) The clinical prediction score was developed after all chart abstractions were complete; research assistants were thus unaware of which clinical variables would be candidate predictor variables. ### Outcome The primary outcome was all-cause ED and in-hospital mortality. All patients had complete follow-up data at the time of the data cut. We categorized patients who were discharged from hospital as alive according to their latest hospitalization. ### Statistical analysis #### Predictor variables All candidate predictor variables were recorded in the ED record. We chose candidate predictors based on literature review and clinical knowledge. They included age, sex, pregnancy, type of residence, ED arrival mode, comorbidities, symptoms, arrival heart rate, systolic blood pressure, oxygen saturation, respiratory rate and Glasgow Coma Scale, ED oxygen delivery, lowest oxygen saturation, physician or nurse impression of respiratory distress, and use of alcohol, tobacco, vaping, and illicit substances (Appendix Table 2). View this table: [Table 2.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/T2) Table 2. Adjusted associations between predictor variables and mortality, and points for the CCEDRRN COVID Mortality Score #### Model development and validation We randomly assigned participating sites to derivation or validation, with the goal of assigning 75% of eligible patients and outcome events to derivation, and the remaining to validation. We examined candidate predictors for co-linearity, and missing and extreme values in the derivation cohort. A few variables had missing values (systolic blood pressure had the most missing at 4.7%; Appendix Table 2). We used five multiple imputations for predictors if missing categorical data could not reasonably be assumed to be absent (e.g., missing documentation of illicit substance use was classified as no substance use). The initial logistic regression model considered all candidate predictors, with continuous predictors fit with restricted cubic splines with three knots. We assessed the strengths of associations between predictors and the outcome using an analysis of variance (ANOVA) plot to inform the degrees of freedom to allocate to each predictor. No additional knots were allocated to continuous predictors. We used a fast step-down procedure to reduce the model to key predictors. We conducted an internal bootstrap validation with 1,000 bootstrap samples to provide an optimism-corrected area under the receiver-operating characteristic (AUC). We categorized continuous predictors based on the relationship between the spline function and outcome. To enable easy clinical use, we categorized age into decades, and arrival respiratory rate cut points of 20 and 30, and assigned integer points to be added to calculate the score. We used a nomogram to assign points to form a score that ranged from −1 to 17. We calculated the sensitivity and specificity at different point thresholds, along with the score’s discrimination and calibration. We validated the model in a cohort of geographically distinct sites that were not part of derivation, and used a single imputation for the few missing respiratory rates (4% were missing). We assessed outcomes independently for ED visits, irrespective of potential subsequent visits leading to death. #### Validation of previously published models We used our study cohort to externally validate other risk prediction tools: the SEIMC score,12 the 4C Mortality Score,10 and the VACO Index.20 We chose these three because they performed well in validation, and the majority of their predictors were available in our data. We calculated the AUCs for these risk prediction tools using cases with complete data on as many predictors as possible.(17) We performed analyses in R using the rms package and used the pmsampsize package for sample size determination.(18) To ensure patient privacy, a cell size restriction policy prohibited reporting counts of less than five. #### Sample size Assuming an event rate of less than 10%, shrinkage of 0.9, and a conservative Cox-Snell R-squared of 0.1, 8.5 events per degree of freedom were required for reliable prediction modeling in the derivation cohort.(19) The 42 candidate predictor variables had 49 degrees of freedom indicating 417 events were required. In the derivation cohort, there were 6,758 patients who made 7,420 ED visits. The derivation cohort exceeded the number of events required. ## RESULTS We assessed 9,704 consecutive COVID-19 patients who made 10,670 ED visits between March 1, 2020 and January 31, 2021 (Figure 1). We excluded 943 patients who met one or more exclusion criteria, and included 8,761 patients who made 9,605 visits in our analyses. The follow-up time was 30 days for discharged patients and between 30 and 229 days for admitted patients. Of these, 618 (7.0%) died in ED or hospital and met the primary outcome. In the derivation cohort, 6,758 patients made 7,420 ED visits to 32 sites (Appendix Table 1). In the validation cohort, 2,054 patients made 2,185 ED visits to 14 different sites. In the derivation cohort 2,705 (36.5%) patients presented during the early pandemic, between March 1 and June 30, 2020, and 4,711 (63.7%) between July 1, 2020 and January 31, 2021 (Table 1). In the validation cohort, 6158 (28.4%) presented during the early pandemic, and 1,567 (71.7%) after July 1, 2020. In derivation, the step-down procedure produced a final model with eight variables (Table 2). The derived model had an optimism-corrected AUC of 0.92. The resulting risk score ranged from −1 to 17. The derivation cohort was well distributed across the CCEDRRN COVID Mortality Score range, and had excellent calibration (calibration intercept of 0 and slope of 1) and discrimination with an AUC of 0.92 (95% CI 0.90 to 0.92, Appendix Figure 1). The CCEDRRN COVID Mortality Score had similar performance in validation. The validation cohort was also distributed across the score ranges, had excellent calibration (calibration intercept of 0 and slope of 1) and discrimination (AUC of 0.92 [95% CI 0.89 to 0.93], Figure 2). ![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/31/2021.07.28.21261283/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/F2) Figure 2. Distribution and performance of the CCEDRRN COVID Mortality Score in the validation cohort (left panel) and combined derivation and validation cohorts (right panel): A) distribution of the score, B) observed in-hospital mortality across the range of the score, C) predicted versus observed probability of in-hospital mortality, and D) receiver operating characteristic curve with area under the curve (AUC) and associated 95% confidence interval. The score had excellent performance across a range of thresholds to rule in and rule out in-hospital mortality (Appendix Tables 4 and 5). These results suggest that scores less than or equal to six would categorize patients at low risk of in-hospital mortality, with a negative predicted value of 99.9%. Patients in the low-risk group had an in-hospital mortality of 0.1%. For scores greater than or equal to 15, the observed in-hospital mortality was 81.0% and the CCEDRRN COVID Mortality Score would categorize patients at high risk of in-hospital mortality, with a specificity of 99.8% and positive predictive value of 81.0%. We conducted an external validation of three risk scores (Figure 3, Appendix Table 5). These scores performed well for the patients with data available yielding AUCs of nearly 0.88, below the AUC of the CCEDRRN COVID Mortality Score (unadjusted for multiple testing and using the validation cohort: AUC for the CCEDRRN COVID Mortality Score was higher compared to the SEIMC score [p=0.035] and the VACO Index [p=0.002], with no evidence of a difference for the 4C Mortality Score [p=0.072]). ![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/31/2021.07.28.21261283/F3.medium.gif) [Figure 3.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/F3) Figure 3. Receiver operator curves and area under the curve (AUC) for the CCEDRRN COVID Mortality Score in the derivation and validation cohorts, the SEIMC score (Berenguer et al.),(11) the 4C Mortality Score (Knight et al.),(9) and the VACO Index (King et al.) in the validation cohort.(20) ## INTERPRETATION ### Main results We derived and validated a parsimonious and simple score to predict in-hospital mortality among non-palliative patients presenting to EDs with COVID-19: the CCEDRRN COVID Mortality Score. We found that eight readily available clinical variables that can be ascertained at the bedside shortly after ED arrival to accurately predict mortality. The CCEDRRN COVID Mortality Score had excellent calibration and discrimination in a geographically distinct cohort of patients who presented to other sites. The CCEDRRN COVID Mortality Score can be used as a highly sensitive score to rule out in-hospital mortality in low-risk patients with a score up to and including six. It can also be used as a rule-in score with scores of 15 or higher being highly specific for in-hospital mortality. Critically ill COVID-19 patients typically require aggressive medical management in the ED shortly after their arrival. Being able to accurately and reliably predict mortality risk on arrival before endotracheal intubation occurs can offer the opportunity to inform discussions about patients’ goals of care, and facilitate early high-quality end of life care for patients most likely to die despite maximum medical intervention. Accurate mortality prediction may be essential when surging cases threaten to overwhelm critical care resources. In those rare situations, the CCEDRRN COVID Mortality Score can guide allocation of scarce resources. ### Explanation of the findings Our model has strengths compared to prior models.(6) We developed the CCEDRRN COVID Mortality Score using simple and readily available variables at the bedside. As a result, our model could be externally validated for use in remote areas without access to laboratory testing or imaging, and in low-income countries where access may be limited or costly. In contrast to prior models, we excluded patients with palliative goals of care, for whom invasive mechanical ventilation was not offered to ensure our model did not predict patients who were expected to succumb, or ineligible for the highest level of critical care.(9–11,13,14,20–24) This avoids the potential for self-fulfilling prophecy bias, whereby the prognostic model predicts the outcome that occurred as a result of a decision to withhold life-sustaining measures.(25–27) Prior models were derived or validated during the early pandemic while COVID-19 testing was restricted to those with severe disease, and did not include consecutive eligible patients, both of which may have resulted in selection bias. Mortality in prior studies ranged from 13% to 30%,(9,11,12,14,22–24) in contrast to 7% in our study. Our rule’s predictive ability depends mostly on age, the patient’s respiratory status. The only comorbidity retained in the final model was moderate to severe liver disease. Two other rules also identified liver disease as a mortality risk factor, perhaps due to the potential for virus-induced liver inflammation.(14,20,28) Other prognostic decision rules have used similar analytic approaches,(20,22,23) but had lower predictive performance with c-statistics ranging from 0.80 to 0.82, and were based on patients from the early pandemic. Other rules have incorporated measures of hypoxemia or respiratory support, corroborating their strong predictive power.(9– 13,20–24) Goodacre et al. identified performance status as a risk factor for increased mortality. In our dataset, this was most closely reflected by arrival from long-term care, which reflected patients with lower performance status.(23) ### Limitations of the study Canada represents a culturally diverse country that offers its citizens universal health coverage. Our model needs to be externally validated in other health systems. Our model predicts in-hospital mortality, and may have missed out-of-hospital deaths that occurred after discharge; however, these are believed to be rare. Our model was based on patients with confirmed COVID-19. While many patients presented to the ED with NAAT confirmed COVID-19, validation in a cohort of patients with suspected COVID-19 is needed. ### Future directions in the area of study As vaccination campaigns roll out around the world, the performance of risk tools will need to be evaluated in patients who are vaccinated and may have a different risk of dying from COVID-19. ### Conclusion In summary, the CCEDRRN COVID Mortality Score is a simple clinical risk score that can be applied in the ED at the bedside to predict a patient’s mortality risk. This tool can be used to inform goals of care decisions. ## Data Availability The CCEDRRN network endorses the guidance put forth by the World Health Organization to enable data sharing to optimize learning. CCEDRRN accepts applications for access to data by external investigators, prioritizing data requests by network Members. ## Authors’ contributions CMH, RJR, JJP, LJM, PA and SCB conceived the study, with input on the design and selection of variables from all other contributors. CMH, LJM, PA, and SCB obtained funding on behalf of the CCEDRRN investigators. CMH, PA, SCB, EM, MW, JH, TJ, VH, and GC managed data collection along with other members of the CCEDRRN, and verify the accuracy of underlying data. CMH, RJR, FOS and JJP developed the analytic plan. FOS performed the analysis, with assistance from RJR, CMH and JJP, including accessing and verification of underlying data. All contributors provided input on the interpretation our findings. CMH and PA drafted the manuscript. All authors reviewed and provided critical input to the final manuscript. ## Data sharing The CCEDRRN network endorses the guidance put forth by the World Health Organization to enable data sharing to optimize learning. CCEDRRN accepts applications for access to data by external investigators, prioritizing data requests by network Members. ## APPENDICES View this table: [Appendix Table 1.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/T3) Appendix Table 1. Characteristics and enrolment periods for participating sites View this table: [Appendix Table 2.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/T4) Appendix Table 2. Candidate variables for entry into regression model View this table: [Appendix Table 3.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/T5) Appendix Table 3. Performance of the CCEDRRN COVID Mortality Score to rule out and rule in in-hospital mortality at different cut-off values in the validation cohort. View this table: [Appendix Table 4.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/T6) Appendix Table 4. Performance of the CCEDRRN COVID Mortality Score to rule out and rule in in-hospital mortality at different cut-off values in the combined derivation and validation cohorts. View this table: [Appendix Table 5.](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/T7) Appendix Table 5. Receiver operator curves and area under the curve (AUC) for the CCEDRRN COVID Mortality Score, the SEIMC score, the 4C Mortality Score, and the VACO Index. ![Appendix Figure 1·](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/31/2021.07.28.21261283/F4.medium.gif) [Appendix Figure 1·](http://medrxiv.org/content/early/2021/07/31/2021.07.28.21261283/F4) Appendix Figure 1· Distribution and performance of the CCEDRRN COVID Mortality Score in the derivation cohort: A) distribution of the score, B) observed in-hospital mortality across the range of the score, C) predicted versus observed probability of in-hospital mortality, and D) receiver operating characteristic curve with area under the curve (AUC) and associated 95% confidence interval. ## Acknowledgements We gratefully acknowledge the assistance of Ms. Amber Cragg in the preparation of this manuscript. We thank the UBC clinical coordinating centre staff, the UBC legal, ethics, privacy and contract staff and the research staff at each of the participating institutions in the network outlined in the attached Supplement. The network would not exist today without the dedication of these professionals. Thank you to all of our patient partners who shared their lived experiences and perspectives to ensure that the knowledge we co-create addresses the concerns of patients and the public. Creating the largest network of collaboration across Canadian Emergency Departments would not have been feasible without the tireless efforts of Emergency Department Chiefs, and research coordinators and research assistants at participating sites. Finally, our most humble and sincere gratitude to all of our colleagues in medicine, nursing, and the allied health professions who have been on the front lines of this pandemic from day one staffing our ambulances, Emergency Departments, ICUs and hospitals bravely facing the risks of COVID-19 to look after our fellow citizens and after one another. We dedicate this network to you. ## Footnotes * * (co-senior authors) * **Funding Acknowledgement:** The network is funded by the Canadian Institutes of Health Research (447679), BC Academic Health Science Network Society, BioTalent Canada, Genome BC (COV024; VAC007), Ontario Ministry of Colleges and Universities (C-655-2129), the Saskatchewan Health Research Foundation (5357) and the Fondation CHU de Québec (Octroi #4007). * **Declaration of interests:** The study authors have no conflicts of interest to declare. * Received July 28, 2021. * Revision received July 28, 2021. * Accepted July 31, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission. ## REFERENCES 1. 1.Tan E, Song J, Deane AM, Plummer MP. Global Impact of Coronavirus Disease 2019 Infection Requiring Admission to the ICU: A Systematic Review and Meta-analysis. Chest. 2021 Feb;159(2):524–36. 2. 2.Rosenbaum L. Facing Covid-19 in Italy — Ethics, Logistics, and Therapeutics on the Epidemic’s Front Line. N Engl J Med. 2020 May 14;382(20):1873–5. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMp 2005492&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 3. 3.Satomi E, Souza PMR de, Thomé B da C, Reingenheim C, Werebe E, Troster EJ, et al. Fair allocation of scarce medical resources during COVID-19 pandemic: ethical considerations. Einstein (Sao Paulo). 2020 Apr 24;18. 4. 4.Arya A, Buchman S, Gagnon B, Downar J. Pandemic palliative care: beyond ventilators and saving lives. CMAJ. 2020 Apr 14;192(15):E400–4. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTkyLzE1L0U0MDAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wNy8zMS8yMDIxLjA3LjI4LjIxMjYxMjgzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 5. 5.Maves RC, Downar J, Dichter JR, Hick JL, Devereaux A, Geiling JA, et al. Triage of Scarce Critical Care Resources in COVID-19 An Implementation Guide for Regional Allocation: An Expert Panel Report of the Task Force for Mass Critical Care and the American College of Chest Physicians. Chest. 2020 Jul;158(1):212–25. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 6. 6.Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020 Apr 7;369:m1328. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjkvYXByMDdfMi9tMTMyOCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIxLzA3LzMxLzIwMjEuMDcuMjguMjEyNjEyODMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 7. 7.Update to living systematic review on prediction models for diagnosis and prognosis of covid-19. BMJ. 2021 Feb 3;372:236. 8. 8.Gupta RK, Marks M, Samuels THA, Luintel A, Rampling T, Chowdhury H, et al. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study. Eur Respir J [Internet]. 2020 Dec 24 [cited 2021 Jun 7];56(6). Available from: [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7518075/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7518075/) 9. 9.Knight SR, Ho A, Pius R, Buchan I, Carson G, Drake TM, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ [Internet]. 2020 Sep 9 [cited 2021 Jun 7];370. Available from: <[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7116472/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7116472/)> 10. 10.Cho S-Y, Park S-S, Song M-K, Bae YY, Lee D-G, Kim D-W. Prognosis Score System to Predict Survival for COVID-19 Cases: a Korean Nationwide Cohort Study. J Med Internet Res. 2021 Feb 22;23(2):e26257. 11. 11.Berenguer J, Borobia AM, Ryan P, Rodríguez-Baño J, Bellón JM, Jarrín I, et al. Development and validation of a prediction model for 30-day mortality in hospitalised patients with COVID-19: the COVID-19 SEIMC score. Thorax. 2021 Feb 25; 12. 12.Nicholson CJ, Wooster L, Sigurslid HH, Li RH, Jiang W, Tian W, et al. Estimating risk of mechanical ventilation and in-hospital mortality among adult COVID-19 patients admitted to Mass General Brigham: The VICE and DICE scores. EClinicalMedicine. 2021 Mar;33:100765. 13. 13.Liang W, Liang H, Ou L, Chen B, Chen A, Li C, et al. Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19. JAMA Intern Med. 2020 Aug 1;180(8):1081. 14. 14.Magro B, Zuccaro V, Novelli L, Zileri L, Celsa C, Raimondi F, et al. Predicting in-hospital mortality from Coronavirus Disease 2019: A simple validated app for clinical use. PLoS One. 2021;16(1):e0245281. 15. 15.Hohl CM, Rosychuk RJ, McRae AD, Brooks SC, Archambault P, Fok PT, et al. Development of the Canadian COVID-19 Emergency Department Rapid Response Network population-based registry: a methodology study. CMAJ Open. 2021 Jan 1;9(1):E261–70. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiY21ham8iO3M6NToicmVzaWQiO3M6ODoiOS8xL0UyNjEiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wNy8zMS8yMDIxLjA3LjI4LjIxMjYxMjgzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 16. 16.Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015 Jan 6;162(1):W1–73. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7326/M14-0698&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25560730&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 17. 17.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988 Sep;44(3):837–45. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2307/2531595&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=3203132&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1988Q069100016&link_type=ISI) 18. 18.Ensor J, Martin EC, Riley RD. pmsampsize: Calculates the Minimum Sample Size Required for Developing a Multivariable Prediction Model [Internet]. 2020. Available from: [https://CRAN.R-project.org/package=pmsampsize](https://CRAN.R-project.org/package=pmsampsize) 19. 19.Riley RD, Snell KI, Ensor J, Burke DL, Jr FEH, Moons KG, et al. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med. 2019;38(7):1276–96. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/sim.7992&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=30357870&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 20. 20.King Jr JT, Yoon JS, Rentsch CT, Tate JP, Park LS, Kidwai-Khan F, et al. Development and validation of a 30-day mortality index based on pre-existing medical administrative data from 13,323 COVID-19 patients: The Veterans Health Administration COVID-19 (VACO) Index. PLoS One. 2020 Nov 11;15(11):e0241825. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0241825&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33175863&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 21. 21.Haimovich AD, Ravindra NG, Stoytchev S, Young HP, Wilson FP, van Dijk D, et al. Development and Validation of the Quick COVID-19 Severity Index: A Prognostic Tool for Early Clinical Decompensation. Ann Emerg Med. 2020 Oct;76(4):442–53. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.annemergmed.2020.07.022&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 22. 22.Chua F, Vancheeswaran R, Draper A, Vaghela T, Knight M, Mogal R, et al. Early prognostication of COVID-19 to guide hospitalisation versus outpatient monitoring using a point-of-test risk prediction score. Thorax. 2021 Mar 10; 23. 23.Goodacre S, Thomas B, Sutton L, Burnsall M, Lee E, Bradburn M, et al. Derivation and validation of a clinical severity score for acutely ill adults with suspected COVID-19: The PRIEST observational cohort study. PLoS One. 2021;16(1):e0245840. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0245840&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 24. 24.Gupta RK, Harrison EM, Ho A, Docherty AB, Knight SR, Smeden M van, et al. Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study. Lancet Respir Med. 2021 Apr 1;9(4):349–59. 25. 25.Turgeon AF, Lauzier F, Simard J-F, Scales DC, Burns KEA, Moore L, et al. Mortality associated with withdrawal of life-sustaining therapy for patients with severe traumatic brain injury: a Canadian multicentre cohort study. CMAJ. 2011 Oct 4;183(14):1581–8. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgzLzE0LzE1ODEiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wNy8zMS8yMDIxLjA3LjI4LjIxMjYxMjgzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 26. 26.Creutzfeldt CJ, Becker KJ, Weinstein JR, Khot SP, McPharlin TO, Ton TG, et al. Do-Not-Attempt-Resuscitation Orders and Prognostic Models for Intraparenchymal Hemorrhage. Crit Care Med. 2011 Jan;39(1):158–62. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/CCM.0b013e3181fb7b49&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21037471&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) 27. 27.Wilkinson D. The self-fulfilling prophecy in intensive care. Theor Med Bioeth. 2009;30(6):401–10. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11017-009-9120-6&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19943193&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F31%2F2021.07.28.21261283.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000273812900001&link_type=ISI) 28. 28.Demirtas CO, Keklikkiran C, Ergenc I, Erturk Sengel B, Eskidemir G, Cinel I, et al. Liver stiffness is associated with disease severity and worse clinical scenarios in coronavirus disease 2019: A prospective transient elastography study. Int J Clin Pract. 2021 May 16;e14363.