Abstract
Objective To review and critically appraise published and preprint reports of models that aim to predict either (i) presence of existing COVID-19 infection, or (ii) future complications in individuals already diagnosed with COVID-19. Any models to identify subjects at risk for COVID-19 in the general population were also included.
Design Rapid systematic review and critical appraisal of prediction models for diagnosis or prognosis of COVID-19 infection.
Data sources PubMed, EMBASE via Ovid, Arxiv, medRxiv and bioRxiv until 13th March 2020.
Study selection Studies that developed or validated a multivariable COVID-19 related prediction model. Two authors independently screened titles and abstracts.
Data extraction Data from included studies were extracted independently by at least two authors based on the CHARMS checklist, and risk of bias was assessed using PROBAST. Data were extracted on various domains including the participants, predictors, outcomes, data analysis, and prediction model performance.
Results 1916 titles were screened. Of these, 15 studies describing 19 prediction models were included for data extraction and critical appraisal. We identified three models to predict hospital admission from pneumonia and other events (as a proxy for covid-19 pneumonia) in the general population; nine diagnostic models to detect COVID-19 infection in symptomatic individuals (seven of which were deep learning models for COVID-19 diagnosis utilising computed tomography (CT) results); and seven prognostic models for predicting mortality risk, or length of hospital stay. None of the 15 studies used data on COVID-19 cases outside of China. Predictors included in more than one of the 19 models were: age, sex, comorbidities, C-reactive protein, lymphocyte markers (percentage or neutrophil-to-lymphocyte ratio), lactate dehydrogenase, and features derived from CT images. Reported C-index estimates for the prediction models ranged from 0.73 to 0.81 in those for the general population (reported for all 3 general population models), from 0.81 to > 0.99 in those for diagnosis (reported for 5 of the 9 diagnostic models), and from 0.90 to 0.98 in those for prognosis (reported for 4 of the 7 prognostic models). All studies were rated at high risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, and poor statistical analysis, including high risk of model overfitting. Reporting quality varied substantially between studies. A description of the study population and intended use of the models was absent in almost all reports, and calibration of predictions was rarely assessed.
Conclusion COVID-19 related prediction models for diagnosis and prognosis are quickly entering the academic literature through publications and preprint reports, aiming to support medical decision making in a time where this is needed urgently. Many models were poorly reported and all appraised as high risk of bias. We call for immediate sharing of the individual participant data from COVID-19 studies worldwide to support collaborative efforts in building more rigorously developed and validated COVID-19 related prediction models. The predictors identified in current studies should be considered for potential inclusion in new models. We also stress the need to adhere to methodological standards when developing and evaluating COVID-19 related predictions models, as unreliable predictions may cause more harm than benefit when used to guide clinical decisions about COVID-19 in the current pandemic.
Systematic review registration osf.io/ehc47/
What is already known on this topic
- The sharp recent increase in COVID-19 infections has put a strain on healthcare systems worldwide, necessitating efficient diagnosis of patients suspected of the infection and prognostication of COVID-19 confirmed cases.
- Viral nucleic acid testing and chest CT are standard methods for diagnosing COVID-19, but are time-consuming.
- Earlier reports suggest that the elderly, patients with comorbidity (COPD, cardiovascular disease, hypertension), and patients presenting with dysapnoea are vulnerable to more severe morbidity and mortality after COVID-19 infection.
What this study adds
- We identified three models to predict hospital admission from pneumonia and other events (as a proxy for covid-19 pneumonia) in the general population.
- We identified nine diagnostic models for COVID-19 detection in symptomatic patients. Seven of these were neural network models based on CT images.
- We identified seven prognostic models, of which five aimed to predict mortality risk in confirmed COVID-19 patients and two aimed to predict a hospital stay of more than 10 days from admission.
- All included studies were appraised at high risk of bias, suggesting concern that the models may be flawed and perform poorly when applied in practice, such that their predictions may be unreliable.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
LW is a post-doctoral fellow of Research Foundation – Flanders (FWO). BVC received support from FWO (grant G0B4716N) and Internal Funds KU Leuven (grant C24/15/037). TD acknowledges financial support from the Netherlands Organization for Health Research and Development (Grant Numbers: 91617050). KGMM gratefully acknowledges financial support from Cochrane Collaboration (SMF 2018). KIES is funded by the National Institute for Health Research School for Primary Care Research (NIHR SPCR). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. GSC was supported by the NIHR Biomedical Research Centre, Oxford, and Cancer Research UK (programme grant: C49297/A27294). The funders played no role in study design, data collection, data analysis, data interpretation, or reporting. The guarantors had full access to all the data in the study, take responsibility for the integrity of the data and the accuracy of the data analysis, and had final responsibility for the decision to submit for publication.
Author Declarations
All relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.
Yes
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
Extracted data is partially reported in Tables 1-2 and Supplementary Tables 1-2. The original manuscripts are published open access (links provided in the reference list). The protocol for this systematic review was registered on OSF (osf.io/ehc47/).
