Systematic review and critical appraisal of prediction models for diagnosis and prognosis of COVID-19 infection

Patient and public involvement

It was not appropriate or possible to involve patients or the public in the design, conduct, or reporting of our research. The study protocol and preliminary results were made publicly available on osf.io/ehc47/ and medRxiv.

No ethical approval was required for the current study.

Primary datasets

Twenty-five studies used data on COVID-19 cases from China (see Supplementary Table 1), one study used data on Italian cases,³¹ and one study used international data (among others, United States, United Kingdom, China).³⁵ Based on 18 of the 25 studies that reported study dates, data were collected between 8^th December 2019 and 15^th March 2020. The duration of follow-up was unclear in most studies, although one reported a median follow-up of 8.4 days,¹⁹ whilst another reported a median follow-up of 15 days.³⁷ Some Chinese centers provided data to multiple studies, but it was unclear how much these datasets overlapped across our 25 identified studies. One study used U.S. Medicare claims data from 2015 to 2016 to estimate COVID-19 vulnerability,⁸ two studies used control CT scans from the USA or Switzerland,^11,25 and one study used simulated data.¹⁸

All but one study²⁴ developed prediction models for use in adults. The median age varied between studies (from 34 to 65 years, see Supplementary Table 1), as did the percentage of men (from 41% to 61%).

Among the six studies that developed prognostic models to predict mortality risk in individuals with confirmed or suspected COVID-19 infection, the percentage of deaths varied between 8% and 59% (See Table 1). This wide variation is in part due to severe sampling bias caused by studies excluding participants who still had the disease at the end of the study period (i.e., neither recovered nor died). ^7,20-22 In addition, length of follow-up may have varied between studies (but was rarely reported), and there may be local and temporal variation in how people were diagnosed or hospitalized (and hence recruited for the studies). Among the 18 diagnostic model studies, there was only one that reported on prevalence of COVID-19 infection in those suspected of having COVID-19; the prevalence was 19% (development dataset) and 24% (validation dataset).³⁰ One study reported 8% of severe cases among confirmed pediatric COVID-19 cases.²⁴ Since 16 diagnostic studies used either case-control sampling or an unclear method of data collection, the prevalence in these diagnostic studies may not have been representative of their target population.

In what follows, we give an overview of the 31 prediction models reported in the 27 identified studies (Table 1). Modeling details are provided in Supplementary Table 2, and the availability of models in a format for use in clinical practice is discussed in Box 1.

Models to predict the risk of hospital admission due to COVID-19 pneumonia in the general population

Three models predicted the risk of hospital admission for COVID-19 pneumonia for individuals in the general population, but used admission due to non-tuberculosis pneumonia, influenza, acute bronchitis, or upper respiratory infections as outcomes in a dataset without any COVID-19 cases (see Table 1).⁸ Among the predictors were age, sex, previous hospital admissions, comorbidity data, and social determinants of health. The study estimated C-indices of 0.73, 0.81 and 0.81 for the three models.

Diagnostic models to detect COVID-19 infection in symptomatic individuals

One study developed a model to detect COVID-19 pneumonia in fever clinic patients (estimated C-index 0.94),¹⁰ one to diagnose COVID-19 in suspected cases (estimated C-index 0.97),³⁰ one to diagnose COVID-19 in suspected and asymptomatic cases (estimated C-index 0.87),¹² one to diagnose COVID-19 using deep learning of genomic sequences (estimated C-index 0.98),³⁵ and one to diagnose severe disease in symptomatic paediatric inpatients based on direct bilirubin and alaninetransaminase (reporting an F1 score of 1.00, indicating 100% observed sensitivity and specificity).²⁴ Only one study reported assessing calibration, but it was unclear how this was done.¹² Predictors used in more than one model were age (n=3), body temperature or fever (n=2), and signs and symptoms (such as shortness of breath, headache, shiver, sore throat, fatigue) (n=2) (see Table 1).

Thirteen prediction models were proposed to support the diagnosis of COVID-19 or COVID-19 pneumonia (and monitor progression) based on CT images. The predictive performance varied widely, with estimated C-index values ranging from 0.81 to nearly 1.

Prognostic models for patients diagnosed with COVID-19 infection

We identified ten prognostic models (Table 1). Of these, six estimated mortality risk in suspected or confirmed COVID-19 patients.^{7,18,19,21,22,37} The intended use of these models (namely when to use it, in whom to use it, and the prediction horizon, e.g., mortality by what time) was not clearly described. Two models aimed to predict a hospital stay of more than 10 days from admission.²⁰ Two models aimed to predict progression to a severe or critical state.^9,32 Predictors included in more than one prognostic model were age (n=5), sex (n=2), features derived from CT-scoring (n=5), C-reactive protein (n=3), lactic dehydrogenase (n=3), and lymphocyte count (n=2) (see Table 1).

Only two studies predicting mortality reported a C-index; they obtained estimates of 0.90 ²² and 0.98 ⁷. One study also evaluated calibration.⁷ When applied to new patients, their model yielded probabilities of mortality that were too high for low-risk patients and too low for high-risk patients (calibration slope >1), despite excellent discrimination.⁷ One study developed two models to predict a hospital stay of >10 days and estimated C-indices of 0.92 and 0.96.²⁰ The two studies that developed models to predict progression to a severe or critical state estimated C-indices of 0.95 and 0.85.^9,32 One of these also reported perfect calibration, but it was unclear how this was evaluated. ³²

Risk of bias

All models were at high risk of bias according to assessment with PROBAST (Table 1), which suggests that their predictive performance when used in practice is likely lower than what is reported, and so gives concern that their predictions are unreliable. Details on common causes for risk of bias are given in Box 2 for each type of model.

Eleven of the twenty-seven studies had a high risk of bias for the “participants” domain (Table 2), indicating that the participants enrolled in the studies may not be representative for the models’ targeted populations. Unclear reporting on the inclusion of participants prohibited a risk of bias assessment in eight studies. Four out of twenty-seven studies had a high risk of bias for the “predictors” domain, indicating that predictors were not available at the models’ intended time of use, not clearly defined, or influenced by the outcome measurement. The diagnostic model studies that used CT imaging predictors were all scored as “unclear” on the “predictors” domain. The publications often lacked clear information on the preprocessing steps (e.g., cropping of images). Moreover, complex machine learning algorithms transform CT images into predictors in an intransparent way, which makes it challenging to fully apply the PROBAST predictors section for such imaging studies. Most studies used outcomes that are easy to assess (e.g., death, presence of COVID-19 by laboratory confirmation). Nonetheless, there was reason to be concerned of bias induced by the outcome measurement in ten studies, due to the use of subjective or proxy-outcomes (e.g., non COVID-19 severe respiratory infections).

All studies were at high risk of bias for the “analysis” domain (Table 2). Many studies had small sample sizes (Table 1), leading to an increased risk of overfitting, particularly if complex modeling strategies were used. Three studies did not report the predictive performance of the developed model, and one study reported only the apparent performance (that is, the performance in the exact same data as was used to develop the model, without adjustment for optimism due to potential overfitting).

Four models were externally validated in the model development study (i.e., in an independent dataset, excluding random train-test splits and temporal splits).^7,12,25,32 However, in three of these studies, the external validation datasets are likely unrepresentative of the target population (Box 2).^7,12,25 Consequently, predictive performance may be different if the model were applied in the target population. Gong, Ou, et al had a satisfactory predictive performance on two unbiased but small external validation datasets.³² One study was a small (n=27) external validation reporting satisfactory predictive performance of a model originally developed for avian influenza H7N9 pneumonia, but patients that had not recovered at the end of the study period were excluded, leading to a selection bias.²² Only three studies assessed calibration, ^7,12,32 but this was likely done suboptimally in two studies.^12,32

Main findings

In this systematic review of prediction models related to the COVID-19 pandemic, we identified and critically appraised 27 studies that described 31 prediction models for detecting individuals at risk for hospital admission for COVID-19 pneumonia in the general population, for diagnosis of COVID-19 in symptomatic individuals, and for prognosis of COVID-19 infected patients. All models reported good to even excellent predictive performance, but all were appraised as high risk of bias, due to a combination of poor reporting and poor methodological conduct for participant selection, predictor description and statistical methods used. As expected, in these early COVID-19 related prediction model studies, clinical data from COVID-19 patients is still scarce and limited to data from China, Italy, and international registries. With few exceptions, the available sample size and number of events for the outcomes of interest were limited, which is a known problem for building prediction models, increasing the risk of overfitting the prediction model.³⁹ A high risk of bias implies that these models are likely to perform worse in practice than the performance that is reported by the researchers. Hence, the estimated C-indices, often close to 1 and indicating near-perfect discrimination, are highly likely to be optimistic. Five studies carried out an external validation, ^{7,12,25,32,22} and only one study assessed calibration correctly.⁷

We reviewed thirteen studies that used advanced machine learning methodology on chest CT scans to diagnose COVID-19 disease, COVID-19 related pneumonia, or to assist in segmentation of lung images. The predictive performance measures showed a high to almost perfect ability to identify COVID-19, although these models and their evaluations also suffered from a high risk of bias, notably due to poor reporting and an artificial mix of COVID-19 cases and non-cases.

Challenges and opportunities

The main aim of prediction models is to support medical decision making. It is therefore key to identify a target population in which predictions serve a clinical need, and a representative dataset (preferably comprising consecutive patients) on which the prediction model can be developed and validated. This target population must also be carefully described such that the performance of the developed or validated model can be appraised in context, and users know in which individuals the model can be applied to make predictions. However, the included studies in our systematic review often lacked an adequate description of the study population, which leaves users of these models in doubt of the models’ applicability. While we recognize that all studies were done under severe time constraints caused by urgency, we recommend that any studies currently in preprint and all future studies should adhere to the TRIPOD reporting guideline¹⁵ to improve the description of their study population as well as their modeling choices. TRIPOD translations (e.g., in Chinese and Japanese) are also available at www.tripod-statement.org.

A better description of the study population may also help understand the observed variability in the reported outcomes across studies, such as COVID-19 related mortality. The variability in the relative frequencies of the predicted outcomes presents an important challenge to the prediction modeler: a prediction model applied in a setting with a different relative frequency of the outcome may produce predictions that are miscalibrated ⁴⁰ and may need to be updated before it can safely be applied in that new setting.^16,41 Indeed, such an update may often be required when prediction models are transported to different healthcare systems, which requires COVID-19 patient data to be available from that system.

COVID-19 prediction problems will often not present as a simple binary classification task. Complexities in the data should be handled appropriately. For example, a prediction horizon should be specified for prognostic outcomes (e.g., 30-day mortality). If study participants have neither recovered nor died within that time period, their data should not be excluded from analysis, as most reviewed studies have done. Instead, an appropriate time-to-event analysis should be considered to allow for administrative censoring.¹⁶ It should be noted that censoring due to other reasons, for instance due to quick recovery and loss to follow-up of patients that are no longer at risk of death from COVID-19, may necessitate analysis in a competing risk framework.⁴²

Instead of developing and updating predictions in their local setting, Individual Participant Data (IPD) from multiple countries and healthcare systems may facilitate better understanding of the generalizability and implementation prediction models across different settings and populations, and may greatly improve their applicability and robustness in routine care.^43-47

The evidence base for the development and validation of prediction models related to COVID-19 will quickly increase over the coming months. Together with the increasing evidence from predictor finding studies ^48-54 and open peer review initiatives for COVID-19 related publications,⁵⁵ data registries ^56-60 are being set up. To maximize the new opportunities and to facilitate IPD meta-analyses, the WHO has recently released a new data platform to encourage sharing of anonymized COVID-19 clinical data.⁶¹ To leverage the full potential of these evolutions, international and interdisciplinary collaboration in terms of data acquisition and model building is crucial.

Limitations of this study

With new publications on COVID-19 related prediction models that are currently quickly entering the medical literature, this systematic review cannot be viewed as an up-to-date list of all currently available COVID-19 related prediction models. Also, 24 of the studies we reviewed were only available as a preprint, and they might improve after peer review, when entering the official medical literature. We have also found other prediction models which are currently implemented in clinical practice without scientific publications ⁶² and web risk calculators launched for use while the scientific manuscript was still under review (and unavailable upon request).⁶³ These unpublished models naturally fall outside the scope of this review of the literature.

Implications for practice

All 31 reviewed prediction models were found to have a high risk of bias and evidence from independent external validation of these models is currently lacking. However, the urgency of diagnostic and prognostic models to assist in quick and efficient triage of patients in the COVID-19 pandemic may encourage clinicians to implement prediction models without sufficient documentation and validation. Although we cannot let perfect be the enemy of good, earlier studies have shown that models were of limited use in the context of a pandemic,⁶⁴ and they may even cause more harm than good.⁶⁵ Hence, we cannot recommend any model for use in practice at this point.

We anticipate that more COVID-19 data on the individual participant level will soon become available. These could be used to validate and update currently available prediction models.¹⁶ For example, one model predicting progression to severe COVID-19 within 15 days after admission showed promising discrimination when validated externally on two small but unselected cohorts.³² As reporting in this study was insufficiently detailed and the validation was in small Chinese datasets, validation in larger, international datasets is needed. Due to differences between health care systems (e.g., Chinese and European) in admission, discharge, and testing criteria for patients with COVID-19, we anticipate most existing models will need to be updated (i.e., adjusted to the local setting).

When building a new prediction model, it is recommended to build on previous literature and expert opinion to select predictors, rather than selecting predictors in a purely data-driven way ¹⁶. This is especially true for datasets with limited sample size.⁶⁶ Based on the predictors included in multiple models identified by our review, we encourage researchers to consider incorporating the following as candidate predictors: (i) for diagnostic models - age, body temperature, and (respiratory) signs and symptoms; (ii) for prognostic models - age, sex, C-reactive protein, lactic dehydrogenase, lymphocyte count, and potentially features derived from CT-scoring. Predictors that were included in both a diagnostic and a prognostic model were albumin (or albumin/globin), direct bilirubin, and red blood cell distribution width; these could be considered as well. By pointing to the most important methodological challenges and issues in design and reporting of the currently available models, we hope to have provided a useful starting point for further studies aiming at developing new models or validating and updating existing ones.

This systematic review aims to be the first stage of a living review of this field, in collaboration with the Cochrane Prognosis Methods Group. We will update this review and appraisal continuously, to provide up-to date information for healthcare decision makers and professionals, as more international research emerges over time.