Predicting Disease Progression in COVID19: A Score Based On Lab Tests At Time Of Diagnosis

Study Design

We obtained the ethical approval from the Kuwait Ministry of Health ethical review committee. We carried out a prospective cohort study to explore the factors associated with COVID19 severity amongst COVID19 in-patients admitted to the Infectious Diseases Hospital in Kuwait between 24 February 2020 and 20 April 2020.

Inclusion Criteria and Case Definition

All consecutive patients meeting the case definition and who had tested positive are diagnosed to have COVID19 and admitted for quarantine and observation. Testing for COVID19 was undertaken via real-time reverse-transcriptase–polymerase-chain-reaction (RT-PCR) assay of nasal swab specimens. All diagnostic tests were performed at the Jaber Hospital laboratory. All positive patients stay in hospital till they have had resolution of symptoms (afebrile for more than 72 hours plus saturation ≥ 94%) so long as they are also more than 7 days since symptoms onset, have completed 14 days since testing positive and are Improving or have regression of abnormalities on imaging. Discharge occurs after two consecutive negative tests >24h apart. A standardized form was completed prospectively for data collection, including demographic data, clinical data and radiographic / laboratory results.

Severity grouping (main outcome)

We pre-specified the main outcome to be moderate-severe COVID19 defined based on need for hospital support, while mild cases will have a mild clinical course needing only symptomatic management (the majority). The severity grouping algorithm was determined prospectively as follows:

1. Assign missing status to everyone

2. Assign moderate to severe status to those with hospital course that led to:

   1. Death

   2. Consolidation on chest x-ray or shortness of breath on admission

   3. ICU admission

   4. Hospital stay >14 days AND in hospital receiving active treatment

3. Assign moderate to severe status additionally to those receiving the following treatments regardless of hospital stay or discharge status:

   1. Systemic gluococorticoids

   2. Intravenous immunoglobulin

   3. Oxygen therapy

   4. Non-invasive ventilation OR mechanical ventilation

   5. Extracorporeal membrane oxygenation

   6. Continuous renal replacement therapy

4. Assign mild status to those meeting the following:

   1. Not on treatment, not discharged and duration in hospital >10 days

   2. Discharged and no treatment received

   3. Discharged and duration in hospital >14 days and severity score not already assigned to severe status above

Apart from the severity group outcome, ICU admission and death were defined as outcomes for the purposes of model evaluation but not in model building as they were subsumed within the main outcome.

Statistical Analysis

Categorical variables were summarized using percentages while continuous variables were summarized using medians with interquartile ranges. Potential predictors for the occurrence of severe COVID19 were investigated using regularized logistic regression (a machine learning algorithm).

All promising predictor variables (demographics, laboratory test results, comorbidities and selected symptoms; see details in supplementary material with details of tests used and the variables considered) with a plausible biological reason for inclusion were assessed at admission prior to the outcome being known. Variables used in the outcome definition were excluded from condideartion as a predictor. All predictors included in the analysis were converted to variable scores (if a continuous variable) prior to entry into the regularized regression procedure. The scores for these transformed continuous variables were 0 or 1 representing values below or above the median while for untransformed binary variables was also 0 or 1 (absent vs present respectively). We decided to categorize continuous variables for the often criticized goal of aiding clinical interpretation and maintaining simplicity. While this may have introduced loss of information we did not plan to reconsider this approach unless there was a problem with predictive performance since using continuous variables would make the model less applicable to rapid implementation during the pandemic. Of note cut-points for continuous variables were predefined (at the median) and decided upon prior to data analysis.

The regularized regression procedure used was a 5-fold cross validated lasso logit regression and we identified the best model using the largest value of the tuning parameter that was within one standard deviation of the optimum value (i.e. the value that minimizes the mean squared error of prediction), which leads to a more parsimonious model. Then an unrestricted logit model was fitted to the selected set of predictors from the lasso model and these coefficients used to determine a preliminary severity risk score for COVID19. This was done by rounding the beta coefficient from the unrestricted logit model on the selected variables to use as integer weights for each variable score. The assigned weights multiplied by variable score were then summed to compute that patient’s severity score (Table 2). Regularized regression was run using lassopack in Stata.⁶ Time to event analyses were not considered even though this was a dynamic cohort because duration of stay varied based on physician discretion and there was no real risk of over-representing those with the moderate to severe disease outcome for COVID19.

To assess discrimination of the model, the C statistic (area under the receiver operating characteristic curve) and its 95% CI were computed. To test model internal validity, a straightforward and fairly popular approach was used which was to randomly split the data in two parts: one to develop the model and another to measure its performance. We used randomtag in Stata to tag the two data-sets (N=700 for training and the rest for validation) and the score was developed on the training data-set. Operating characteristics of the score were then assessed using application to the full data-set. Finally, calibration of the model was assessed using pmcalplot in Stata.⁷ All analyses were performed using Stata MP version 15 (College Station, TX, USA) and the confidence level was set at 95%.

Baseline Characteristics

From 24 February 2020 till 20 April 2020, 1096 consecutive patients with COVID19 were included in the study. 888 (81%) were males and 335 (30%) had comorbid conditions. The mean age was 41.4 years (range, 1-93 years) and the average time between the onset of symptoms and admission was 4.2 days (95% CI 3.8 - 4.5 days). The most frequent clinical symptoms of COVID19 were chills and cough. Of the 1096 patients, clinical course could be defined (see next section) for 643 patients (the rest had not yet reached the outcome) and these formed the cohort for the model development. Details of the basic characteristics of these patients are given in Table 1. Of note, the final severity score could be computed for 1008 individuals (642/700 in the training cohort and 366/396 in the validation cohort).

The clinical course was a mild COVID19 course for 549 patients, a moderate – severe COVID19 course for 94 patients leading to 15 events per variable for the selected model. The course of illness was set to missing for 453 patients. Of the 94 severe cases, 42 were admitted to the intensive care unit for acute respiratory care and 19 died.

In total, 581 out of 643 patients had data on both the clinical course and the severity score parameters and of these 363 were randomly allocated to the training set and 218 to the validation set (using randomtag in Stata). The prediction model was built on the 363 subjects in the training data-set and validated on the 218 in the validation data-set. However final severity scores were calculable on 1008 patients and these were used for score assessment when the outcome was not clinical course in hospital.

Severity Score of COVID19

Variables selected through lasso logit regression included age, CRP, procalcitonin, lymphocyte percentage, monocyte percentage and serum albumin (Table 2). The range of scores seen were from minus 32 to plus 22 for this cohort.

The area under the curve (AUC) for the training sample (N=363) model was equal to 0.834 (95% CI, 0.779-0.889), which indicates good model discrimination (figure 1). The validation cohort (N=218) demonstrated equally good discrimination with AUC 0.794 (95% CI, 0.710 – 0.879) and is also depicted in Figure 1. A calibration plot of observed against expected probabilities for assessment of prediction model performance on the validation cohort demonstrated reasonably good model calibration (Supplementary material Figure A). As expected, the training cohort showed perfect calibration (Supplementary material Figure B)..

• Download figure
• Open in new tab

Figure 1:

Area under the receiver operating characteristic curve for the severity score in the training (top) and validation (bottom) cohorts

A cutoff for low, intermediate and high risk was chosen according to the score’s performance (thresholds at 90% sensitivity and 90% specificity) and patients with a score −7 or less were at low risk and those with a score 16 or above were at high risk of more severe illness requiring hospital management. The risk groups also demonstrated good discrimination of the various outcomes (Table 3) and the interval likelihood ratios by risk level (based on the score) are presented in Table 4. From the interval likelihood ratios and the baseline prevalence of a severe clinical course (14.6%) we could compute posterior probabilities. The interval likelihood ratio for the severe risk level was 5.33. This means that for a patient at this risk level, the posterior probability of a severe clinical course is 48%. For a patient at the low risk level (with a score of - 7 or less), the posterior probability of a severe clinical course is down to only 4%. Given the way the thresholds were created, at the low risk threshold the score has a sensitivity of 90 % and similarly at the high risk threshold it has a specificity of 90%.