Bayesian Prediction of Severe Outcomes in the LabMarCS: Laboratory Markers of COVID-19 Severity - Bristol Cohort ================================================================================================================ * Brian Sullivan * Edward Barker * Philip Williams * Louis MacGregor * Ranjeet Bhamber * Matt Thomas * Stefan Gurney * Catherine Hyams * Alastair Whiteway * Jennifer A Cooper * Chris McWilliams * Katy Turner * Andrew W. Dowsey * Mahableshwar Albur ## Abstract **Objectives** To develop cross-validated prediction models for severe outcomes in COVID-19 using blood biomarker and demographic data; Demonstrate best practices for clinical data curation and statistical modelling decisions, with an emphasis on Bayesian methods. **Design** Retrospective observational cohort study. **Setting** Multicentre across National Health Service (NHS) trusts in Southwest region, England, UK. **Participants** Hospitalised adult patients with a positive SARS-CoV 2 by PCR during the first wave (March – October 2020). 843 COVID-19 patients (mean age 71, 45% female, 32% died or needed ICU stay) split into training (n=590) and validation groups (n=253) along with observations on demographics, co-infections, and 30 laboratory blood biomarkers. **Primary outcome measures** ICU admission or death within 28-days of admission to hospital for COVID-19 or a positive PCR result if already admitted. **Results** Predictive regression models were fit to predict primary outcomes using demographic data and initial results from biomarker tests collected within 3 days of admission or testing positive if already admitted. Using all variables, a standard logistic regression yielded an internal validation median AUC of 0.7 (95% Interval [0.64,0.81]), and an external validation AUC of 0.67 [0.61, 0.71], a Bayesian logistic regression using a horseshoe prior yielded an internal validation median AUC of 0.78 [0.71, 0.85], and an external validation median AUC of 0.70 [0.68, 0.71]. Variable selection performed using Bayesian predictive projection determined a four variable model using Age, Urea, Prothrombin time and Neutrophil-Lymphocyte ratio, with a median AUC of 0.74 [0.67, 0.82], and external validation AUC of 0.70 [0.69, 0.71]. **Conclusions** Our study reiterates the predictive value of previously identified biomarkers for COVID-19 severity assessment. Given the small data set, the full and reduced models have decent performance, but would require improved external validation for clinical application. The study highlights a variety of challenges present in complex medical data sets while maintaining best statistical practices with an emphasis on showcasing recent Bayesian methods. ## Introduction Globally, as of 14 July 2022, there have been 556 million confirmed cases of COVID-19, including 6.35 million deaths, with 23.1 million cases in the UK, resulting in over 181,000 deaths (WHO Coronavirus (COVID-19) Dashboard, [https://covid19.who.int/](https://covid19.who.int/)). COVID-19 has a wide spectrum of clinical features ranging from asymptomatic to severe systemic illness with a significant attributable mortality, while clinical manifestations are variable especially in the most vulnerable groups and immunocompromised people [1]. COVID-19 is a multi-system disease resulting in the derangements of homeostasis affecting pulmonary, cardiovascular, coagulation, haematological, oxygenation, hepatic, renal and fluid balance [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. Although the majority of people with COVID-19 will have mild or no symptoms, a small but significant proportion will suffer from a severe infection needing hospitalisation for supportive care, oxygen, or admission to intensive care units(ICU) for respiratory support. Early identification of hospitalised COVID-19 patients who are likely to deteriorate, i.e. transfer to ICU or who may die, is vital for clinical decision making. Healthcare systems across the world including highly developed countries continue to face challenges in terms of capacity and resources to manage this pandemic, as lock down measures have been relaxed, including opening of schools and businesses. Published prediction models to date have evaluated case level factors that might predict poor outcomes (critical illness or death). A recent living systematic review [17] identified 265 prognostic models for mortality and 84 for progression to severe or critical state. The majority of the studies looked at vital signs, age, comorbidities, and radiological features. Models were unlikely to include a broad range of variables concerning co-infection, biochemical factors (outside of C-reactive protein), and other haematological factors on an individual patient level. Most of the prognostic models did not describe the target population or care setting adequately, did not fully describe the regression equation, showed high or unclear risk of bias and/or were inadequately evaluated for performance. ### Goals The present study analyzes a range of laboratory blood marker values across metabolic pathways affected by COVID-19 infection (i.e. a core set of biomarkers feasible for clinical collection) and evaluates predictive models of severe outcomes. The main objectives of the study are: (1) Examine statistical associations of routinely measured physiological and blood biomarkers, and age and gender, to predict severe COVID-19 outcomes. (2) Develop cross-validated logistic regression prediction models using the best candidate biomarkers, and highlight biomarkers worthy of future research. (3) Use variable selection techniques including least absolute shrinkage and selection operator (LASSO) regularisation [18] and Bayesian Projective Prediction [19] to illustrate the process of creating a reduced model that maintains reasonable performance and is more feasible to use clinically (4) In each of these steps demonstrate best analytic practices for explaining clinical data curation and statistical modelling decisions, with an emphasis on showcasing the capabilities of recent Bayesian methods. ## Methods ### Study Cohort and Demographics Pseudonymised data was obtained from laboratory information management system (LIMS) linking patient data for laboratory markers to key clinical outcomes. Three hospitals in the Southwest region of England, UK, participated in the study, two of them were tertiary teaching hospitals and the third was a district general hospital (DGH). A system wide data search was conducted on LIMS for all patients who tested positive for SARS-CoV-2 by polymerase chain reaction (PCR) at these three hospitals during the first wave of COVID-19 pandemic (01/03/2020 to 31/10/2020). The serial pathology data collected as a part of standard of care of patients admitted with/for COVID-19 were included-bacteriology, virology, mycology, haematology, and biochemistry. All patients testing negative for SARS CoV 2 by PCR were excluded. All laboratory markers including clinical outcomes from LIMS were extracted and the final dataset was anonymized with no patient identifying data to link back. ### Inclusion and exclusion criteria We included all adult patients admitted to study hospitals and tested positive for SARSCoV-2 by PCR. Pediatric patients (<18 years old) and staff/healthcare workers and their household contacts were excluded. Figure 1 depicts the decision flow for inclusion and exclusion of patient data. ![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F1.medium.gif) [Figure 1:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F1) Figure 1: Flowchart of patient exclusion and inclusion criteria. The initial set of 1159 candidate patients was narrowed to a training set (n=590) and a validation set (n=253). ### Data Covariates The LabMarCS dataset includes a variety of host, clinical severity indices, microbiological, immunological, haematological and biochemistry parameters used as predictive variables in the regression models. A full list of recorded data items is shown in Figure 2 ![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F2.medium.gif) [Figure 2:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F2) Figure 2: Variables recorded in the LabMarCS dataset, including plain text description, abbreviation, place of record, frequency in the dataset, and criteria used for converting continuous readings into categorical values. ### Outcomes For all sites, the primary prediction outcome was death or transfer to the ICU within 28 days of admission to hospital, or the first positive COVID-19 PCR test result if already admitted. This generally corresponds to WHO-COVID-19 Outcomes Scale Score 6–10 (severe) versus 0–5 (mild/moderate) [20]. ### Patient Timelines The collected laboratory biomarkers are continuous measures and provide a time series representation of the course of a patient’s admission. Figure 3 shows an example of a single patient’s readings over the course of 18 days between testing positive for COVID-19 and being released from hospital care. This provides a representative example of the heterogeneity seen in our dataset, i.e. not all tests are taken and others are taken regularly or intermittently (further examples in Supplementary Figures 13 - 17). ![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F3.medium.gif) [Figure 3:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F3) Figure 3: Example a single patient’s time series laboratory biomarker data. See Figure 2 for biomarker abbreviations. Biomarker results are normalised to span 0 to 1 via offsetting by the absolute value of the minimum value and dividing by the maximum value. ### Transformation of Biomarker Data Prediction modelling of irregularly sampled time-series data is a challenging open research question [21]. In this study we focused on established and available tools for conventional and Bayesian prediction. To balance inclusion of test data not available on the day of admission and the need for clinical decisions to be guided soon after admission, we chose to consider the first value recorded for each biomarkers within three days of their ‘critical date’. In addition, we transformed continuous biomarkers into categorical variables via reference ranges for clinical use in the typical healthy population ranges, see Figure 2. As an example, Figure 4 shows the histogram of readings for all values recorded for Neutrophils, including clinical thresholds to transform into categorical data. No missing data imputation was performed, instead missingness was coded as as an additional category ‘Test not taken’. ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F4.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F4) Figure 4: Example distribution of biomarker readings for Neutrophil Training and Validation Data. Vertical lines indicate clinical thresholds for bounds on Normal, Mild, Moderate, and Severe categorization. For further elaboration on the challenges of these modelling choices, please see Discussion Sectionc. ## Statistical Analysis Analytics were carried out using the R statistical language (v4.13) and R Studio (Prairie Trillium release). We used the following packages: Standard logistic regression analyses used the R Stats GLM package (v3.6.2); LASSO analyses, GLMnet (v4.1-4); and for Bayesian analyses, BRMS (v2.17) and ProjPred (v2.1.2). Source code for this analysis pipeline can be found at [https://github.com/biospi/LABMARCS](https://github.com/biospi/LABMARCS). ### Analysis of Individual Biomarkers Before running full regression models we examined the independent contribution of individual biomarkers in predicting ICU entry or death via standard logistic regressions and Bayesian logistic regressions with either a flat (aka uniform) or horseshoe prior. This allowed calculation of p-values and odds ratios for each biomarker. A 5-fold cross-validation repeated 20 times was run for each biomarker to estimate the median AUC and 95% interquartile intervals. Each individual biomarker model includes age and gender (except univariate age and gender models) and was compared against a standard model including only age and gender. Regressions were fit using all associated dummy variables for a given biomarker (e.g. ‘Mild’, ‘Moderate’, ‘Severe’) using ‘Normal’ as the reference. Only complete cases of training data available for that marker were considered, i.e. we did not include data for variables marked ‘Test not taken’. ### Analysis Using All Valid Biomarker Data After individual biomarker evaluation, logistic regression models considering all valid biomarkers (Results Section c) and demographic variables were fit to the data. Their predictions were tested via internal and external validation using cross-validation procedures, additionally we fit models that used all available training data. The models include a standard logistic regression, a logistic regression regularised with LASSO, and two Bayesian models using a flat and a horseshoe prior [22]. LASSO and Bayesian horseshoe prior models (with projective prediction) and regularization constraints that push models to converge on sparse solutions with most coefficients near zero, and lend themselves to variable reduction as discussed in the Reduced Variable Models Section c. ### Analysis Using Reduced Variable Models While a model using all biomarker data may have strong predictive power, it is clinically desire-able to have a strong prediction with the least amount of biomarkers possible to save on resources devoted to biomarker collection. We used two methodologies to choose reduced variable models to predict COVID-19 severe outcomes, LASSO and Bayesian Projective Prediction. LASSO is an optimization constraint that shrinks parameters according to their variance, reduces over-fitting, and enables variable selection [18]. The optimal degree of regularisation is determined for each cross-by identifying a tuning parameter *λ* within a LASSO specific inner loop of each cross-validation step. LASSO has a drawback of having biased coefficient and log-odds estimates, as such after evaluating LASSO models we run a final ‘LASSO inspired’ standard GLM model. To evaluate LASSO coefficient estimates, we performed repeated nested cross-validation (5-folds the for the inner LASSO loop; 5-folds for the outer loop, and 20 repeats). For a particular dataset fit, LASSO optimises for a sparse representation with many coefficients close to zero. Across cross-validated trials these variables will vary. LASSO fits are statistically biased and are better suited as a guide for variable selection in a reduced variable standard GLM. As recommended in Heinze et al [23], we consider the frequency of how often a particular biomarker has a coefficient greater than zero and count across cross-validation trials. For a ‘LASSO inspired’ reduced variable standard GLM, it was chosen that if at least one categorical level for a particular biomarker (e.g. ‘Severe’) met this requirement, all levels for that biomarker were included in the model. This resulted in a final set of variables that could then fit with standard logistic GLM. The second variable selection method explored was Bayesian projective prediction [19], a technique for constructing an optimal reference model (in our case a Bayesian logistic regression with a horseshoe prior /citecarvalho2009handling over the distribution of coefficient values) that generates a ranking of individual variable informativeness via leave-one-out (LOO) cross-validation. This ranking of variables can be used to create a projection model where one can arbitrarily remove variables post-hoc. This approach allows one to evaluate the trade-off between AUC performance and the number of variables included in the model and use a reduced model projection at a desired AUC cutoff. Bayesian methods have the benefit of allowing co-efficient shrinkage via the horseshoe prior and provide unbiased odds estimates. Further projective prediction allows the flexibility to train one model on all valid available data, perform variable selection, and then use any projected sub-model with reduced variables to predict outcomes for novel data. ## Results ### Cohort Description The initial cohort included 1159 patients which was narrowed down to 843 patients who met all inclusion criteria described above, see Figure 1. 57% of patients were hospitalised for COVID-19 and the remainder had nosocomial infection. For our statistical models, the training cohort (n=590) was defined as all adults admitted to hospital and testing positive for SARS-Cov-2 by PCR, or testing positive while already admitted between March and October 2020. For external validation, we held the DGH cohort (n=253) out of training. Figure 5 depicts the distribution of ages and genders in the training and validation data sets. Patients in the training set had a mean age of 70, were 44% female, and 29% had severe outcomes. The validation set had a mean age of 75, were 47% female, and 38% had a severe outcome. ![Figure 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F5.medium.gif) [Figure 5:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F5) Figure 5: Distribution of age and gender for hospitalized patients with coronavirus disease 2019 (COVID-19) for (Top) training data (n=590) and (Bottom) hold out validation data (n=253) cohorts. ### Prediction Using Individual Variables Figure 6 shows descriptive statistics on individual biomarker readings and their odds ratio contributions in a 5-fold 20-repeat cross-validated logistic regression including the particular biomarker and age and gender. Figure 7 details performance using the area under the receiver operating characteristic curve (AUC) metric, comparing biomarker models (a particular biomarker plus age and gender) to a model using only age and gender. Due to the categorical representation of the biomarkers, individual levels may be significant while another is not (e.g. ‘Severe’ is a predictor, but ‘Mild’ is not). Statistically significant predictors (i.e. odds ratios deviating from one with p-value at 0.05 or lower) associated with increasing risk of a severe outcome (as shown in Figure 6) include age, and the biomarkers: Activated Partial Thromboplastin Time (Mild), Prothrombin time (Abnormal), blood pH (Abnormal), Haemoglobin (Severe), Platelet count (Moderate), Lymphocytes (Moderate, Severe), Neutrophils (Severe), Neutrophil-Lymphocyte Ratio (Mild, Moderate, Severe), C-Reactive Protein (Abnormal), Urea (Abnormal), and Troponin-T (Abnormal). Nosocomial transmission was included due to the high number of cases in our cohort but was not a significant predictor and excluded from further analyses. Due to small numbers preventing cross validation, Triglycerides, Glycated Haemoglobin, and Procalcitonin (also invalid due to being recorded only in ICU) were excluded from further analysis and require future research. ![Figure 6:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F6.medium.gif) [Figure 6:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F6) Figure 6: Descriptive statistics and logistic regression model outcomes (Standard, Bayesian with flat prior, and Bayes with horseshoe prior). All models included age and gender (except univariate age and gender models). Regressions were fit using all associated dummy variables for a given biomarker (e.g. normal, mild, moderate, severe) and using only complete cases of training data, i.e. not using a variable for ‘Test not taken.’ 95% inter-quantile ranges were calculated via 5-fold cross-validation with 20 repeats (100 models total). Categorical variables use a reading of ‘Normal’ as a reference in the fitted model, except ‘Male’ used as the reference category for gender. ![Figure 7:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F7.medium.gif) [Figure 7:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F7) Figure 7: Predictive performance of models in 7 as described by the median area under the curve (AUC) in receiver operating curve (ROC) analysis and median difference between an Age and Gender reference model and the same model with the particular biomarker included (except univariate age and gender models). Regressions were fit using all associated dummy variables for a given biomarker (e.g. mild, moderate, severe) and using only complete cases of training data (n=590), i.e. not using a variable for ‘Test not taken.’ 95% inter-quantile ranges calculated via 5-fold cross-validation with 20 repeats (100 models total). Categorical variables use a reading of ‘Normal’ as a reference in the fitted model, except ‘Male’ used as the reference category for gender. ### Regression Models Using All Valid Biomarker Data Each model was evaluated via 5-fold cross-validation with 20 repeats (100 models total). As such, each model is trained with a randomised sample of 80% of the training data set (n=472). Internal validation evaluates a model predictions on the 20% (n=118) held out. External validation uses the same model, but is instead tested on the held out validation data set (n=253). Missing data for each biomarker is coded as ‘Test Not Taken’ and is included as a predictor variable. Figure 8 shows the performance of these models (AUC, Sensitivity, Specificity). For comparison, Figure 9 shows the performance of each model using all valid training data (n=590) and testing on the same data (internal validation) and testing on the held out external validation data (n=253). ![Figure 8:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F8.medium.gif) [Figure 8:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F8) Figure 8: Cross-validated performance of models trained using valid biomarker data. 95% inter-quantile ranges are presented for each estimate. Specificity is obtained by evaluating at a set sensitivity of either 90% or 95%. All reduced variable models include age, and a stated number of biomarkers. The reduced variable LASSO inspired standard GLM uses 15 biomarkers that had non-zero coefficients on >=50% LASSO Cross-validation trials. If at least one categorical level for a particular biomarker (e.g. severe) met this requirement, all levels for that biomarker were included in the model. The 3 biomarker projective prediction model uses all categorical levels for Urea, PT, and NLR. ![Figure 9:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F9.medium.gif) [Figure 9:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F9) Figure 9: Performance of models using all valid biomarker data trained on all training data available (n=590). Internal validation is trained on all of the training data and tested on the same. External validation uses the same model and is tested on held out validation data set (n=253). Missing data for each biomarker is coded as ‘Test Not Taken’. Specificity and sensitivity evaluated using a probability threshold of 0.5 (i.e. assumes a well-calibrated model). All reduced variable models include age, and a stated number of biomarkers. The reduced variable LASSO inspired standard GLM uses 15 biomarkers that had non-zero coefficients on >=50% LASSO Cross-validation trials. If at least one categorical level for a particular biomarker (e.g. severe) met this requirement, all levels for that biomarker were included in the model. The 3 biomarker projective prediction model uses uses all categorical levels for Urea, PT, and NLR. Models trained on the full data have improved AUC scores, but do not provide a direct uncertainty estimate, this could be done via bootstrapping for a single model but we instead compute inter-quantile ranges across 5-fold 20 repeat cross-validation models. Cross-validation results provide 95% inter-quantile ranges that clearly illustrate that in general, all models perform similarly, with a median AUC in the mid 0.70’s in internal validation, and near the high 0.60’s in external validation. There is a trend for the models that encourage sparse representations, LASSO and Bayes with horseshoe prior, to have slightly higher AUC’s coupled with higher sensitivity and lower specificity. ### Reduced Variable Models The models detailed above are moderately good predictors of severe COVID-19 outcomes, but for clinicians with limited time and resources, reduced models can balance predictive performance with ease of clinical use by using only the most informative biomarkers. To address this, we use two variable selection approaches, LASSO and projective prediction, that allow the creation of reduced models with fewer biomarkers but similar performance to the larger models. ### LASSO Models After performing 5-fold 20 repeat cross-validation we examined the frequency of how often a particular biomarker has a coefficient greater than zero and count across cross-validation trials. Figure 10 shows the frequency of variables having a coefficient great than zero in the cross-validated LASSO analysis. If we select variables that appear at least 50% of the time, our reduced model would include: Age, CRP (abnormal), FER (mild), FIB (mild), HB (severe), PLT (mild, moderate, severe), Lymphocytes (Severe), Neutrophils (Mild, Severe), NLR (Severe), APTT (mild, moderate), PT (abnormal), blood pH (abnormal), Urea (abnormal), and positive viral and blood culture co-infections. ![Figure 10:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F10.medium.gif) [Figure 10:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F10) Figure 10: Frequency of LASSO logistic regression variables having a coefficient greater or less than 0. Red and black lines indicate thresholds for 20% and 50% frequency. For a ‘LASSO inspired’ reduced variable standard GLM, this resulted in a model using the 15 biomarkers above for all categorical levels, and was evaluated via both cross-validation and as fit to all available training data. This model had performance very similar to the models using all valid biomarker data, with a median external validation AUC of 0.68 [0.63, 0.72], see Figures 8 and 9. Note, ‘Test Not Taken’ is a significant predictor for LDH and Lactate on over 50% of cross-validation trials. The potential significance of missing data is complex and is addressed in the Discussion Sectionc. Due to this confounding, biomarkers whose top predictive contribution was from ‘Test Not Taken’ were excluded from both LASSO reduced variable models and projective prediction models described below. ### Projective Prediction Models When all biomarkers were considered, projective prediction identifies the following predictors in the top 20, in order of contribution to AUC: Urea (abnormal), Age, PT (abnormal), NLR (Severe), pH (abnormal), Lymphocytes (severe), APPT(mild), eGFR (abnormal), Neutrophils (Severe), APPT(moderate), CRP (abnormal), DDM (abnormal), Hemoglobin (severe). Thus age and 12 biomarkers are candidates for a reduced model. Note, several predictors of ‘Test Not Taken’ were also selected including Lactate, O2, CO2, LDH, Ferritin and Fibrinogen. As mentioned above, these biomarkers are set aside due to this confounding. Supplementary Figure 11 displays the output from projective prediction ranking the contribution of each variable to the model. A model using a projection incorporating all biomarker and demographic data is equivalent to the standard Bayesian GLM we evaluated in the prior section, see Figures 8 and 9. Reduced variable projections were evaluated by manual inspection of AUC performance among groups of models using the top biomarkers. Guided by the projective prediction ranking, we ran a model using only the top biomarker, using only the top two, the top three, and so on. As described above we omit biomarkers with significant contributions from ‘Test Not Taken’ and include all categorical levels for a given biomarker as long as one level is highly ranked. Ultimately, we found a 3 biomarker projective prediction model using age and including urea, prothrombin time, neutrophil-lymphocyte ratios had similar performance to larger models with a median internal validation AUC of 0.74 [0.67, 0.82], and external validation AUC of 0.70 [0.69, 0.71], as shown in Figures 8 and 9. ## Discussion ### Challenges of Complex Medical Data Curating the LabMarCS data is challenging as the data are heterogeneous in multiple ways. Biomarkers are recorded for different reasons, e.g. routine upon admission, investigatory tests, or tests primarily or exclusively taken in ICU. Further some biomarkers are typically recorded together (but not always) as part of a test suite, including: Urea and electrolytes, full blood count, COVID-19 and co-infection swab test, blood clotting, and blood gas tests (arterial or venous). The schedule when some these markers are recorded vary by patient and clinical decision, leading to records being present in highly varying amounts, e.g. only 3% up to 100% of patients depending on the particular biomarker, see Supplementary Figure 12. ### Modelling Choices When constructing and evaluating models, there are many choice points that should be explicitly highlighted with justification, be it based on convenience, computational complexity, clinical advice, or a heuristic. The space of potential models is vast and most studies will constrain the model search space, delineating why these choices are made will facilitate understanding and reproduction by other researchers. These include key choices relating to: patient inclusion/exclusion criteria, data missingness protocols, data transformations, training and validation data selection, and performance evaluation. ### Missing Data Missingness, in the context of this study and in healthcare data more generally, can sometimes be informative and missing not at random (MNAR), with the presence or absence of a test correlated with the measurement of said test. Imputation of missing data relies on key statistical assumptions that imputed variables are missing at random (MAR) or missing completely at random (MCAR), else the imputation will be faulty and models may be fit to non-representative data. Conversations with our clinical colleagues established some routinely collected biomarkers might be inferred to be MAR. However, the routines identified were specific to a small a subset of our cohort and not likely to extrapolate. We ultimately erred to be conservative and avoid all imputation, and instead include missing values as a data point [24, 25]. As such, in the current study we chose to use placeholders for ‘Test not taken’ if there was no recorded value for a particular biomarker within the evaluated 3-day window. This approach however, allows the possibility that a ‘Test Not Taken’ may be a significant predictor. This has many potential meanings, as it may convey that when a patient is doing well and unlikely to experience a severe outcome, clinicians are unlikely to request some biomarker tests. Alternatively, if a patient is in palliative care and has a poor prognosis, a clinician may consider further testing unnecessary. As such, the likelihood of a test being administered may follow an inverted-U function as patients to healthy or too ill may not have tests administer. Furthermore, as our data was collected early in the pandemic, there may be other underlying clinical decisions or resource limitations that drove why some tests were taken but not others. Lastly, because we only consider results from the first 3 days from a patients critical date, it may be that some tests are simply taken later in a patient’s stay, and hence may be more predictive as they were taken closer to the outcome. Hence, when these instances occurred, we were conservative and excluded biomarkers with ‘Test Not Taken’ as the most informative category from our reduced variable models. ### Data Transforms - Time Windows Ideally clinicians can make a decision based on readings the day of admission. However, not all tests are administered on admission. To balance inclusion of test data not available on the day of admission and the need for clinical decisions to be guided soon after admission, we chose to consider the first value recorded for each biomarkers within three days of their ‘critical date’, i.e. date of admission if already COVID-19 positive, or if already in hospital, the date of testing COVID-19 positive. However, given the richness of the time series data collected, further research into models that leverage this extra information is needed. Focusing on early detection reflects the intent for the model to improve early stage clinical decision making when potential treatments or changes in care may be introduced. This focus on the first reading in a 3-day interval loses information, but greatly simplifies the modelling approach. Note, this choice is not without risk of reducing statistical power, increasing the risk of false positives, and underestimation of the extent of variation in biomarker readings and outcomes between groups [26]. It is likely that representing biomarker data as time series (assuming regular measures across patients) instead of single points would add considerable information. ### Data Transforms - Continuous vs. Categorical A key modelling decision must be made on whether to use continuous data or transformed categorical data. Clinicians often use biomarker thresholds to provide semantic categories (e.g. normal, mild, moderate, severe) which sometimes use non-linear or discontinuous mappings that require special care if using continuous data. While clinical thresholds are likely established with evidence, it may be the case that thresholds for one use may not apply to a novel one. This led [27, 28] to use machine learning approaches to build categorisation models on continuous biomarker data dependent on the training data at hand. However, using machine learning to establish categorisation thresholds on our biomarker data is difficult with a small training data set and the heterogeneity of biomarker recordings. If missing data imputation is done, it raises another decision point on whether to impute the continuous or the transformed categorical data. Another important factor to recognise is that some biomarkers lack a linear relationship between a reading and a semantic category. Biomarkers can have a lower and upper bound for what is considered normal, and both below and above this range reflects clinically meaningful yet sometimes separate abnormalities. This means modelling needs to factor in non-linear curves if persevering continuous data or trying to map to a categorical space. In our position, categorical transforms had the advantage as we were able to collaborate with ICU consultants in conjunction with using pre-established clinically acceptable ranges defined our categorisation, see Figure 2. ### Training and Validation Data Selection There are multiple ways that our data set could be split between training and validation sets, e.g. randomly sampling 1/3 of the data to hold out as a validation set. Given our rather small sample, random selection of training data should in principle generate data more representative of the validation set left out. However, realistically hospitals may have differing practices and randomization of may inflate performance at the cost of real world validity. We chose to separate our training and validation datasets by hospital to provide a stronger test of generalisation that should mimic generalisation to novel hospitals completely outside the original training data. ### Model Performance Evaluation and Dissemination There are a variety of ways statistical model performance can be evaluated. Here we have chose here to emphasize cross-validated estimates of AUC, sensitivity, and specificity. Interquartile intervals over these measures reveal that the variety of models perform in similar ways. While the full models have higher median performance, the reduced models are within the 95% bounds of the other models. With a larger data set trade-offs may become more apparent. ### Advantages of Bayesian Modelling While the predictive performance across models presented here is generally quite similar, there are several reasons for researchers to favor Bayesian approaches. The coefficients estimated via Bayes should on average deliver slightly better predictive performance. Additionally, if a sparse model is needed, a horseshoe prior can provide advantages similar to LASSO without biased coefficient estimates. Computationally, Bayesian techniques can be slow due Markov Chain Monte Carlo used to sample the coefficient space. If one is interested in variable selection, projective prediction offers the ability to take a single Bayesian model fit, run a variable selection algorithm to rank variable contributions, and then arbitrarily create submodel projections with any number of original variables. While the initial model fit and variable selection are computationally intensive, sub-model projections are fast to create and performance test. ## Summary & Conclusions ### Limitations This is a retrospective cohort study involving a relatively small cohort in Southwest England where case numbers have varied widely, and were well below national figures during the first wave. This results in less precise parameter estimates for prediction models (less power/smaller sample size) and likely reduced generalizability of the model to other settings. The timing of biomarker collection was highly varied both within and between patients, with many types of readings missing. While we replicated prior findings on several biomarkers, gender was not significant, suggesting our sample may not be representative. ### Strengths The primary strength of our study is the granularity of serial laboratory data available linked to clinical outcomes. This study was performed during the first wave where there was the original Wuhan strain circulating amongst the unvaccinated naïve population without any specific immunomodulating therapies such as steroids or antiviral agents, reflecting the “true” homeostasis derangements at a population level. This study highlights a variety of challenges present in complex medical data sets while maintaining best statistical practices with an emphasis on recent Bayesian methodology. Our study reiterates the predictive value of previously identified biomarkers for COVID-19 severity assessment (e.g. age, urea, prothrombin time, and neutrophil-lymphocyte ratio). Both the full and reduced variable models have moderately good training performance, but improved external validation is needed for all models to be clinically viable. The methods presented here should generalise well to a larger dataset. ## Data Availability Due to NHS data governance, all data produced in the present study are unavailable directly through the authors, but reasonable requests for data to Southwest England NHS can be arranged via the authors. ## Ethics approval The study [IRAS project ID: 283439] underwent a rigorous ethical and regulatory approval process, and a favourable opinion was gained from Research Ethics Service, Wales REC 7, c/o Public Health Wales, Building 1, Jobswell Road, St David’s Park, SA31 3HB on 11/09/2020. ## Funding This work is funded by Health Data Research UK via the Better Care Partnership Southwest (HDR CF0129), Medical Research Council Research Grant MR/T005408/1, and the Elizabeth Blackwell Institute for Health Research, University of Bristol and the Wellcome Trust Institutional Strategic Support Fund. ## Declaration of competing interest The authors have no competing interests. ## Supplementary materials ![Figure 11:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F11.medium.gif) [Figure 11:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F11) Figure 11: Summary statistics of Bayesian projective prediction ranking the contribution of each variable by change in AUC and expected log-predictive density (ELPD) ![Figure 12:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F12.medium.gif) [Figure 12:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F12) Figure 12: Heat map displaying missing values across recorded biomarkers. Light blue indicates a value is missing and dark blue indicate it is present ![Figure 13:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F13.medium.gif) [Figure 13:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F13) Figure 13: Example biomarker time series for a patient admitted to hospital COVID-19 positive and who subsequently died almost two weeks later. ![Figure 14:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F14.medium.gif) [Figure 14:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F14) Figure 14: Example biomarker time series for a patient admitted to hospital with subsequent nosocomial transmission and discharge a week later. ![Figure 15:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F15.medium.gif) [Figure 15:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F15) Figure 15: Example biomarker time series for a patient admitted to hospital COVID-19 positive, with subsequent entrance to ICU and death over one month later. ![Figure 16:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F16.medium.gif) [Figure 16:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F16) Figure 16: Example biomarker time series for a patient admitted to hospital and ICU, with subsequent nosocomial transmission and discharge about one week later. ![Figure 17:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F17.medium.gif) [Figure 17:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F17) Figure 17: Example biomarker time series for a patient with two hospital admissions and testing COVID-19 positive on the first, with discharge almost two weeks after second admission. ![Figure 18:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F18.medium.gif) [Figure 18:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F18) Figure 18: Distribution of D-Dimer readings with clinical classification requiring age and gender bands ![Figure 19:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/09/17/2022.09.16.22279985/F19.medium.gif) [Figure 19:](http://medrxiv.org/content/early/2022/09/17/2022.09.16.22279985/F19) Figure 19: Standard logistic regression odds ratio and confidence intervals per biomarker using all valid biomarker training data available (n=590). Note most biomarkers include a ‘Test Not Taken’ stand in variable. ## Acknowledgements This research was supported by the National Institute for Health and Care Research (NIHR) Applied Research Collaboration West (NIHR ARC West). The views expressed in this article are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. * Received September 16, 2022. * Revision received September 16, 2022. * Accepted September 17, 2022. * © 2022, Posted by Cold Spring Harbor Laboratory The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission. ## References 1. [1]. A. B. Docherty, E. M. Harrison, C. A. Green, H. Hardwick, R. Pius, L. Norman, K. A. Holden, J. M. Read, F. Dondelinger, G. Carson, L. Merson, J. Lee, D. Plotkin, L. Sigfrid, S. Halpin, C. Jackson, C. Gamble, P. W. Horby, J. S. Nguyen-Van-Tam, I. Investigators, J. Dunning, P. J. M. Openshaw, J. K. Baillie, M. G. Semple, Features of 16,749 hospitalised UK patients with COVID-19 using the ISARIC WHO Clinical Characterisation Protocol, medRxiv (2020) 2020.04.23.20076042 doi: 10.1101/2020.04.23.20076042. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4wNC4yMy4yMDA3NjA0MnYxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDkvMTcvMjAyMi4wOS4xNi4yMjI3OTk4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 2. [2]. C. Wu, X. Chen, Y. Cai, J. Xia, X. Zhou, S. Xu, H. Huang, L. Zhang, X. Zhou, C. Du, Y. Zhang, J. Song, S. Wang, Y. Chao, Z. Yang, J. Xu, X. Zhou, D. Chen, W. Xiong, L. Xu, F. Zhou, J. Jiang, C. Bai, J. Zheng, Y. Song, Risk Factors Associated With Acute Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease 2019 Pneumonia in Wuhan, China, JAMA internal medicine 180 (7) (2020) 934–943. doi:10.1001/jamainternmed.2020.0994. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jamainternmed.2020.0994&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32167524&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 3. [3]. L. Bowles, S. Platton, N. Yartey, M. Dave, K. Lee, D. P. Hart, V. Mac-Donald, L. Green, S. Sivapalaratnam, K. J. Pasi, P. MacCallum, Lupus Anticoagulant and Abnormal Coagulation Tests in Patients with Covid-19, New England Journal of Medicine 383 (3) (2020) 288–290. doi: 10.1056/NEJMc2013656. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMc2013656&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32369280&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 4. [4]. N. Tang, D. Li, X. Wang, Z. Sun, Abnormal coagulation parameters are associated with poor prognosis in patients with novel coronavirus pneumonia, Journal of thrombosis and haemostasis: JTH 18 (4) (2020) 844–847. doi:10.1111/jth.14768. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/jth.14768&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 5. [5]. H. Han, L. Yang, R. Liu, F. Liu, K.-L. Wu, J. Li, X.-H. Liu, C.-L. Zhu, Prominent changes in blood coagulation of patients with SARS-CoV-2 infection, Clinical Chemistry and Laboratory Medicine 58 (7) (2020) 1116–1120. doi:10.1515/cclm-2020-0188. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1515/cclm-2020-0188&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32172226&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 6. [6]. X. Bi, Z. Su, H. Yan, J. Du, J. Wang, L. Chen, M. Peng, S. Chen, B. Shen, J. Li, Prediction of severe illness due to COVID-19 based on an analysis of initial Fibrinogen to Albumin Ratio and Platelet count, Platelets 31 (5) (2020) 674–679. doi:10.1080/09537104.2020.1760230. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/09537104.2020.1760230&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32367765&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 7. [7]. F. Liu, L. Li, M. Xu, J. Wu, D. Luo, Y. Zhu, B. Li, X. Song, X. Zhou, Prognostic value of interleukin-6, C-reactive protein, and procalcitonin in patients with COVID-19, Journal of Clinical Virology: The Official Publication of the Pan American Society for Clinical Virology 127 (2020) 104370. doi:10.1016/j.jcv.2020.104370. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jcv.2020.104370&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 8. [8]. G. Vaseghi, M. Mansourian, R. Karimi, K. Heshmat-Ghahdarijani, P. Rouhi, M. Shariati, S. H. Javanmard, Inflammatory markers in Covid-19 Patients: A systematic review and meta-analysis, medRxiv (2020) 2020.04.29.20084863 doi:10.1101/2020.04.29.20084863. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4wNC4yOS4yMDA4NDg2M3YxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDkvMTcvMjAyMi4wOS4xNi4yMjI3OTk4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 9. [9]. Q. Ruan, K. Yang, W. Wang, L. Jiang, J. Song, Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China, Intensive Care Medicine 46 (5) (2020) 846–848. doi:10.1007/s00134-020-05991-x. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00134-020-05991-x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32125452&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 10. [10]. B. E. Young, S. W. X. Ong, S. Kalimuddin, J. G. Low, S. Y. Tan, J. Loh, O.-T. Ng, K. Marimuthu, L. W. Ang, T. M. Mak, S. K. Lau, D. E. Anderson, K. S. Chan, T. Y. Tan, T. Y. Ng, L. Cui, Z. Said, L. Kurupatham, M. I.-C. Chen, M. Chan, S. Vasoo, L.-F. Wang, B. H. Tan, R. T. P. Lin, V. J. M. Lee, Y.-S. Leo, D. C. Lye, Singapore 2019 Novel Coronavirus Outbreak Research Team, Epidemiologic Features and Clinical Course of Patients Infected With SARS-CoV-2 in Singapore, JAMA 323 (15) (2020) 1488–1494. doi:10.1001/jama.2020.3204. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2020.3204&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32125362&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 11. [11]. J. Liu, Y. Liu, P. Xiang, L. Pu, H. Xiong, C. Li, M. Zhang, J. Tan, Y. Xu, R. Song, M. Song, L. Wang, W. Zhang, B. Han, L. Yang, X. Wang, G. Zhou, T. Zhang, B. Li, Y. Wang, Z. Chen, X. Wang, Neutrophil-to-Lymphocyte Ratio Predicts Severe Illness Patients with 2019 Novel Coro-navirus in the Early Stage, medRxiv (2020) 2020.02.10.20021584 doi: 10.1101/2020.02.10.20021584. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4wMi4xMC4yMDAyMTU4NHYxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDkvMTcvMjAyMi4wOS4xNi4yMjI3OTk4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 12. [12]. L. E. Gralinski, A. Bankhead, S. Jeng, V. D. Menachery, S. Proll, S. E. Belisle, M. Matzke, B.-J. M. Webb-Robertson, M. L. Luna, A. K. Shukla, M. T. Ferris, M. Bolles, J. Chang, L. Aicher, K. M. Waters, R. D. Smith, T. O. Metz, G. L. Law, M. G. Katze, S. McWeeney, R. S. Baric, Mechanisms of severe acute respiratory syndrome coronavirus-induced acute lung injury, mBio 4 (4) (Aug. 2013). doi:10.1128/mBio.00271-13. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoibWJpbyI7czo1OiJyZXNpZCI7czoxMzoiNC80L2UwMDI3MS0xMyI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzA5LzE3LzIwMjIuMDkuMTYuMjIyNzk5ODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 13. [13]. Z. Xu, L. Shi, Y. Wang, J. Zhang, L. Huang, C. Zhang, S. Liu, P. Zhao, H. Liu, L. Zhu, Y. Tai, C. Bai, T. Gao, J. Song, P. Xia, J. Dong, J. Zhao, F.-S. Wang, Pathological findings of COVID-19 associated with acute respiratory distress syndrome, The Lancet Respiratory Medicine 8 (4) (2020) 420–422. doi:10.1016/S2213-2600(20)30076-X. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S2213-2600(20)30076-X&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32085846&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 14. [14]. M. Arentz, E. Yim, L. Klaff, S. Lokhandwala, F. X. Riedo, M. Chong, M. Lee, Characteristics and Outcomes of 21 Critically Ill Patients With COVID-19 in Washington State, JAMA 323 (16) (2020) 1612–1614. doi:10.1001/jama.2020.4326. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2020.4326&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32191259&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 15. [15]. N. Wilson, A. Kvalsvig, L. T. Barnard, M. G. Baker, Case-Fatality Risk Estimates for COVID-19 Calculated by Using a Lag Time for Fatality - Volume 26, Number 6—June 2020 - Emerging Infectious Diseases journal - CDC, Emerging Infectious Diseases (2020). doi:10.3201/eid2606.200320. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3201/eid2606.200320&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 16. [16]. H. Barrasa, J. Rello, S. Tejada, A. Martin, G. Balziskueta, C. Vinuesa, B. Fernandez-Miret, A. Villagra, A. Vallejo, A. San Sebastian, S. Cabanes, S. Iribarren, F. Fonseca, J. Maynar, Alava COVID-19 Study Investigators, SARS-CoV-2 in Spanish Intensive Care Units: Early experience with 15-day survival in Vitoria, Anaesthesia, Critical Care & Pain Medicine 39 (5) (2020) 553–561. doi:10.1016/j.accpm.2020.04.001. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.accpm.2020.04.001&link_type=DOI) 17. [17]. L. Wynants, B. V. Calster, G. S. Collins, R. D. Riley, G. Heinze, E. Schuit, M. M. J. Bonten, D. L. Dahly, J. A. Damen, T. P. A. Debray, V. M. T. de Jong, M. D. Vos, P. Dhiman, M. C. Haller, M. O. Harhay, L. Henckaerts, P. Heus, M. Kammer, N. Kreuzberger, A. Lohmann, K. Luijken, J. Ma, G. P. Martin, D. J. McLernon, C. L. A. Navarro, J. B. Reitsma, J. C. Sergeant, C. Shi, N. Skoetz, L. J. M. Smits, K. I. E. Snell, M. Sperrin, R. Spijker, E. W. Steyerberg, T. Takada, I. Tzoulaki, S. M. J. van Kuijk, B. C. T. van Bussel, I. C. C. van der Horst, F. S. van Royen, J. Y. Verbakel, C. Wallisch, J. Wilkinson, R. Wolff, L. Hooft, K. G. M. Moons, M. van Smeden, Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal, BMJ 369 (2020) m1328. doi:10.1136/bmj.m1328. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjkvYXByMDdfMi9tMTMyOCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzA5LzE3LzIwMjIuMDkuMTYuMjIyNzk5ODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 18. [18]. R. Tibshirani, Regression Shrinkage and Selection Via the Lasso, Journal of the Royal Statistical Society: Series B (Methodological) 58 (1) (1996) 267–288. doi:10.1111/j.2517-6161.1996.tb02080.x. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.2517-6161.1996.tb02080.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=WOS:A1996TU3&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1996TU31400017&link_type=ISI) 19. [19]. J. Piironen, M. Paasiniemi, A. Vehtari, Projective inference in highdimensional problems: Prediction and feature selection, Electronic Journal of Statistics 14 (1) (2020) 2155–2197. doi:10.1214/20-EJS1711. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1214/20-EJS1711&link_type=DOI) 20. [20]. J. C. Marshall, S. Murthy, J. Diaz, N. Adhikari, D. C. Angus, Y. M. Arabi, K. Baillie, M. Bauer, S. Berry, B. Blackwood, et al., A minimal common outcome measure set for covid-19 clinical research, The Lancet Infectious Diseases 20 (8) (2020) e192–e197. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1473-3099(20)30483-7&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32539990&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F09%2F17%2F2022.09.16.22279985.atom) 21. [21]. M. van der Schaar Lab, Time series in healthcare: challenges and solutions, [https://www.vanderschaar-lab.com/time-series-in-healthcare/](https://www.vanderschaar-lab.com/time-series-in-healthcare/) (2022). 22. [22]. C. M. Carvalho, N. G. Polson, J. G. Scott, Handling sparsity via the horse-shoe, in: Artificial Intelligence and Statistics, PMLR, 2009, pp. 73–80. 23. [23].Variable selection – A review and recommendations for the practicing statistician - Heinze - 2018 - Biometrical Journal - Wiley Online Library, [https://onlinelibrary.wiley.com/doi/full/10.1002/bimj.201700067](https://onlinelibrary.wiley.com/doi/full/10.1002/bimj.201700067) (2018). 24. [24]. R. H. Groenwold, Informative missingness in electronic health record systems: the curse of knowing, Diagnostic and prognostic research 4 (1) (2020) 1–6. 25. [25]. S. Van Buuren, Flexible imputation of missing data, CRC press, 2018. 26. [26]. D. G. Altman, P. Royston, The cost of dichotomising continuous variables, BMJ : British Medical Journal 332 (7549) (2006) 1080. 27. [27]. S. R. Knight, A. Ho, R. Pius, I. Buchan, G. Carson, T. M. Drake, J. Dunning, C. J. Fairfield, C. Gamble, C. A. Green, et al., Risk stratification of patients admitted to hospital with covid-19 using the isaric who clinical characterisation protocol: development and validation of the 4c mortality score, bmj 370 (2020). 28. [28]. J. Zhou, S. Lee, X. Wang, Y. Li, W. K. K. Wu, T. Liu, Z. Cao, D. D. Zeng, K. S. K. Leung, A. K. C. Wai, et al., Development of a multivariable prediction model for severe covid-19 disease: a population-based study from hong kong, NPJ digital medicine 4 (1) (2021) 1–9.