Severe COVID-19 is characterised by inflammation and immature myeloid cells early in disease progression ======================================================================================================== * Liam Townsend * Adam H Dyer * Aifric Naughton * Sultan Imangaliyev * Jean Dunne * Rachel Kiersey * Dean Holden * Aoife Mooney * Deirdre Leavy * Katie Ridge * Jamie Sugrue * Mubarak Aldoseri * Jo Hannah Kelliher * Martina Hennessy * Declan Byrne * Paul Browne * Christopher L Bacon * Catriona Doyle * Ruth O’Riordan * Anne-Marie McLaughlin * Ciaran Bannan * Ignacio Martin-Loeches * Arthur White * Rachel M McLoughlin * Colm Bergin * Nollaig M Bourke * Cliona O’Farrelly * Niall Conlon * Clíona Ní Cheallaigh ## Summary SARS-CoV-2 infection causes a wide spectrum of disease severity. Immune changes associated with severe disease include pro-inflammatory cytokine production and expansion of immature myeloid populations. The relative importance of the immunological changes in driving progression to severe disease remain poorly understood. We aimed to identify and rank clinical and immunological features associated with progression to severe COVID-19. We sought to use tests available in an on-site diagnostic hospital laboratory to identify an immunological signature for severe disease development which could be detected prior to peak severity thereby allowing initiation of therapeutic interventions. We used univariate and multivariate analysis, including unbiased machine learning, to investigate the relationships between clinical and demographic characteristics, inflammatory markers, and leukocyte immunophenotypes with progression to severe disease in 108 patients and to rank these in importance. A combination of four features (elevated levels of interleukin-6 and C-reactive protein, coupled with reduced monocyte HLA-DR expression and reduced neutrophil CD10 expression), were strongly predictive of severe disease with an average prediction score of 0.925. **Highlights** * Severe COVID-19 can be predicted by a combination of emergency myelopoiesis (CD10-neutrophils and HLA DR-monocytes) and inflammation (raised IL-6 and CRP) * These changes can be identified from tests carried out prior to peak illness severity in a diagnostic laboratory * This predictive model was derived from a cohort of patients with a wide range of ages, frailty and COVID-19 severity ![Figure1](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/09/03/2021.09.01.21262953/F1.medium.gif) [Figure1](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/F1) Keywords * COVID-19 * immune phenotype * neutrophil maturity * machine learning * biomarkers ## Introduction COVID-19, caused by infection with the SARS-CoV-2 virus, is responsible for the current global pandemic.1 The significant morbidity and mortality associated with this infection has placed unprecedented pressures on healthcare systems worldwide.2 The clinical course of SARS-CoV-2 infection is remarkably variable. COVID-19 appears to have a biphasic pattern of illness.3-5 The initial phase, due to viral replication and the initial immune response, is characterised by elevation in inflammatory cytokines and influx of monocytes and T lymphocytes into the lungs.6-10 An immune-mediated phase develops in approximately 20% of individuals approximately 6-10 days post symptom onset.11 This phase is characterised by further elevation of inflammatory markers, increase in D-dimer levels, shortness of breath, and lung infiltrates.12,13 In approximately 5% of cases, progressive difficulty in breathing occurs, resulting in ARDS and requirement for mechanical ventilation.14 Multiple recent studies have described diverse immune mechanisms underlying COVID-19 pathogenesis. Most have looked at individual associations between immune variables and disease severity and identified alterations in cytokine levels and within immune cell populations.15-20 However, the pathophysiology of COVID-19 results from complex interactions across multiple parameters. Looking at individual associations between immune variables and disease in isolation using simple univariate statistical hypothesis testing cannot assess such interactions. Integration of multiple immunological and clinical parameters using multivariate testing is required.21 In addition, interventions carried out in the care of patients with severe COVID-19 including intubation and mechanical ventilation are also likely to cause alterations in immune parameters, and it is therefore important to study immune mechanisms early in the course of disease. Recent reports have highlighted severity-associated changes in myeloid and lymphocyte subsets but the relative importance of such changes in relation to other important demographic and clinical risk factors that can influence trajectories of disease progression is unknown.19, 20 Finally, the rapid and early identification of those who will develop severe disease is essential for the prompt administration of therapies. Dexamethasone has been shown to reduce mortality in patients requiring oxygen therapy.22 However, administering this in a timely fashion has proven difficult.23 There is an unmet need in for accurate immunological predictors of severe disease which can be used in a clinical setting. In this study, we describe early clinical and immunological features in one hundred and eight patients with COVID-19, of whom thirty-nine progressed to severe disease. All results were generated on the same day as the sample was obtained. We objectively ranked these features as independent predictors for COVID-19 severity and confirmed the important role for emergency myelopoiesis as demonstrated by increased proportions of immature neutrophils and HLA-DR-monocytes in combination with elevated levels of IL-6 and CRP, acting as proof of concept that these markers can be reliably and rapidly used in routine clinical practice to identify patients who will progress to severe disease and allow for the timely administration of therapeutics. ## Results ### Participant characteristics One-hundred and eight participants were recruited (42/108 (38.9%) female). Baseline characteristics are shown in **Table 1**. Seventeen had mild disease, 52 had moderate disease (of whom 30 required supplemental oxygen) and 39 had severe disease (**Table 2**). Of those with severe disease, 26/39 were admitted to ICU for intensive monitoring and mechanical ventilation and an additional 13/39 patients were assessed for ICU admission for mechanical ventilation but were not admitted to ICU due to high likelihood of non-survival. Ten patients with severe COVID-19 died (2/26 admitted to ICU and 8/13 assessed but not admitted to ICU). The median inpatient stay for those with moderate/severe disease was 11.5 days (range 2 – 108). The median length of stay in ICU was 12 days (range 1 – 39). Disease characteristics are shown in **Table 2**, while baseline laboratory parameters are shown in **Table 3**. As shown in **Supplemental Table 2**, clinical blood tests were conducted at a median of six days post symptom onset and, for those requiring supplemental oxygen, a median of 1.5 days prior to peak oxygen requirement, which was deemed to be a marker of peak respiratory illness. View this table: [Table 1:](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/T1) Table 1: Baseline Demographics View this table: [Table 2:](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/T2) Table 2: Disease Characteristics View this table: [Table 3:](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/T3) Table 3: Laboratory Parameters ### Severe SARS-CoV-2 infection is associated with evidence of profound immune dysregulation Analysis of routine laboratory tests revealed widespread changes in leukocyte populations with a marked coagulopathy and increased markers of cell turnover and inflammation (**Figure 1**). The degree of lymphopenia significantly increased with disease severity, while marked neutrophilia was seen particularly in those with severe disease. Anaemia with a left shift, characterised by increased red cell distribution width, was also noted with severe disease. Greater disease severity was associated with greater D-dimer and fibrinogen levels, as well as. markers of cellular inflammation (CRP, IL-6, ferritin) and cell turnover (LDH) increased with disease severity. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/09/03/2021.09.01.21262953/F2.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/F2) Figure 1. Severity of acute COVID-19 and Markers of Inflammation, Cell Turnover and Coagulation. Severe COVID-19 is accompanied by **(A)** leukocytosis **(B)** lymphopenia **(C)** neutrophil and **(D)** increased neutrophil: lymphocyte ratio in disease (N = 108 total). Severe COVID-19 is also associated with **(E)** lower haemoglobin **(F)** greater red cell distribution width **(G)** increased D-dimer **(H)** increased fibrinogen. Severe COVID-19 was associated with increasing **(I)** LDH **(J)** Ferritin **(K)** CRP **(L)** IL-6, **(M)** lower albumin **(N)** increased AST **(O)** increased ALT. No change in **(P)** creatinine with severity. * p <0.05, ** p <0.01, \***|p<0.001, NS = Not Significant In order to further delineate the panlymphopenia seen on the full blood count, detailed immunophenotyping of lymphoid cells was conducted (**Figure 2**). This confirmed panlymphopenia, with total reduction in total CD3+ cells across all disease severities, most marked in those with severe illness. There was reduction in both CD4+ and CD8+ T cell counts in a similar pattern. On further subset analysis, there was global reduction of naïve CD4+ and CD8+ cells and increased activated CD4+ and CD8+ cells, as well as increased effector CD8+ cells across all disease states. The effect of disease severity was less marked in these subsets, with similar numbers of activated and effector cells across all patients, and similar naïve cell counts in both moderate and severe disease. ![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/09/03/2021.09.01.21262953/F3.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/F3) Figure 2. Analysis of Major Lymphoid Subsets Reveals a Widespread Lymphopenia in COVID-19 Disease. Peripheral blood immunophenotyping in those with mild, moderate and severe coronavirus disease revealed significant decreases in **(A)** CD45 positive cells (leukocytes), most pronounced in moderate/severe disease; significant decreases in **(B)** CD3, **(C)** CD4 and **(D)** CD8 cell counts, greatest in those with moderate/severe disease in comparison to controls; **(E)** naïve CD4 and **(F)** CD8 cells both significantly decreased in COVID-19 with increasing disease severity; whilst **(G)** effector CD8 cells were non-significantly elevated, there was a significant expansion in **(H)** activated CD4+ and **(I)** CD8+ T cells, which did not reflect disease severity. Both **(J)** B cells and **(K)** Natural Killer cells were significantly decreased in number in COVID-19. Decreases in Natural Killer cell number reflected disease severity. * p <0.05, ** p <0.01, \***|p<0.001, NS = Non-Significant Detailed immunophenotyping of myeloid populations was also carried out, given the profound neutrophilia noted on initial investigations (**Figure 3**). Marked severity-associated neutrophilia was confirmed. There was loss of CD10 and CD16 expression by neutrophils with increasing disease severity. This finding was confirmed in two ways, looking at both CD10 and CD16 expression on neutrophils as well as the median frequency intensity of both of these markers. In addition to these neutrophil changes, perturbations amongst monocyte populations were found. While absolute number of monocytes was not associated with infection, there was progressive loss of HLA-DR expression by monocytes with increasing disease severity. This was again confirmed by two methods, in a similar manner to neutrophils. There was an increase in proportion of intermediate monocytes in SARS-CoV-2 patients when compared to healthy controls. This increase was most marked in those with mild disease. While there were no significant differences in non-classical monocyte proportions between infected and healthy patients, there were differences across disease severities, with milder patients having a higher proportion when compared to moderate and severe patients. The classical monocyte proportions remained unchanged when compared to healthy controls and when assessed across disease severities. ![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/09/03/2021.09.01.21262953/F4.medium.gif) [Figure 3.](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/F4) Figure 3. Analysis of Myeloid Cells Reveals a Widespread Immune Dysregulation in COVID-19 Disease. Peripheral blood immunophenotyping in those with mild, moderate and severe coronavirus disease revealed significant changes in neutrophil markers with **(A)** neutrophilia, **(B, E)** reduced neutrophil CD10 expression and **(C)** reduced neutrophil CD16 expression as well **(D, F)** reduced Mean Fluorescence Index (MFI) of these markers, most pronounced in those with severe COVID-19 disease. **(G)** Monocyte numbers were not significantly altered, but there were significant changes in monocyte subsets, with **(H)** reduced HLA-DR+ monocytes, **(J)** lower number of non-classical monocytes, **(K)** increased intermediate monocytes **(L)** no change in classical monocytes. The MFI changes are shown in **(M-O)**. Monocyte subpopulations are shown as proportions of total monocyte population, while neutrophil CD10+ and CD16+ populations are shown as a proportion of total neutrophil population, based on flow cytometry gating. * p <0.05, ** p <0.01, \***|p<0.001, NS = Non-Significant We then proceeded to analyse between-group differences in those with severe COVID-19 in comparison to those with mild or moderate disease. Univariate analysis of the 71 variables in our clinical and laboratory dataset identified a series of 15 factors associated with severe COVID-19 that remained statistically significant when corrected for false discovery rate (**Figure 4A**). These are ranked by strength of association, with strong associations seen with some of the differences noted in our earlier analysis. Specifically, elevation of the acute phase reactants CRP and IL-6 were closely associated with severe disease, with increased soluble CD25 also seen with severe disease. Furthermore, reduced expression of the neutrophil marker CD10 and monocyte marker HLA-DR were strongly associated with severe disease. No lymphocyte marker approached significance. Chi-squared testing of categorical variables ranked homelessness, male gender, higher clinical frailty scores and being a current smoker as the categorical variables most strongly associated with severe COVID-19, although no association met the threshold for statistically significance (**Figure 4B**). ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/09/03/2021.09.01.21262953/F5.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/F5) Figure 4: Univariate analysis of numerical demographic, clinical and immunological variables associated with severe COVID-19. **(A)**. t-test scores for differences in means between patients with severe and non-severe COVID-19 are indicated in the bar graph, corrected for false discovery rate using Benjamini–Yekutieli procedure The scores were derived from p-values by calculating -log10 (p-value). The red line indicates threshold of statistically significance (p<0.05). Univariate analysis of categorical variables associated with severe COVID-19 **(B)** Sorted Chi-squared test scores were used to test for significant associations between categorical variables and severe COVID-19. The red line indicates threshold of statistically significance (p<0.05). ### Machine learning identifies and ranks predictors of severity Having identified strong associations between specific individual immunological features and severity, we utilised multivariate analysis to establish the relative importance of the clinical and immunological parameters when adjusted for high-dimensional interactions. To establish this, we fitted a logistic regression model with elastic net penalty to the data. Our model demonstrated a high sensitivity and specificity of prediction for severity. The area under the receiving operating characteristic (AUROC) curve was 0.93 on both training and test sets, indicating excellent model predictive performance and absence of overfitting (**Figure 5B**). These results were confirmed with a permutation test resulting in statistically significant output (**Supplemental Figure 3A**; p<0.01). A precision-recall curve was also applied to estimate the model’s predictive discriminating performance due to the class imbalance in the ICU outcome measure (39 versus 69 cases). The average precision (AP) score was 0.89 on the training set and 0.88 on the test set (**Figure 5B**). Both of these scores indicate excellent generalisation properties for this model. The model’s true positive and true negative rates were assessed on training and test sets, visualized by confusion matrices (**Supplemental Figure 3B**), which indicate the model’s excellent discrimination between classes and absence of overfitting. ![Figure 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/09/03/2021.09.01.21262953/F6.medium.gif) [Figure 5:](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/F6) Figure 5: Machine learning modelling of variables identifies immunological features as the strongest predictors of severe COVID-19. **(A)** Sorted normalised feature weight values of variables identified by the model as predictive of progression to severe COVID-19 **(B)** Model performance for training and test sets assessed using receiver operating characteristic (ROC) curves and precision-recall curves. **(C)** Violin plots representing the difference of values between severe and non-severe COVID for the four features with the highest weight values in the predictive model **(D)** Predictive power of top four variables **(E)** Correlation matric of predictive variables within the model, analysed using Spearman correlation Blue indicates negative correlation between variables, red indicates positive correlation. Darker colour indicates a stronger association. COVID-19. Data was fitted to a logistic regression model with elastic net penalty. For the training set, the hyperparameters were optimized using a 10-fold cross-validation procedure and an exhaustive grid search on a training subset comprising 80% of the data. The remaining 20% of the data was used for the test set. When applied to our integrated dataset, the machine learning multivariate approach identified 12 variables *predictive* of progression to severe COVID-19 (i.e. with associated weights of non-zero values) when considered in interaction with all 71 variables. Normalised weight values associated with the top features of the model are shown in **Figure 5A**. The top four features stand out as most important with normalised weight values between 1.0 and 0.56. (**Figure 5C**) The normalised weight values of subsequent features were lower (0.28 to 0.01). The top feature predictive of severe COVID-19 was a reduced proportion of HLA-DR+ monocytes. The next most important features were higher CRP and IL-6 levels, and a lower percentage of CD10+ neutrophils (**Figure 5A**). Strikingly, a combination of these four features alone could predict COVID-19 severity with an average precision score of 0.925. Additional immunological features associated with severe COVID-19 with lower weight values included lower CD16 expression by neutrophils, lower percentages of non-classical (CD14-CD16+) and intermediate (CD14+CD16+) monocytes and lower numbers of NK cells (**Figure 5A and Supplemental Figure 4A, 4B, 4H**). Other parameters associated with severe COVID-19 were a high clinical frailty score of 6, higher AST levels, higher B cell counts and higher sCD25 levels (**Figure 5A and Supplemental Figures 4C-G**). Importantly, introducing additional variables to the four highest-ranked features did not improve the predictive ability of the model (**Figure 5D**). Potential relationships between each variable identified in the machine learning analysis were also investigated using a clustered Spearman correlation matrix (**Figure 5E**). Unsurprisingly, there was a positive correlation between levels of CRP and IL-6, with both of these markers negatively correlated with levels of HLA-DR+ monocytes and CD10+ neutrophils. Levels of HLA-DR+ monocytes and CD10+ neutrophils correlated positively with each other. There was a positive correlation between IL-6 and CRP with AST and sCD25. Proportional changes in non-classical and intermediate monocytes were also strongly associated with each other. ## Discussion COVID-19 has presented an enormous challenge to human health and society. The dramatic variability in clinical outcomes of SARS-CoV-2 infections has been proposed to be driven by variation in immune responses.10 Known risk factors for severe COVID-19 include older age, male gender, high BMI, and co-morbidities such as Type 2 diabetes and cardiovascular disease.28 The nature of the early stages of immune dysregulation in COVID-19, and how predictive it may be of disease severity in relation to these other important risk factors, is an area that still warrants further exploration. The data presented here on 108 patients with SARS-CoV-2 infection confirms experimental and research laboratory work on the characteristics of severe infection. Notably, it includes patients across the spectrum of disease severities; 69 (60%) of patients required supplemental oxygen therapy, 39 (36%) progressing to severe COVID-19, and 17 (16%) had mild disease managed in ambulatory care. In addition, and importantly, our methodology is suitable for use in a diagnostic immunology laboratory. Our cohort is broadly reflective of those reported elsewhere, with a median age of 61, more males than females requiring admission and a mortality rate of 9% which was higher for men.24 The similarities to cohorts reported elsewhere are crucial, given that early attempts to describe the immunopathological signatures of COVID-19 have been poorly applicable across populations.25-27 We used a whole blood flow protocol, with minimal handling and analysis within 4 hours of blood draw. This minimized artefactual alterations in myeloid populations. Patients presented to hospital and had their bloods drawn a median of 2 days prior to peak COVID-19 as defined by peak oxygen requirement, giving us insight into the mechanisms leading to severe respiratory distress. Severe disease in our cohort is characterised by markedly elevated inflammatory markers and a shift towards emergency myelopoiesis. Those with severe illness demonstrate increased inflammatory cytokines IL-6, CRP and soluble CD25. This works builds on several studies that have highlighted the importance of elevated CRP, IL-6 and the acute phase reactants in COVID-19.29 We also see a strong association with immature myeloid cells, both neutrophils and monocytes, with severe disease. This shift in myelopoiesis has been highlighted previously30, with left-shift of myeloid cells along with increased cytokine production has been seen in early infection.31 We also demonstrate the emergence of anaemia and increased RDW in severe disease, suggestive of emergency haematopoiesis. Indeed, increased RDW over the course of inpatient stay has been associated with increased COVID-19 mortality.32 Our data demonstrates evidence of inflammation and immune perturbation early in progression to severe COVID-19, with the emergence of a quartet of markers strongly associated with progression to severe disease. These markers are seen before the onset of peak clinical illness in our cohort. The strongest single predictor of severity on multivariate analysis was a reduced proportion of HLA-DR+ monocytes. Reductions in HLA-DR+ monocytes and reduced HLA-DR expression are appear to be an early predictor of poor outcome across many scenarios other than COVID-19.33-36 Increasing data have emerged showing reduced monocyte HLA-DR expression is associated with critical illness in COVID-19.16, 20, 37 The second member of the prediction quartet identified by this study is a proportional reduction in CD10+ neutrophils, another marker of emergency myelopoiesis. CD10 is a cell membrane metalloprotein that serves as a marker of neutrophil maturity.38 Reduced neutrophil CD10 expression is suggestive of an immature and proinflammatory population. Immature neutrophils, characterised by lower or absent CD10, have been reported to be immunostimulatory by promoting T-cell proliferation, survival and IFNγ production and have been associated with poor prognosis in sepsis.39, 40 Immature neutrophils characterised by low CD10+ have recently been associated with aggressive SARS-CoV-2 infection.20 The final two members of the predictive quartet are the acute phase reactants IL-6 and CRP. Elevated levels of IL-6 (and downstream protein CRP) are well-described as predictors of poor outcome in COVID-19.41-44 Elegant studies have highlighted a potent pro-inflammatory cytokine response, with prominent IL-6 elaboration, accompanied by a curious blunted type I and III interferon responses that seems unique to SARS-CoV-2 infection.28, 45 IL-6 also appears to be an important driver of monocyte HLA-DR loss, placing this cytokine at the centre of the immunopathological signature of COVID-19.16 Surprisingly, only a single categorical clinical variable, higher clinical frailty scores, emerged on multivariate analyses as associated with severity. The clinical frailty score captures biological, rather than chronological age and is a predictor of mortality in a wide-range of conditions.47, 48 Remarkably, a large number of other expected variables, including age and gender, failed to emerge as risk factors for severity in our cohort – however, this must be qualified by the fact that our cohort included few individuals with mild disease and does not rule out a role for these factors as risk factors for hospitalisation, for example. This is a single centre study in a largely Caucasian Irish cohort of predominantly hospitalised patients. Validation of these findings in other centres and with larger samples earlier in the course of disease is desirable. Understanding of the mechanisms causing some individuals with SARS-COV-2 infection to progress to severe disease is required to inform therapeutic design. Through robust analysis of extensive demographic, clinical and immunological parameters captured early in disease trajectory, we rank these parameters and adjust them for high-dimensional interactions and report a signature predictive of severity which is characterised by inflammation (elevated IL-6 and CRP) and emergency myelopoiesis (reduced neutrophil maturity and monocyte HLA-DR expression). This study builds on prior work describing severe COVID-19 infection, demonstrating that these features of emergency myelopoiesis develop prior to the point of peak illness and are of central importance in predicting progression to severe COVID-19. We include patients with mild disease who did not require hospitalisation; these patients represent a heretofore understudied cohort. Furthermore, we confirm that these approaches can be delivered in an accredited clinical laboratory and have the potential to be developed into routine tests. This is of particular importance given the need for early identication of individuals who may benefit from therapeutic interventions.22 Our work emphasises the central role of expanded immature neutrophil and monocyte populations in the setting of COVID-19 which may be similar to the expanded low density granulocyte and monocytic myeloid-derived suppressive cells seen in sepsis.46 ## Data Availability All relevant data is included in the manuscript. Further queries can be addressed to the corresponding author. ## Author Contributions Conceptualization: L.T., J.D., N.M.B., C.O’F., N.C. and C.N.C.; Methodology: L.T., A.H.D., S.I. and R.M.M.; Software: S.I. and A.W.; Validation: A.N., S.I., J.D., R.K., D.H., A.M., D.L., J.S., A.W. and R.M.M.; Formal Analysis: L.T., A.H.D., S.I. and A.W.; Investigation: L.T., A.H.D., A.N., J.D., R.K., D.H., A.M., D.L., K.R., J.S., M.A. and J.H.K.; Resources: M.H., D.B., P.B., C.L.B., C.D., R.O’R., A.M.M., C.Ba., I.M.L. and C.Be.; Data Curation: L.T., A.H.D., A.N., S.I., J.D. and A.W.; Writing-Original Draft: L.T., A.H.D., A.N., N.M.B., C.O’F., N.C. and C.N.C; Writing-Review & Editing: D.B., P.B., C.L.B., C.D., R.O’R., A.M.M., C.Ba., I.M.L. and C.Be.; Visualization: A.N., S.I., J.S., A.W., N.M.B., N.C. and C.N.C.; Supervision: J.D., A.W., R.M.M., C.Be., N.M.B., C.O’F, N.C. and C.N.C.; Funding Acquisition: L.T., R.M.M. and N.C. ## Competing Interests The authors have no competing interests to declare. ## STAR Methods View this table: [Table4](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/T4) ### Subject Details This study was carried out in accordance with the Code of Ethics of the World Medical Association.49 Ethical approval was granted by the Tallaght University Hospital / St. James’s Hospital Joint Research Ethics Committee (reference REC 2020-03). Informed consent was obtained from all participants. We recruited patients with a positive real-time PCR test for SARS-CoV-2 on nasopharyngeal aspirates at our institution over a two-month period (**Supplemental Figure 1**). 71 demographic, clinical and immunological measures were recorded for participants enrolled in this study (**Supplemental Table 1**). These included social and medical history (smoking status, alcohol and drug use, housing status, co-morbidities, medication use and self-reported date of symptom onset), and clinical frailty scores (adjudicated by the admitting physician).50 Routine haematological and biochemical laboratory results at the earliest time point after the first positive SARS-CoV-2 test, radiologist-reported chest X-ray results, and clinical parameters including peak oxygen requirement and peak CRP value (within fourteen days of positive SARS-CoV-2 test), assessment for ICU admission, admission to ICU and outcome were extracted from the electronic patient record. Mild COVID-19 was defined as no requirement for hospital admission (WHO scale 1 and 2), moderate COVID-19 as requirement for hospital admission (WHO scale 3 and 4) and severe COVID-19 as requirement for high-level respiratory support (high flow oxygen, invasive ventilation (WHO scale 5 and 6). ### Methods Details Immunophenotyping was carried out on fresh whole EDTA treated blood and samples were analysed on a FACS Canto II Flow Cytometer (BD San Jose USA), using BD DIVA v8 and FLO Jo v10 software. T cell (CD4+ and CD8+), B cell (CD19+) and Natural Killer cell (CD16+CD56+) percentages and absolute counts were generated by standard methods (BD TruCount). Naive (CD27+) and Effector (CD27-) T cells were characterised using CD27, CD45RA and CD197 antibodies. T cell activation was assessed by CD38 and HLA-DR expression. These assays were previously validated and accredited in line with ISO15189 standards. Classical, intermediate and non-classical monocytes were characterised by CD14 and CD16 expression. Proportions of these subpopulations were reported as percentages of the total monocyte population. Neutrophil functional maturity was interrogated by evaluating CD10 and CD16 expression. Proportions of these populations were reported as percentages of the total neutrophil population. IL-6 and soluble CD25 (sCD25) levels were measured in serum by ELISA (R&D Systems, Minneapolis, USA). Results were compared to a cohort of 40 pre-pandemic healthy controls, 20 of whom had extensive analysis performed for lymphoid and myeloid cell subsets, with 20 controls having extensive lymphoid analysis only. ### Statistical Analysis Between group differences were examined in the first instance using standard univariate statistics (t-tests, Wilcoxon rank-sum and chi-square tests as appropriate). In order to compare differences across the three groups of increasing COVID-19 severity, we used Kruscal-Wallis testing with a Dunn’s post-hoc test to estimate p-values. An alpha level of p<0.05 was considered statistically significant. We analysed a total of 71 demographic, clinical, laboratory and immunophenotyping parameters (independent variables) to analyse factors associated with severe COVID-19 (**Supplemental Table 1**). All data analysis, visualizations and modelling were performed using open source packages available in Python 3.7.51, 52 For univariate analysis, a FDR corrected t-test was applied on power transformed numerical variables. The power transformation was implemented using Yeo-Johnson transformation to ensure normal distribution of variables. FDR-correction was implemented using Benjamini-Yekuteli procedure.53 Categorical variables were tested using Chi2-test. An elastic net regularized logistic regression model was used for multivariate analysis and implemented for simultaneous feature selection and ICU status prediction.54 To train each model, the hyperparameters were optimized using a 10-fold cross-validation procedure and an exhaustive grid search on a training subset comprising 80% of the data. Generalization of the trained model was then assessed using the remaining 20% of the data as a test set. ## Supplemental material titles and legends View this table: [Supplemental Table 1:](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/T5) Supplemental Table 1: 71 parameters included in analysis View this table: [Supplemental Table 2:](http://medrxiv.org/content/early/2021/09/03/2021.09.01.21262953/T6) Supplemental Table 2: Timing of samples **Supplemental Figure 1: Patient enrolment flow chart** **Supplemental Figure 2: Flow cytometry gating strategy. (A)** Naïve and effector T lymphocytes **(B)** activated T lymphocytes **(C)** neutrophils **(D)** monocytes **Supplemental Figure 3: Performance of machine learning model. (A)** Permutation test of model **(B)** Confusion matrices, showing performance of model on training and test sets **Supplemental Figure 4: Violin plots of predictive features 5 – 12. (A)** CD16 MFI **(B)** Proportion of non-classical monocytes **(C)** Clinical Frailty Score **(D)** Aspartate transaminase **(E)** Proportion of natural killer cells **(F)** proportion of B cells **(G)** Soluble CD25 **(H)** Proportion of intermediate monocytes. MFI=median frequency intensity ## Acknowledgements The authors would like to thank all patients who took part in this study, our colleagues who provided care for patients and St James’s Hospital and the Wellcome CRF who supported the study. L.T. has been awarded the Irish Clinical Academic Training (ICAT) Programme, supported by the Wellcome Trust and the Health Research Board (Grant Number 203930/B/16/Z), the Health Service Executive, National Doctors Training and Planning and the Health and Social Care, Research and Development Division, Northern Ireland. R.M.M. has been awarded a Science Foundation Ireland (SFI) Investigator Award (15/IA/3041). N.C. and C.N.C. are part-funded by an SFI grant, Grant Code 20/SPP/3685. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. ## Footnotes * CNC and NC are joint senior authors on this paper. * STAR COVID Bioresource * Received September 1, 2021. * Revision received September 1, 2021. * Accepted September 3, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.Zhou F, Yu T, Du R, Fan G, Liu Y, Liu Z, et al. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. The lancet. 2020. 2. 2.Armocida B, Formenti B, Ussai S, Palestra F, Missoni E. The Italian health system and the COVID-19 challenge. The Lancet Public Health. 2020;5(5):e253. 3. 3.Chen N, Zhou M, Dong X, Qu J, Gong F, Han Y, et al. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study. The Lancet. 2020;395(10223):507–13. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/s01406736(20)30211-7&link_type=DOI) 4. 4.Lescure F-X, Bouadma L, Nguyen D, Parisey M, Wicky P-H, Behillil S, et al. Clinical and virological data of the first cases of COVID-19 in Europe: a case series. The Lancet Infectious Diseases. 2020. 5. 5.Shi Y, Wang Y, Shao C, Huang J, Gan J, Huang X, et al. COVID-19 infection: the perspectives on immune responses. Nature Publishing Group; 2020. 6. 6.Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The Lancet. 2020;395(10223):497–506. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0146736(20)30183-5&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 7. 7.Xu Z, Shi L, Wang Y, Zhang J, Huang L, Zhang C, et al. Pathological findings of COVID-19 associated with acute respiratory distress syndrome. The Lancet respiratory medicine. 2020;8(4):420–2. 8. 8.Tian S, Hu W, Niu L, Liu H, Xu H, Xiao S-Y. Pulmonary pathology of early phase 2019 novel coronavirus (COVID-19) pneumonia in two patients with lung cancer. Journal of Thoracic Oncology. 2020. 9. 9.Siddiqi HK, Mehra MR. COVID-19 illness in native and immunosuppressed states: a clinical-therapeutic staging proposal. The Journal of Heart and Lung Transplantation. 2020. 10. 10.Bonam SR, Kaveri SV, Sakuntabhai A, Gilardin L, Bayry J. Adjunct Immunotherapies for the Management of Severely Ill COVID-19 Patients. Cell Reports Medicine. 2020;1(2):100016. 11. 11.Wang D, Hu B, Hu C, Zhu F, Liu X, Zhang J, et al. Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus–infected pneumonia in Wuhan, China. Jama. 2020;323(11):1061–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2020.1585&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 12. 12.Hosseiny M, Kooraki S, Gholamrezanezhad A, Reddy S, Myers L. Radiology perspective of coronavirus disease 2019 (COVID-19): lessons from severe acute respiratory syndrome and Middle East respiratory syndrome. American Journal of Roentgenology. 2020;214(5):1078–82. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2214/AJR.20.22969&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 13. 13.Fogarty H, Townsend L, Ni Cheallaigh C, Bergin C, Martin-Loeches I, Browne P, et al. COVID19 coagulopathy in Caucasian patients. British Journal of Haematology. 2020;189(6):1044–9. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 14. 14.Phua J, Weng L, Ling L, Egi M, Lim C-M, Divatia JV, et al. Intensive care management of coronavirus disease 2019 (COVID-19): challenges and recommendations. The Lancet Respiratory Medicine. 2020. 15. 15.Zhou Y, Fu B, Zheng X, Wang D, Zhao C. Pathogenic T cells and inflammatory monocytes incite inflammatory storm in severe COVID-19 patients. National Science Review. 2020. 16. 16.Giamarellos-Bourboulis EJ, Netea MG, Rovina N, Akinosoglou K, Antoniadou A, Antonakos N, et al. Complex immune dysregulation in COVID-19 patients with severe respiratory failure. Cell host & microbe. 2020. 17. 17.Zheng H-Y, Zhang M, Yang C-X, Zhang N, Wang X-C, Yang X-P, et al. Elevated exhaustion levels and reduced functional diversity of T cells in peripheral blood may predict severe progression in COVID-19 patients. Cellular & molecular immunology. 2020:1–3. 18. 18.Schulte-Schrepping J, Reusch N, Paclik D, Baßler K, Schlickeiser S, Zhang B, et al. Severe COVID-19 Is Marked by a Dysregulated Myeloid Cell Compartment. Cell. 2020;182(6):1419-40.e23. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 19. 19.Laing AG, Lorenc A, del Molino del Barrio I, Das A, Fish M, Monin L, et al. A dynamic COVID-19 immune signature includes associations with poor prognosis. Nature Medicine. 2020;26(10):1623–35. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 20. 20.Silvin A, Chapuis N, Dunsmore G, Goubet A-G, Dubuisson A, Derosa L, et al. Elevated Calprotectin and Abnormal Myeloid Cell Subsets Discriminate Severe from Mild COVID-19. Cell. 2020;182(6):1401-18.e18. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 21. 21.Liu J, Li S, Liu J, Liang B, Wang X, Wang H, et al. Longitudinal characteristics of lymphocyte responses and cytokine profiles in the peripheral blood of SARS-CoV-2 infected patients. EBioMedicine. 2020:102763. 22. 22.Horby P, Lim WS, Emberson JR, Mafham M, Bell JL, Linsell L, et al. Dexamethasone in Hospitalized Patients with Covid-19-Preliminary Report. The New England journal of medicine. 2020. 23. 23.Johnson RM, Vinetz JM. Dexamethasone in the management of covid-19. British Medical Journal Publishing Group; 2020. 24. 24.Wedderburn L, Peckham H, De Gruijter N, Raine C, Radziszewska A, Ciurtin C, et al. Male sex identified by global COVIS-19 meta-analysis as a risk factor for death and ITU admission. Nature Communications. 2020. 25. 25.Yan L, Zhang H-T, Goncalves J, Xiao Y, Wang M, Guo Y, et al. An interpretable mortality prediction model for COVID-19 patients. Nature Machine Intelligence. 2020:1–6. 26. 26.Barish M, Bolourani S, Lau LF, Shah S, Zanos TP. External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19. Nature Machine Intelligence. 2020:1–3. 27. 27.Quanjel MJ, van Holten TC, Gunst-van der Vliet PC, Wielaard J, Karakaya B, Söhne M, et al. Replication of a mortality prediction model in Dutch patients with COVID-19. Nature Machine Intelligence. 2020:1–2. 28. 28.Blanco-Melo D, Nilsson-Payant BE, Liu W-C, Uhl S, Hoagland D, Møller R, et al. Imbalanced host response to SARS-CoV-2 drives development of COVID-19. Cell. 2020. 29. 29.Overmyer KA, Shishkova E, Miller IJ, Balnis J, Bernstein MN, Peters-Clarke TM, et al. Large-scale Multi-omic Analysis of COVID-19 Severity. Cell systems. 2020. 30. 30.Mann ER, Menon M, Knight SB, Konkel JE, Jagger C, Shaw TN, et al. Longitudinal immune profiling reveals key myeloid signatures associated with COVID-19. Science immunology. 2020;5(51). 31. 31.Su Y, Chen D, Yuan D, Lausted C, Choi J, Dai CL, et al. Multi-omics resolves a sharp disease-state shift between mild and moderate COVID-19. Cell. 2020. 32. 32.Foy BH, Carlson JC, Reinertsen E, Valls RPI, Lopez RP, Palanques-Tost E, et al. Association of Red Blood Cell Distribution Width With Mortality Risk in Hospitalized Adults With SARS-CoV-2 Infection. JAMA Network Open. 2020;3(9):e2022058–e. 33. 33.Landelle C, Lepape A, Voirin N, Tognet E, Venet F, Bohé J, et al. Low monocyte human leukocyte antigen-DR is independently associated with nosocomial infections after septic shock. Intensive care medicine. 2010;36(11):1859–66. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00134-010-1962-x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20652682&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000282702100010&link_type=ISI) 34. 34.Le Tulzo Y, Pangault C, Amiot L, Guilloux V, Tribut O, Arvieux C, et al. Monocyte human leukocyte antigen–DR transcriptional downregulation by cortisol during septic shock. American journal of respiratory and critical care medicine. 2004;169(10):1144–51. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1164/rccm.200309-1329OC&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15028560&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000221314700013&link_type=ISI) 35. 35.Cheron A, Floccard B, Allaouchiche B, Guignant C, Poitevin F, Malcus C, et al. Lack of recovery in monocyte human leukocyte antigen-DR expression is independently associated with the development of sepsis after major trauma. Critical Care. 2010;14(6):R208. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/cc9331&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21092108&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 36. 36.Lekkou A, Karakantza M, Mouzaki A, Kalfarentzos F, Gogos C. Cytokine production and monocyte HLA-DR expression as predictors of outcome for patients with community-acquired severe infections. Clin Diagn Lab Immunol. 2004;11(1):161–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1128/CDLI.11.1.161-167.2004&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=14715564&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 37. 37.Spinetti T, Hirzel C, Fux M, Walti LN, Schober P, Stueber F, et al. Reduced monocytic HLA-DR expression indicates immunosuppression in critically ill COVID-19 patients. Anesthesia & Analgesia. 2020. 38. 38.Brandau S, Hartl D. Lost in neutrophil heterogeneity? CD10! Blood, The Journal of the American Society of Hematology. 2017;129(10):1240–1. 39. 39.Yan L, Zhang H-T, Goncalves J, Xiao Y, Wang M, Guo Y, et al. An interpretable mortality prediction model for COVID-19 patients. Nature Machine Intelligence. 2020. 40. 40.Marini O, Costa S, Bevilacqua D, Calzetti F, Tamassia N, Spina C, et al. Mature CD10+ and immature CD10– neutrophils present in G-CSF–treated donors display opposite effects on T cells. Blood, The Journal of the American Society of Hematology. 2017;129(10):1343–56. 41. 41.Han H, Ma Q, Li C, Liu R, Zhao L, Wang W, et al. Profiling serum cytokines in COVID-19 patients reveals IL-6 and IL-10 are disease severity predictors. Emerging Microbes & Infections. 2020;9(1):1123–30. 42. 42.Dong Y, Zhou H, Li M, Zhang Z, Guo W, Yu T, et al. A novel simple scoring model for predicting severity of patients with SARS-CoV-2 infection. Transboundary and Emerging Diseases. 2020. 43. 43.Ruan Q, Yang K, Wang W, Jiang L, Song J. Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China. Intensive care medicine. 2020:1–3. 44. 44.Herold T, Jurinovic V, Arnreich C, Lipworth BJ, Hellmuth JC, von Bergwelt-Baildon M, et al. Elevated levels of interleukin-6 and CRP predict the need for mechanical ventilation in COVID-19. Journal of Allergy and Clinical Immunology. 2020. 45. 45.Lucas C, Wong P, Klein J, Castro TB, Silva J, Sundaram M, et al. Longitudinal analyses reveal immunological misfiring in severe COVID-19. Nature. 2020;584(7821):463–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41586-020-2588-y&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 46. 46.Darcy CJ, Minigo G, Piera KA, Davis JS, McNeil YR, Chen Y, et al. Neutrophils with myeloid derived suppressor function deplete arginine and constrain T cell function in septic shock patients. Critical Care. 2014;18(4):R163. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/cc14003&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25084831&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 47. 47.Alfaadhel TA, Soroka SD, Kiberd BA, Landry D, Moorhouse P, Tennankore KK. Frailty and mortality in dialysis: evaluation of a clinical frailty scale. Clinical Journal of the American Society of Nephrology. 2015;10(5):832–40. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiY2xpbmphc24iO3M6NToicmVzaWQiO3M6ODoiMTAvNS84MzIiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wOS8wMy8yMDIxLjA5LjAxLjIxMjYyOTUzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 48. 48.Galizia G, Cacciatore F, Testa G, Della-Morte D, Mazzella F, Langellotto A, et al. Role of clinical frailty on long-term mortality of elderly subjects with and without chronic obstructive pulmonary disease. Aging clinical and experimental research. 2011;23(2):118–25. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/BF03351076&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21743290&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) 49. 49.Association WM. World Medical Association Declaration of Helsinki. Ethical principles for medical research involving human subjects. Bulletin of the World Health Organization. 2001;79(4):373. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11357217&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F09%2F03%2F2021.09.01.21262953.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000168086900015&link_type=ISI) 50. 50.Rockwood K, Song X, MacKnight C, Bergman H, Hogan DB, McDowell I, et al. A global clinical measure of fitness and frailty in elderly people. Cmaj. 2005;173(5):489–95. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNzMvNS80ODkiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wOS8wMy8yMDIxLjA5LjAxLjIxMjYyOTUzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 51. 51.Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine learning in Python. the Journal of machine Learning research. 2011;12:2825–30. [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000298103200003&link_type=ISI) 52. 52.Vallat R. Pingouin: statistics in Python. Journal of Open Source Software. 2018;3(31):1026. 53. 53.Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Annals of statistics. 2001:1165–88. 54. 54.Zou H, Hastie T. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology). 2005;67(2):301–20. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1467-9868.2005.00503.x&link_type=DOI) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000227498200007&link_type=ISI)