Predicting mortality in SARS-COV-2 (COVID-19) positive patients in the inpatient setting using a Novel Deep Neural Network
==========================================================================================================================

* Maleeha Naseem
* Hajra Arshad
* Syeda Amrah Hashmi
* Furqan Irfan
* Fahad Shabbir Ahmed

## ABSTRACT

**Background** The second wave of COVID-19 pandemic is anticipated to be worse than the initial one and will strain the healthcare systems even more during the winter months. Our aim was to develop a machine learning-based model to predict mortality using the deep learning Neo-V framework. We hypothesized this novel machine learning approach could be applied to COVID-19 patients to predict mortality successfully with high accuracy.

**Methods** The current Deep-Neo-V model is built on our previously statistically rigorous machine learning framework [Fahad-Liaqat-Ahmad Intensive Machine (FLAIM) framework] that evaluated statistically significant risk factors, generated new combined variables and then supply these risk factors to deep neural network to predict mortality in RT-PCR positive COVID-19 patients in the inpatient setting. We analyzed adult patients (≥18 years) admitted to the Aga Khan University Hospital, Pakistan with a working diagnosis of COVID-19 infection (n=1228). We excluded patients that were negative on COVID-19 on RT-PCR, had incomplete or missing health records. The first phase selection of risk factor was done using Cox-regression univariate and multivariate analyses. In the second phase, we generated new variables and tested those statistically significant for mortality and in the third and final phase we applied deep neural networks and other traditional machine learning models like Decision Tree Model, k-nearest neighbor models and others.

**Results** A total of 1228 cases were diagnosed as COVID-19 infection, we excluded 14 patients after the exclusion criteria and (n=)1214 patients were analyzed. We observed that several clinical and laboratory-based variables were statistically significant for both univariate and multivariate analyses while others were not. With most significant being septic shock (hazard ratio [HR], 4.30; 95% confidence interval [CI], 2.91-6.37), supportive treatment (HR, 3.51; 95% CI, 2.01-6.14), abnormal international normalized ratio (INR) (HR, 3.24; 95% CI, 2.28-4.63), admission to the intensive care unit (ICU) (HR, 3.24; 95% CI, 2.22-4.74), treatment with invasive ventilation (HR, 3.21; 95% CI, 2.15-4.79) and laboratory lymphocytic derangement (HR, 2.79; 95% CI, 1.6-4.86). Machine learning results showed our DNN (Neo-V) model outperformed all conventional machine learning models with test set accuracy of 99.53%, sensitivity of 89.87%, and specificity of 95.63%; positive predictive value, 50.00%; negative predictive value, 91.05%; and area under the curve of the receiver-operator curve of 88.5.

**Conclusion** Our novel Deep-Neo-V model outperformed all other machine learning models. The model is easy to implement, user friendly and with high accuracy.

Keywords
*   COVID-19
*   deep learning
*   mortality prediction
*   Neo-V Framework

## INTRODUCTION

Severe acute respiratory syndrome coronavirus 2 (SARS-COV 2) has caused 60 million infections and 1.4 million deaths worldwide (1) and 800, 000 deaths in the United States (2). Despite strict measures being deployed and special instructions given to the mass public, the second wave is anticipated to be far worse than the first one (3). Some countries have started to ease those earlier restrictions because of economic implications from the initial lockdown, which may create a further deepening of the current crisis, as cases continue to rise. This could overwhelm the already strained healthcare systems across the United States and the world.

Machine learning has been extensively used in the automotive, defense and fin-tech industry over the past couple of years with great success. The use of these systems to predict health outcomes have been limited. Epidemic Renormalization Group (eRG) has used a machine learning framework to predict the time evolution of the first and second wave based on the data from the first wave in Europe (4). We have previously developed algorithms that predict mortality in the clinical setting and performed better than most clinical scales utilized currently to predict mortality (5-7) During the current COVID-19 pandemic crisis, the aim was to develop a mortality prediction tool that can predict death in COVID-19 patients at admission. This would help the already strained healthcare systems and physicians around the world in crucial clinical decision making, resource management and family-counselling. We hypothesize that machine learning, specifically deep-learning could be applied to COVID-19 patients with high accuracy. Using deep-learning to predict mortality in these patients may assist in clinical decision making, risk stratification and planning strategies in future for such pandemics at a larger scale. Not much work has been done in mortality prediction in COVID-19 patients in with lower socio-economic countries (LMIC) or developed countries. Our hypothesis is that machine learning could be applied to COVID-19 patients with high accuracy. Hence, predicting mortality and clinical outcomes using ML algorithms may assist in clinical decision making, risk stratification and planning strategies in future for such pandemics at a larger scale.

## METHODS

### Clinical setting and dataset

Clinical data was conducted at the Aga Khan University Hospital (AKUH). All patients’ records were completely anonymous, and the data collected has received Institutional Review Board/Ethical Review Committee (IRB/ERC) approval from Aga Khan University Hospital (AKUH), Pakistan. The dataset was de-identified and our study complied with the ethical principles recommended by Helsinki declaration (1964) and its amendments. We retrospectively collected data from AKUH – electronic medical record (EMR) that were admitted with a primary diagnosis of COVID-19 infection to the hospital between February 2020 and September 2020.

### Patient data collection and selection criteria

We included adult patients (>18 years of age) that were admitted to the hospital with a diagnosis of COVID-19 or were tested positive during their admission on reverse_transcriptase polymerase chain reaction (RT-PCR) based on Center for Disease Control and Prevention (CDC) and College of American Pathologist (CAP) guidelines (8, 9). Data was collected on demographics, and comorbidities at admission, the first 24-hours of laboratory investigations (hematological and blood biochemistry, (**Table 1**.), imaging and complete clinical characteristics (history, examination, treatment, hospital course and outcomes). We excluded all patients that had RT-PCR negative tests for COVID-19 and incomplete records or inaccurate medical record information.

View this table:
[Table 1.](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/T1)

Table 1. 
Demographics with Univariate and multivariate analysis of clinical variables as part of Phase I of the Neo-V and FLAIM machine learning frameworks.

### The Neo-V Framework

Neo-V is a tri-phase bio-statically rigorous machine learning that builds on our previous framework that had better accuracy then the currently used clinical scoring systems in predicting mortality in the intensive care unit (ICU) patients(5, 6).

**Phase I:** Also known as the statistical-phase; in which data was analyzed by univariate and multivariate Cox-regression analysis (X) using IBM SPSS (version 24.0.0.0) (X) for outcome assessment with hazard ratio and confidence intervals. A *p*-value of <0.05 was considered statistically significant. We also did demographic data frequency analysis. Statistical analysis was carried out on all the variables included in **Table 1**.

**Phase II:** In contrast to our previously published model we created new variables for the existing dataset called neo-variables. These variables included a combination of two clinically relevant labs that were significant in both the univariate and the multivariate analysis **(Table 1.)**. These variables also underwent univariate and multivariate analysis for outcome assessment with hazard ratio and confidence intervals

**Phase III:** Biological datasets are highly imbalanced with respect to the outcomes (i.e. more people were survived, then those who didn’t) and machine learning models are very sensitive to imbalanced data and can produce variable and non-reproducible results. To address this, we optimized the dataset using Synthetic Minority Over-sampling Technique (SMOTE) algorithm during the training process (10). In the machine learning phase, we used all variables that were statistically significant in phase I and II in both the univariate and multivariate analysis (non-significant risk factors were excluded). The final dataset was randomized and divided into a training and testing set with a 70/30 percent split respectively (30% data left out to test the models). After partitioning the data, we allocated feature vectors of the training instances by X\_train with corresponding outcome label as Y\_train. Similarly, for the test set we allocated X\_test and Y\_test as testing vector instances and corresponding outcomes, respectively. The models trained on X\_train and Y\_train. The models tried to learn the behavior/distribution of the data and generate a hypothesis/fitting function. Once the training is concluded the model will then test the X\_test and produce an output (prediction) called Y_pred. A comparison is done between Y_pred and Y_test. We had previously discussed that reduction of the number of irrelevant risk factors can produce better performances and significantly improve classifications. In this study we used conventional machine learning models including Random Trees (CART), K-Nearest Neighbor (KNN), Support Vector Classifier - Radial Basis Function (SVC - RBF), Ada-Boost-Classifier (ABC) and Quadratic Discriminant Analysis (QDA) and a deep neural network (DNN).

### Deep-FLAIM

FLAIM framework only has phase I and III and we used it to compare it with Neo-V Framework. The Deep-FLAIM model is a 4 layered model and details have been reported previously(5).

### Performance, Primary and Secondary Outcomes Analyses

Performance of all models was evaluated by comparing their accuracies and area under the receiver-operator curves (AUROC). Primary outcomes included sensitivity and specificity, while secondary outcomes included positive predictive values (PPV) and negative predictive values (NPV).

## RESULTS

From a total of 1228 patients we selected 1214 patients that were adult patients with complete data and RT-PCR proven COVID-19 infections. Demographics of this population showed a median age of 55 years (range 19-96 years), around 28% (n=336) of the admitted population being elderly (>65 years of age). Median length of stay 5 days (range 1-54 days), most patients admissions were male (61.5%).

The clinical characteristics of these patients included hypertension (43%, n=521) and diabetes (36%, n=441) being the most common comorbid. Presenting symptoms ranged widely from being asymptomatic to shortness. The most significant clinical risk factors for death during the hospital admission included chronic kidney disease (n=71, HR=2.74, 95%CI=1.63-4.61), ischemic heart disease (n=158, HR=1.80, 95%CI=1.19-2.73), other comorbidities (n=571, HR=1.72, 95%CI=1.21-2.45), shortness of breath (n=570, HR=1.96, 95%CI=1.35-2.86), other symptoms (non-respiratory and non-gastrointestinal symptoms, n=321, HR=1.66, 95%CI=1.16-2.38), acute kidney injury (n=179, HR=2.47, 95%CI=1.74-3.51), acute respiratory distress syndrome / respiratory failure (n=147, HR=2.53, 95%CI=1.77-3.63), septic shock (n=49, HR=4.30, 95%CI=2.91-6.37), intensive care unit (ICU) admission (n=106, HR=3.24, 95%CI=2.22-4.74), number of ICU admissions during current hospitalization (range=0-2, HR=2.01, 95%CI=1.49-2.87), invasive ventilation (n=74, HR=3.21, 95%CI=2.15-4.79), non-invasive ventilation (BiPAP/CPAP) (n=243, HR=1.87, 95%CI=1.32-2.67), supportive treatment (n=617, HR=1.87, 95%CI=1.32-2.67) and blood group AB+ (n=48, HR=2.48, 95%CI=1.26-4.93); while the utilization of systemic steroids (n=430, HR=0.66, 95%CI=0.45-0.97) and presence of fever (n=768, HR=0.57, 95%CI=0.40-0.81) were associated with better overall all survival. The most significant (statistical association with mortality) laboratory abnormalities for these patients included white cell count (n=455, HR=2.09, 95%CI=1.47-2.59), lymphocyte counts(n=378, HR=2.79, 95%CI=1.60-4.86), neutrophil count (n=1044, HR=2.67, 95%CI=1.24-5.70), platelets count (n=363, HR=1.41, 95%CI=1.00-2.01), hematocrit (n=505, HR=1.81, 95%CI=1.26-2.61), hemoglobin (n=879, HR=1.63, 95%CI=1.05-2.55), blood urea nitrogen (BUN, n=382, HR=2.78, 95%CI=1.94-4.01), creatinine (n=373, HR=2.73, 95%CI=1.90-3.94), international normalized ratio (INR, n=198, HR=3.24, 95%CI=2.28-4.63), prothrombin time (PT, n=378, HR=1.79, 95%CI=1.24-2.59). There were some risk factors that were significant for mortality but had too few patients including; shock liver (n=3, HR=11.47, 95%CI=2.77-47.68), blood group AB- (n=6, HR=7.63, 95%CI=2.39-24.48), Rhinorrhea (n=7, HR=0.001, 95%CI=0.01-1.7e200), treatment with intravenous IgG (n=6, HR=3.95, 95%CI=1.25-12.49) and pneumothorax (clinical or radiological, n=22, HR=2.01, 95%CI=1.18-3.46). **Table 1**. shows results from univariate and multivariate analysis (hazard ratios, confidence intervals and *p*-values) of all the clinical and laboratory data.

In phase II (new variable phase) we created 11 new variables that included; total number of comorbidities (range 0-6, HR=1.30, 95%CI=1.14-1.49), more than 2 comorbidities (n=499, HR=1.79, 95%CI=1.23-2.61), total number of symptoms(range 0-6, HR=1.07, 95%CI=0.91-1.26), total types of treatments received (range 0-6, HR=1.27, 95%CI=1.11-1.46), abnormal labs: creatinine or blood urea nitrogen (CR or BUN, (n=517, HR=0.36, 95%CI=0.36-0.89)), hemoglobin or hematocrit (HB or HCT, (n=883, HR=0.56, 95%CI=0.36-0.89)), platelets or INR (PLT or INR, (n=460, HR=2.09, 95%CI=1.47-2.98)), PT or INR (PT or INR, (n=502, HR=0.414, 95%CI=)), total leukocyte counts of lymphocytes (TLC or LYMP, (n=0.19, HR=, 95%CI=0.09-0.43)), total leukocyte counts or neutrophils (TLC or NEU, (n=1085, HR=0.18, 95%CI=0.06-0.57)), total number of abnormal labs(range 0-17, HR=1.22, 95%CI=1.15-1.30), univariate and multivariate statistical analysis can be seen in **Table 2**. The new variables in phase II referred to as ‘neo-variables’ were statistically significant in the uni and the multivariate analysis except the total number of symptoms.

View this table:
[Table 2.](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/T2)

Table 2. 
Derivative variables and there univariate and multivariate analysis with mortality during hospital stay.

The performance of our previously designed Deep-FLAIM model was compared to the Neo-V framework (including Deep-Neo-V and other conventional machine learning algorithms) see **Table 3**. Performance results show Deep-FLAIM (training accuracy = 86.7%, testing accuracy = 84.7%, sensitivity = 68.9, specificity = 86.9, PPV = 42.5, NPV = 95.2, FPR =13.1 and AUROC = 86.9); while conventional machine learning show: Random Forest (RF, training accuracy = 94.9%, testing accuracy = 85.8%, sensitivity = 28.9, specificity = 93.4, PPV = 32.4, NPV = 90.4, FPR = 6.25 and AUROC = 69.5), k-Nearest Neighbors (k-NN, training accuracy = 92.3%, testing accuracy = 77.8%, sensitivity = 42.2, specificity = 82.8, PPV = 25.7, NPV = 91.1, FPR = 17.2 and AUROC = 66.5), Simple Vector Classifier – Radial Basis Function (SVC – RBF, training accuracy = 69.0%, testing accuracy = 65.8%, sensitivity = 75.6, specificity = 64.4, PPV = 23.0, NPV = 94.9, FPR = 35.6 and AUROC = 80.8), Decision Trees (DT, training accuracy = 98.9%, testing accuracy = 85.2%, sensitivity = 42.2, specificity = 91.3, PPV = 40.4, NPV = 91.8, FPR = 8.8 and AUROC = 66.7), Adaptive Boosted Classifier (ABC, training accuracy = 90.1%, testing accuracy = 85.5%, sensitivity = 48.9, specificity = 90.6, PPV = 42.3, NPV = 82.7, FPR = 9.4 and AUROC = 77.65) and Quadratic discriminant analysis (QDA, training accuracy = 94.6%, testing accuracy = 87.4%, sensitivity = 60.0, specificity = 91.3, PPV = 49.1, NPV = 94.2, FPR = 8.74 and AUROC = 80.84). While our best model was a deep neural network (Deep-Neo-V) with training accuracy = 98.7%, testing accuracy = 87.7%, sensitivity = 33.3, specificity = 95.3, PPV = 50.0, NPV = 91.1, FPR = 4.69 and AUROC = 88.5. All receiver operator curves are shown in **Supplemental Figure X**.

View this table:
[Table 3.](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/T3)

Table 3. 
Primary and secondary outcomes machine learning results. Test set accuracy, training set accuracy, precision, sensitivity, specificity, positive predictive value, negative predictive value and AUROC.

View this table:
[Table 4.](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/T4)

Table 4. 
Comparison of currently available Deep learning models for mortality in the inpatient setting.

## DISCUSSION

As the second wave of COVID-19 has started to unfold the already stained healthcare systems globally are being pushed to the limit with hospital and intensive care unit (ICU) beds reaching full-capacity. Impact of the virus has been global with developed countries even struggling with infection rates and hospitalization (11). The second wave is anticipated to be much tougher than the first one (12). With new vaccines on the horizon infection rates in the United states have skyrocketed to 1.34 million cases being diagnosed in the second week of December 2020 and more than 1100 deaths (13). The vaccines have just received FDA approval for widespread use of the vaccine for prevention of the disease while the actual logistics and distribution plans are still unknown (14). However, there is a need for the existence of clinical biomarkers and predictive models for mortality in these patients. machine learning has been used to predict mortality in cancer(15), cardiac disease (16); while our own work on mortality prediction has been on trauma patients, postoperative ileus cases in the ICU (5, 6) and diverticulitis in the inpatient setting (17) with good accuracies. A lot of epidemiological studies reporting clinical, laboratory and mortality outcomes have been done worldwide including developed and developing countries, but very few actually reported or developed a machine learning model for predicting the outcomes with a set accuracy.

In the current study we developed a new method of machine learning (Neo-V Framework) that uses a smaller amount of cases to train a deep neural network to give better predictions. This model is different from our previous FLAIM Framework (two-phase) and has a tri-phase structure (**Figure 1**.) Clinically we used just 1214 hospitalized patients for mortality prediction in RT-PCR positive COVID-19 cases using only data from the first 24-hours after admission. Clinical data analysis showed that with increasing age the patients’ mortality also increases. There are number of clinical risk factors that were associated with worse outcomes and documented in the clinical literature like chronic obstructive lung disease [COPD](18), chronic kidney disease [CKD](19), ischemic heart disease [IHD](20), pneumothorax (radiological or clinical diagnosis)(21), acute respiratory syndrome [ARDS](22), septic shock(23), shortness of breath(24), ICU admission(25), AB+ Blood group(26) and recurrent admission to the ICU. Hematological labs that were associated with mortality (previously presented in the results) were also seen in other studies(27). Biochemical laboratory abnormalities like creatinine, blood urea nitrogen(28), INR and PT(29) were also associated with mortality. Patients managed with invasive ventilation(30) and non-invasive ventilation (31) which actually signifies that the patients that were not able to maintain normal respiratory physiology had worse outcomes. Having fever(32) and the use of systemic steroids(33) as an early symptom of COVID-19 had better prognosis in our patient population.

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/08/29/2020.12.13.20247254/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/F1)

Figure 1. 
Experimental design of the study and the machine frameworks

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/08/29/2020.12.13.20247254/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/F2)

Figure 2. 
Receiver Operating Curve for Deep-Neo-V with AUROC

Machine learning has been used to predict mortality in the inpatient and the ICU setting in a number of different clinical conditions including our prior papers. The methods need to continue to evolve to have better outcome predictions, previously we developed the FLAIM-Framework approach, which was an attempt to build a workflow pipeline to produce more accurate results. The Neo-V Framework builds on our previous work and has the statistical power of FLAIM, but it can be used to apply deep learning to smaller datasets. We call this technique “horizontal expansion” of the dataset in which we horizontally expand the data by combining two or more variables to create new-variables (Neo-V). The combination was conditional that the variables were clinically relevant e.g. BUN and creatinine. In contrast, vertical expansion of the dataset is adding new patients.

We wanted to do a complete and thorough analysis of the Deep-Neo-V algorithm, so we compared it to other deep-learning models out there on COVID-19 mortality prediction currently available in medical literature(34). Our Deep-Neo-V model outperforms all our conventional models and our Deep-FLAIM model. It also outperformed the currently available Deep-learning model by Zhu J.S. *et al* in terms of training set accuracy, testing set accuracy, AUROC, Precision, specificity and positive predictive value(34). However, the Deep-Neo-V model underperformed in terms of sensitivity and slightly with negative predictive value. The Deep-Neo-V will continue to improve and develop and will potentially be replaced by a model with better performance parameters (accuracy, PPV and NPV). This model in its current configuration can be used to predict mortality after day-1 (considers labs and clinical characteristics in the first 24 hours) of hospital admission and can help in stratification of patients. It can help clinicians answer a number of questions and aid in decision-making like triaging patients, should an elderly patient receive aggressive treatment, or a younger patient receive life supportive management. These are tough questions and the model will give clinicians clarity about the course and plan for the patient. This model can also help clinicians with family counselling, appropriate decision making, limit excessive intervention or aggressive treatments and effective resource management. Like most digital tools this current algorithm is user friendly, can provide results instantaneously and easy to use. After further validation this model can be incorporated into hospital patient management systems and ready for clinical use.

The Deep-Neo-V model has some limitations in terms of the available dataset, retrospective nature of the dataset and data form a single hospital, analyzed at admission and day-one data, other observational study confounders may exist and are unaccounted for. In the immediate future we actively look to validate these findings in an external dataset. In the longer term we will continue to develop an algorithm built on the Neo-V Framework approach that has the potential to be implemented, initially in future pandemics because of its ability to accurately predict outcomes using smaller datasets.

## CONCLUSION

Deep-Neo-V is a statistically robust machine learning model that is developed for clinical use to predict mortality risk in patients admitted with RT-PCR proven COVID-19 infection. The mortality prediction was modeled based on clinically relevant variables (patient associated risk factors and the first 24-hours labs. Our experimental results show that with a high accuracy and specificity it has the potential to develop as a test of choice for predicting mortality in COVID-19 patients. These findings need further external validation.

## Supporting information

Supplemental Figure 1. [[supplements/247254_file08.docx]](pending:yes)

## Data Availability

Available on request

## AUTHORS CONTRIBUTION

Maleeha Naseem: Study design, methodology, protocol design, IRB/ERC approval, data collection supervision, data collection tool and quality control, statistical analysis review, medical literature review and article writing.

Hajra Arshad: Data collection, protocol design, data collection tool, statistical analysis review, medical literature review and article writing.

Syeda Amrah Hashmi: Data collection, statistical analysis review, medical literature review and article writing.

Furqan Irfan. Medical literature review, statistical analysis review, and article writing.

Fahad Shabbir Ahmed. MD: Original concept, study design, methodology, data collection tool and construction of FLAIM, Deep-FLAIM and Neo-V frameworks, statistical analysis and review, python-coding for machine learning and article writing, editing, review.

## CONFLICT OF INTEREST

None

## FUNDING STATEMENT

None

![](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/08/29/2020.12.13.20247254/F3/graphic-10.medium.gif)

[](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/F3/graphic-10)

![](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/08/29/2020.12.13.20247254/F3/graphic-11.medium.gif)

[](http://medrxiv.org/content/early/2021/08/29/2020.12.13.20247254/F3/graphic-11)

Supplemental Figure 1. 
Receiver Operating Curve with AUROC for a) RF, b) kNN, c) SVC-RBF, d) DT, e) ABC, f) QDA and g) Deep-FLAIM

## Footnotes

*   **Conflict of Interest:** None.

*   **Disclosures:** None

*   Name Correction of Syeda Amrah Hashmi

*   Received December 13, 2020.
*   Revision received August 27, 2021.
*   Accepted August 29, 2021.


*   © 2021, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## REFERENCES

1.  1.Hsiang S, Allen D, Annan-Phan S, Bell K, Bolliger I, Chong T, et al. The effect of large-scale anti-contagion policies on the COVID-19 pandemic. Nature. 2020;584(7820):262–7.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

2.  2.Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis. 2020;20(5):533–4.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1473-3099(20)30120-1&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

3.  3.Hunter DJ. Trying to “Protect the NHS” in the United Kingdom. New England Journal of Medicine. 2020.
    
    

4.  4.Cacciapaglia G, Cot C, Sannino F. Second wave COVID-19 pandemics in Europe: a temporal playbook. Sci Rep. 2020;10(1):15514.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

5.  5.Ahmed FS, Ali L, Joseph BA, Ikram A, Ul Mustafa R, Bukhari SAC. A statistically rigorous deep neural network approach to predict mortality in trauma patients admitted to the intensive care unit. J Trauma Acute Care Surg. 2020;89(4):736–42.
    
    

6.  6.Ahmad FS, Ali L, Raza Ul M, Khattak HA, Hameed T, Wajahat I, et al. A hybrid machine learning framework to predict mortality in paralytic ileus patients using electronic health records (EHRs). Journal of Ambient Intelligence and Humanized Computing. 2020.
    
    

7.  7.Naseem M, Akhund R, Arshad H, Ibrahim MT. Exploring the Potential of Artificial Intelligence and Machine Learning to Combat COVID-19 and Existing Opportunities for LMIC: A Scoping Review. J Prim Care Community Health. 2020;11:2150132720963634.
    
    

8.  8.Pathologist CoA. Guidance for COVID-19 Testing for CAP-Accredited Laboratories. CAP Guidelines, caporg. 2020.
    
    

9.  9.Prevention CfDCa. CDC’s Diagnostic Test for COVID-19 Only and Supplies. CDC Guidelines, cdcgov. 2020.
    
    

10. 10. N. V. Chawla,  K. W. Bowyer,  L. O. Hall,  W. P. Kegelmeyer. SMOTE: Synthetic Minority Over-sampling Technique. J Artif Int Res. 2002;16(1):321–57.
    
    

11. 11.Keni R, Alexander A, Nayak PG, Mudgal J, Nandakumar K. COVID-19: Emergence, Spread, Possible Treatments, and Global Burden. Front Public Health. 2020;8:216.
    
    

12. 12.Xu S, Li Y. Beware of the second wave of COVID-19. Lancet. 2020;395(10233):1321–2.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-6736(20)30845-X&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

13. 13.(CDC) CfDCaP. CDC COVID Data Tracker: Maps, charts, and data provided by the CDC. [http://www.cdcgov](http://www.cdcgov). 2020.
    
    

14. 14.(FDA) FDA. COVID-19 Vaccines: FDA has rigorous scientific and regulatory processes in place to facilitate development and ensure the safety, effectiveness and quality of COVID-19 vaccines. Online. 2020.
    
    

15. 15.Parikh RB, Manz C, Chivers C, Regli SH, Braun J, Draugelis ME, et al. Machine Learning Approaches to Predict 6-Month Mortality Among Patients With Cancer. JAMA Netw Open. 2019;2(10):e1915997.
    
    

16. 16.Motwani M, Dey D, Berman DS, Germano G, Achenbach S, Al-Mallah MH, et al. Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis. Eur Heart J. 2017;38(7):500–7.
    
    

17. 17.Ahmed FS, Raza-Ul-Mustafa, Ali L, Imadud-Deen, Hameed T, Ikram A, et al. Machine Learning Can Predict Deaths in Patients with Diverticulitis During their Hospital Stay. medRxiv. 2020:2020.02.04.20020222.
    
    

18. 18.Pranata R, Soeroto AY, Huang I, Lim MA, Santoso P, Permana H, et al. Effect of chronic obstructive pulmonary disease and smoking on the outcome of COVID-19. Int J Tuberc Lung Dis. 2020;24(8):838–43.
    
    

19. 19.Ozturk S, Turgutalp K, Arici M, Odabas AR, Altiparmak MR, Aydin Z, et al. Mortality analysis of COVID-19 infection in chronic kidney disease, haemodialysis and renal transplant patients compared with patients without kidney disease: a nationwide analysis from Turkey. Nephrol Dial Transplant. 2020;35(12):2083–95.
    
    

20. 20.Hendren NS, Drazner MH, Bozkurt B, Cooper LT, Jr.. Description and Proposed Management of the Acute COVID-19 Cardiovascular Syndrome. Circulation. 2020;141(23):1903–14.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1161/CIRCULATIONAHA.120.047349&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

21. 21.Wang XH, Duan J, Han X, Liu X, Zhou J, Wang X, et al. High incidence and mortality of pneumothorax in critically Ill patients with COVID-19. Heart Lung. 2020.
    
    

22. 22.Li X, Ma X. Acute respiratory failure in COVID-19: is it “typical” ARDS? Crit Care. 2020;24(1):198.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13054-020-02911-9&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

23. 23.Coz Yataco AO, Simpson SQ. Coronavirus Disease 2019 Sepsis: A Nudge Toward Antibiotic Stewardship. Chest. 2020;158(5):1833–4.
    
    

24. 24.Shi L, Wang Y, Wang Y, Duan G, Yang H. Dyspnea rather than fever is a risk factor for predicting mortality in patients with COVID-19. J Infect. 2020;81(4):647–79.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jinf.2020.06.055&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32389782&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

25. 25.Zhao Z, Chen A, Hou W, Graham JM, Li H, Richman PS, et al. Prediction model and risk scores of ICU admission and mortality in COVID-19. PLoS One. 2020;15(7):e0236618.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

26. 26.Hultström M, Persson B, Eriksson O, Lipcsey M, Frithiof R, Nilsson B. Blood type A associates with critical COVID-19 and death in a Swedish cohort. Critical Care. 2020;24(1):496.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13054-020-03223-8&link_type=DOI) 

27. 27.Henry BM, de Oliveira MHS, Benoit S, Plebani M, Lippi G. Hematologic, biochemical and immune biomarker abnormalities associated with severe illness and mortality in coronavirus disease 2019 (COVID-19): a meta-analysis. Clin Chem Lab Med. 2020;58(7):1021–8.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1515/cclm-2020-0369&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

28. 28.Cheng Y, Luo R, Wang K, Zhang M, Wang Z, Dong L, et al. Kidney disease is associated with in-hospital death of patients with COVID-19. Kidney Int. 2020;97(5):829–38.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.kint.2020.03.005&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

29. 29.Altschul DJ, Unda SR, Benton J, de la Garza Ramos R, Cezayirli P, Mehler M, et al. A novel severity score to predict inpatient mortality in COVID-19 patients. Sci Rep. 2020;10(1):16726.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41598-020-73962-9&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

30. 30.Becher T, Frerichs I. Mortality in COVID-19 is not merely a question of resource availability. The Lancet Respiratory medicine. 2020;8(9):832–3.
    
    

31. 31.Rahim F, Amin S, Noor M, Bahadur S, Gul H, Mahmood A, et al. Mortality of Patients With Severe COVID-19 in the Intensive Care Unit: An Observational Study From a Major COVID-19 Receiving Hospital. Cureus. 2020;12(10):e10906–e.
    
    

32. 32.Leung C. Risk factors for predicting mortality in elderly patients with COVID-19: A review of clinical data in China. Mech Ageing Dev. 2020;188:111255-.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.mad.2020.111255&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32353398&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

33. 33.Sterne JAC, Murthy S, Diaz JV, Slutsky AS, Villar J, Angus DC, et al. Association Between Administration of Systemic Corticosteroids and Mortality Among Critically Ill Patients With COVID-19: A Meta-analysis. Jama. 2020;324(13):1330–41.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2020.17023&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F29%2F2020.12.13.20247254.atom) 

34. 34.Zhu JS, Ge P, Jiang C, Zhang Y, Li X, Zhao Z, et al. Deep-learning artificial intelligence analysis of clinical variables predicts mortality in COVID-19 patients. J Am Coll Emerg Physicians Open. 2020:doi:10.1002/emp2.12205.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/emp2.12205&link_type=DOI)