Development and Validation of a Diagnostic Nomogram to Predict COVID-19 Pneumonia ================================================================================= * Zhiyi Wang * Jie Weng * Zhongwang Li * Ruonan Hou * Lebin Zhou * Hua Ye * Ying Chen * Ting Yang * Daqing Chen * Liang Wang * Xiaodong Liu * Xian Shen * Shengwei Jin ## Abstract **Background** The COVID-19 virus is an emerging virus rapidly spread worldwide This study aimed to establish an effective diagnostic nomogram for suspected COVID-19 pneumonia patients. **METHODS** We used the LASSO aggression and multivariable logistic regression methods to explore the predictive factors associated with COVID-19 pneumonia, and established the diagnostic nomogram for COVID-19 pneumonia using multivariable regression. This diagnostic nomogram was assessed by the internal and external validation data set. Further, we plotted decision curves and clinical impact curve to evaluate the clinical usefulness of this diagnostic nomogram. **RESULTS** The predictive factors including the epidemiological history, wedge- shaped or fan-shaped lesion parallel to or near the pleura, bilateral lower lobes, ground glass opacities, crazy paving pattern and white blood cell (WBC) count were contained in the nomogram. In the primary cohort, the C-statistic for predicting the probability of the COVID-19 pneumonia was 0.967, even higher than the C-statistic (0.961) in initial viral nucleic acid nomogram which was established using the univariable regression. The C-statistic was 0.848 in external validation cohort. Good calibration curves were observed for the prediction probability in the internal validation and external validation cohort. The nomogram both performed well in terms of discrimination and calibration. Moreover, decision curve and clinical impact curve were also beneficial for COVID- 19 pneumonia patients. **CONCLUSION** Our nomogram can be used to predict COVID-19 pneumonia accurately and favourably. ## Introduction Corona Virus Disease 2019 (COVID-19) pneumonia is confirmed to be infected with a novel coronavirus which is a β coronavirus that belongs to the family Coronaviridae. It has spread rapidly throughout Wuhan (Hubei province) to other provinces in China and around the world. 1,2 On January 30, 2020, the World Health Organization declared COVID-19 was a public health emergency of international concern (PHEIC). As of March 13, 2020, a total of 51767 laboratory-confirmed patients and 1775 deaths have been documented outside China. This pandemic has been disastrous for people all over the world, and many countries are failing to control its spread. The main reason was lack of rapid response which is depend on very early detection and diagnosis3. As with all infectious diseases, the early and reliable diagnosis is key to block COVID-19 transmission. However, COVID-19 pneumonia has a wide range of clinical manifestations, such as fever, cough, fatigue, pharyngeal pain, etc4, Many patients have mild symptoms or asymptomatic in the early stage. Due to lack of specific clinical symptoms and signs, it is easy to be ignored.5,6 Real-time polymerase chain reaction (PCR) has become an important tool in the diagnosis of many infectious diseases7,8. However, it has several disadvantages, including long detection time, cumbersome steps, and high cost. Even the sensitivity of the COVID-19 nucleic acid detection is low. In addition, COVID-19 nucleic acid detection kits are still insufficient in many epidemic countries now. Detection of viral nucleic acid may not be the only ideal diagnostic method during the early stages of this epidemic outbreak. Given the rapid spread of COVID-19 and low detection rate by pharyngeal swab COVID-19 nucleic acid test. It is also believed that throat swab samples are not suitable for the detection of SARS coronavirus RNA9. The development of a diagnostic method with decreased complexity and expense are urgently needed to facilitate timely intervention.10 Previous studies with significantly large sample sizes have been done to delineate the epidemiological and clinical characteristics of COVID- 19 pneumonia.11 Although some studies have shown that lung computed tomographic (CT) plays an important role in early diagnosis12,13. However, there is still a lack of systematic and standard imaging diagnostic criteria for COVID-19 pneumonia. To this end, 294 suspected COVID-19 pneumonia patients were included in our study. Trying to establish a diagnosis model to quickly identify COVID-19 pneumonia from suspected COVID-19 pneumonia. ## Methods ### Study design and participants Two cohorts of adult suspected COVID-19 pneumonia patients were included in this retrospective study. The patients in primary cohort were from the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University (Wenzhou, China), and the patients in validation cohort were from the People’s Hospital of Yueqing (Yueqing, China). We relied on new coronavirus pneumonia control and prevention plan (trial version 6) to identify suspected COVID-19 pneumonia patients. Suspected diagnostic criteria are as follows:1) epidemiological history. 2) fever and / or respiratory symptoms. 3) suspected COVID-19 pneumonia imaging features. 4) normal range or decrease of white blood cell (WBC) count or lymphocyte at beginning of disease. Patients who have epidemiological history + any two clinical symptoms or no epidemiological history + three clinical symptoms can be diagnosed as suspected COVID-19 pneumonia. All patients with suspected COVID-19 pneumonia were taken throat swab samples or sputum sample at admission (at least twice samples were taken, at least a 24 h apart) and stored in virus medium, which were transported to Wenzhou Center for Disease Control and Prevention (CDC) for COVID-19 diagnosis. The study was approved by the Ethics Committee of the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University. ### Data collection Demographic data, epidemiological history, comorbidity, vital signs, clinical symptoms, laboratory indicators including WBC count, neutrophils count, lymphocyte count, hemoglobin, platelet and C-reactive protein (CRP) and chest CT imaging features were collected. The data of laboratory indicators and chest CT imaging features were the first recorded data. ### Image acquisition and analysis All chest CT images features in two cohorts were analyzed by four chest radiologists (two radiologists for one cohort) with least 10 years of experience in chest CT imaging and all decisions were reached by consensus. CT images features mainly include four parts: (1) lesion distribution, such as bilateral lower lobes, multiple lobes, periphery distribution (2) lesion patterns, such as patchy or large patchy distribution, wedge-shaped or fan-shaped lesion parallel to or near the pleura, crazy paving pattern, (3) lesion density, such as ground glass opacities, consolidation, cavitation, (4) other signs in the lesion, such as lung nodule, nodules halo sign, subpleural nodules, centrilobular nodules, other nodules, pleural effusion, air bronchogram sign, bronchiectasis, fibrotic proliferation, bronchial wall thickening, tree-in-bud pattern, white lung (Figure 1). ![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/04/06/2020.04.03.20052068/F1.medium.gif) [Figure 1:](http://medrxiv.org/content/early/2020/04/06/2020.04.03.20052068/F1) Figure 1: Classical COVID-19 pneumonia CT imaging features. Crazy paving pattern (A and B), wedge-shaped or fan-shaped lesion parallel to or near the pleura (C and D), Groun d glass opacities (E and F). ### Statistical analysis Data were presented as mean ± standard deviation (SD) and median (IQR) for continuous variables with normal or non-normal distribution and the frequency (proportion) for categorical variables. The Student t-test, Mann-Whitney U test, chi- squared test, or Fisher’s exact test was performed where appropriate. For the development of the nomograms, the least absolute shrinkage and selection operator (LASSO) method was performed to identify potential significant predictors from the primary cohort. Predictive variables that were considered clinically relevant (based on our clinical experience and literature report), and that showed statistical relationship in LASSO method were entered into multivariate logistic regression model14. The Variance Inflation Factor (VIF) was used to identify variables collinearity before the model estimation. According to the Akaike information criterion (AIC) and clinically relevant variables, we established the final multivariable models. The ‘rms’ package was used for nomogram and calibration curve15. The accuracy of the nomogram to predict the COVID-19 pneumonia were quantified using C-statistic, the calibration of the model is assessed by the calibration curves in the primary cohort and validation cohort. Moreover, we performed decision curve analysis (DCA) and clinical impact curve by quantifying the net benefits and cost benefit ratio to assess the clinical value of the model16. We did the statistical analyses and figures production using R software (version 3.6.1). All statistical tests were two-sided, differences of P < 0.05 were considered statistically significant. ## Results The primary cohort consisted of 178 suspected COVID-19 pneumonia patients who received treatment in our hospital between Jan 25, 2020, and Mar 3, 2020. 89 patients were diagnosed with COVID-19 pneumonia (7 COVID-19 pneumonia patients were negative or weakly positive in initial viral nucleic acid detection). The validation cohort consisted of 116 suspected COVID-19 pneumonia patients. 68 patients were diagnosed with COVID-19 pneumonia (no COVID-19 pneumonia patients were negative or weakly positive in initial viral nucleic acid detection). No deaths occurred in the two cohorts. Patient characteristics, clinical symptoms, laboratory findings and chest CT imaging features were show in Table 1. View this table: [Table 1:](http://medrxiv.org/content/early/2020/04/06/2020.04.03.20052068/T1) Table 1: patients demographic, comorbidity, clinical symptoms, blood parameters and imaging features Data are mean±SD, median (IQR) or n (%). *p values compare the primary cohort validation cohort and using the Student t-test, Mann-Whitney U test, chi-squared test, or Fisher’s exact test was performed where appropriate. CRP= C-reaction protein, WBC= white blood cell. According to the univariate analysis (Supplementary materials), 16 related variables (gender, epidemiological history, two laboratory indicators: WBC and neutrophils count, twelve imaging features: multiple lobes, bilateral lower lobes, periphery distribution, nodules halo sign, patchy opacities, wedge-shaped or fan-shaped lesion parallel to or near the pleura, crazy paving pattern, ground glass opacities, consolidation, pleural effusion, air bronchogram sign and fibrotic proliferation) were included in the LASSO regression analysis(Figure 2). The results showed that gender, epidemiological history, neutrophils count, bilateral lower lobes, periphery distribution, wedge-shaped or fan-shaped lesion parallel to or near the pleura, ground glass opacities obtained from the primary cohort were predictive factors for COVID-19 pneumonia. According to the results of collinearity test and clinical practice, the epidemiological history, wedge-shaped or fan-shaped lesion parallel to or near the pleura, bilateral lower lobes, ground glass opacities, crazy paving pattern and WBC count were included in the final multivariate logistic regression analyses. ![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/04/06/2020.04.03.20052068/F2.medium.gif) [Figure 2:](http://medrxiv.org/content/early/2020/04/06/2020.04.03.20052068/F2) Figure 2: Predictive factors for COVID-19 pneumonia were selected by LASSO binary regression analyses. (A) The curve of the coefficient path of 16 predictive factors in primary cohort. The vertical line was set at the nonzero coefficients selected via 10-fold cross-validation, and there are 7 variables are included. (B) The adjustment penalty parameter *λ*was selected in the LASSO model by 10-fold cross-validation. To provide a quantitative method to estimated COVID-19 pneumonia patients probability for clinician, we developed a nomogram (the ‘rms’ package was used in the R software). This nomogram had a C-statistic of 0.967 (95% confidence interval [CI], 0.943–0.992) for predicting the probability in suspected COVID-19 pneumonia patients (Figure 3). The radiomics nomogram was further internal validated using bootstrapping and external validated based on validation cohort. Good calibration curves were observed for the probability in suspected COVID-19 pneumonia patients in the internal validation data set and external validation cohort. The C-statistic of the radiomics nomogram were 0.848 in external validation cohort (Figure 4). Meanwhile, we developed another nomogram, based on initial viral nucleic acid detection, on the basis of univariable logistic analysis. The viral nucleic acid nomogram had a C-statistic of 0.961 (95% CI: 0.933–0.989) (Supplementary materials). Our nomogram showed higher discrimination, as compared with initial viral nucleic acid detection. ![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/04/06/2020.04.03.20052068/F3.medium.gif) [Figure 3:](http://medrxiv.org/content/early/2020/04/06/2020.04.03.20052068/F3) Figure 3: Developed radiomics nomogram. This nomogram was developed with four imaging features, epidemiological history and WBC count. ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/04/06/2020.04.03.20052068/F4.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2020/04/06/2020.04.03.20052068/F4) Figure 4: Calibration curves for predicting COVID-19 pneumonia probability by the nomogram in the in primary cohort (A) and validation cohort (B). Y-axis represents the actual COVID-19 pneumonia probability value, X-axis represents the predicted COVID-19 pneumonia probability value. Black solid line represents the prediction performance of the nomogram, the diagonal gray line represents an ideal nomogram model. The DCA showed that radiomics nomogram had a very large threshold probability range, which means this nomogram had excellent net benefit to the outcome of suspected COVID-19 pneumonia patients. Clinical impact curve of the nomogram showed the predicted probability of COVID-19 pneumonia was very close to the actual COVID-19 pneumonia (Figure 5). ![Figure 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/04/06/2020.04.03.20052068/F5.medium.gif) [Figure 5:](http://medrxiv.org/content/early/2020/04/06/2020.04.03.20052068/F5) Figure 5: DCA of the nomogram model (A). The Y-axis represents the net benefit. The red line represents the predicted COVID-19 pneumonia nomogram model. Clinical impact curve of the nomogram model (B). The nomogram model is used to predict risk stratification for 1000 people. The red line represents the number of people classified high-risk by nomogram model under different threshold probability; the blue curve (Number high risk with event) is the number of truly positive people under different threshold probability. ## Discussion In this study, we developed and validated a novel nomogram to predict COVID- 19 infection among patients who were suspected viral pneumonia. This diagnostic nomogram mainly relies on CT findings. Our study found that COVID-19 pneumonia is not significantly different from other suspected viral pneumonia in clinical symptoms and signs. There was also no significant difference in blood routine test, liver and kidney function test. Thus, according to clinical symptoms and signs and laboratory examinations, the COVID-19 pneumonia is difficult to distinguish from other viral pneumonia. However, they have similar and different manifestations on lung imaging detected by CT scan. 10,17 To this end, we have established this nomogram mainly based on lung imaging. All CT findings were analyzed from three aspects including distribution characteristics, morphology and density of pulmonary inflammation lesions. The imaging features of viral pneumonia usually appear as multifocal ground glass opacities which correspond to pathological diffuse alveolar damage.18According to univariate analysis, ground glass opacities, crazy paving pattern, a wedge-shaped or fan-shaped lesion parallel to or near the pleura, the distribution characteristics of bilateral lower lobes and peripheral distribution of lesions are the characteristic imaging manifestation of COVID-19 pneumonia. But, the crazy paving pattern on the basis of multivariable was unassociated with COVID-19 pneumonia, which may be due to other potential confounding factors.19 But this does not mean that crazy paving pattern in COVID-19 pneumonia are unimportant. In addition, many studies have shown that the crazy paving pattern formed by interlobular septal thickening which was regarded as a typical imaging of viral pneumonia13. Therefore, we kept this factor in our model development. Another important imaging feature related to the characteristics of morphology and distribution, the wedge-shaped or fan-shaped lesion parallel to or near the pleura, which actually includes peripheral distribution of lesions. Because of their strong collinearity, it will seriously affect the accuracy of our research results20. Therefore, the imaging features of peripheral distribution was removed. No matter from univariate or multivariate analysis results, epidemiological history plays a central role in the clinical diagnosis of COVID-19 pneumonia. Lymphocyte count did not show enough predictive strength. It may be related to the fact that these patients in our study are mainly suspected virus infection, and their lymphocyte count are generally low, with little difference. In view of the WBC of patients with viral infection is usually not high21,22, although it was unassociated in multivariate analysis, we still kept it in the process of establishing the diagnostic nomogram. Finally, the nomogram incorporates 4 items of the imaging features, epidemiological contact history and WBC count status. Nomogram is a visualization of regression analysis, which is more and more widely used in clinical disease diagnosis, prognosis evaluation and efficacy evaluation23-26. Our results show that the nomogram based on imaging features has good sensitivity and specificity in the diagnosis of COVID-19 pneumonia. Moreover, its discrimination for COVID-19 pneumonia is better than the first detection of viral nucleic acid. If there is a lack of virus nucleic acid test kit, COVID-19 pneumonia can be determined by lung CT preferentially. In order to prove the calibration of the nomogram, clinical data was collected from different institutions. As is well known, the internal validity associated with the explanation of the results, and the external validity related to the generalizability of the results27,28. Through the internal and external validation data set analysis, the calibration of our nomogram has been proved to be highly consistent. This means that our nomogram may be popularized and applied widely in other hospital. However, to evaluate its clinical usefulness, it depends on how much it benefits the patient, not just its popularization29. DCA is an novel method30,31, it offers insight into clinical consequences on the basis of threshold probability, from which the net benefit could be derived32. The DCA showed that if we choose to diagnose COVID-19 pneumonia with a 60% threshold probability, 40 out of every 100 people will benefit. ### Limitations of this study Our study has several limitations. Firstly, only 178 patients were included in primary cohort and another hospital was selected for external validation (116 patients). Whether this nomogram is applicable to patients with other areas background is still unclear. A large number of patients as data need to be collected to verify its clinical application. Secondly, this nomogram is mainly used to identify COVID-19 pneumonia in the patients with suspected viral pneumonia, not all types of pneumonia. Although the decrease of lymphocyte count is more common among COVID-19 pneumonia, not observed in our study. It may be related to our inclusion criteria. ## Conclusion In conclusion, this study presents a novel nomogram that incorporates both the imaging features, epidemiological history and WBC. It can predict COVID-19 pneumonia conveniently and accurately. Using this nomogram has high net benefit for patients with suspected COVID-19 infection. ## Data Availability All data referred to in the manuscript is available. ## FUNDING None. ## AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST The author(s) indicated no potential conflict of interest. ## AUTHOR CONTRIBUTION Conception and design: Zhiyi Wang, Daqing Chen, Liang Wang, Xiaodong Liu, Xian Shen, Shengwei Jin Collection and assembly of data: Jie weng, Zhongwang Li, Ruonan Hou, Lebin Zhou,Hua Ye,Ying Chen, Ting Yang Zhiyi Wang1,2*#, Jie Weng1*, Zhongwang Li3, Ruonan Hou1, Lebin Zhou4, Hua Ye4, Ying Chen1, Ting Yang1, Daqing Chen1, Liang Wang5, Xiaodong Liu2, Xian Shen6#, Shengwei Jin3# * Received April 3, 2020. * Revision received April 3, 2020. * Accepted April 6, 2020. * © 2020, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.Phelan AL, Katz R, Gostin LO. The Novel Coronavirus Originating in Wuhan, China: Challenges for Global Health Governance. JAMA 2020. 2. 2.Zhu N, Zhang D, Wang W, et al. A Novel Coronavirus from Patients with Pneumonia in China, 2019. The New England journal of medicine 2020; 382(8): 727–33. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa2001017&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F04%2F06%2F2020.04.03.20052068.atom) 3. 3.MacIntyre CR, Costantino V, Kunasekaran MP. Health system capacity in Sydney, Australia in the event of a biological attack with smallpox. PLoS One 2019; 14(6): e0217704. 4. 4.Liu K, Fang YY, Deng Y, et al. Clinical characteristics of novel coronavirus cases in tertiary hospitals in Hubei Province. Chin Med J (Engl) 2020. 5. 5.Huang C, Wang Y, Li X, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The Lancet 2020; 395(10223): 497–506. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F04%2F06%2F2020.04.03.20052068.atom) 6. 6.Xu XW, Wu XX, Jiang XG, et al. Clinical findings in a group of patients infected with the 2019 novel coronavirus (SARS-Cov-2) outside of Wuhan, China: retrospective case series. BMJ 2020; 368: m606. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzNjgvZmViMTlfNC9tNjA2IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjAvMDQvMDYvMjAyMC4wNC4wMy4yMDA1MjA2OC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 7. 7.Krejmer-Rabalska M, Rabalski L, Jukes MD, Lobo de Souza M, Moore SD, Szewczyk B. New Method for Differentiation of Granuloviruses (Betabaculoviruses) Based on Real-Time Polymerase Chain Reaction (Real-Time PCR). Viruses 2019; 11(2). 8. 8.Kelly-Cirino C, Mazzola LT, Chua A, Oxenford CJ, Van Kerkhove MD. An updated roadmap for MERS-CoV research and product development: focus on diagnostics. BMJ Glob Health 2019; 4(Suppl 2): e001105. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiYm1qZ2giO3M6NToicmVzaWQiO3M6MTc6IjQvU3VwcGxfMi9lMDAxMTA1IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjAvMDQvMDYvMjAyMC4wNC4wMy4yMDA1MjA2OC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 9. 9.Hong TC, Mai QL, Cuong DV, et al. Development and evaluation of a novel loop-mediated isothermal amplification method for rapid detection of severe acute respiratory syndrome coronavirus. J Clin Microbiol 2004; 42(5): 1956–61. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNtIjtzOjU6InJlc2lkIjtzOjk6IjQyLzUvMTk1NiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzA0LzA2LzIwMjAuMDQuMDMuMjAwNTIwNjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 10. 10.Xia W, Shao J, Guo Y, Peng X, Li Z, Hu D. Clinical and CT features in pediatric patients with COVID-19 infection: Different points from adults. Pediatr Pulmonol 2020. 11. 11.Li Q, Guan X, Wu P, et al. Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus-Infected Pneumonia. N Engl J Med 2020. 12. 12.Pan Y, Guan H, Zhou S, et al. Initial CT findings and temporal changes in patients with the novel coronavirus pneumonia (2019-nCoV): a study of 63 patients in Wuhan, China. European radiology 2020. 13. 13.Xu X, Yu C, Qu J, et al. Imaging and clinical features of patients with 2019 novel coronavirus SARS-CoV-2. Eur J Nucl Med Mol Imaging 2020. 14. 14.Sun K, Huang SH, Wong DS, Jang SS. Design and Application of a Variable Selection Method for Multilayer Perceptron Neural Network With LASSO. IEEE Trans Neural Netw Learn Syst 2017; 28(6): 1386–96. 15. 15.Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol 2013; 13: 33. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2288-13-33&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23496923&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F04%2F06%2F2020.04.03.20052068.atom) 16. 16.Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res 2019; 3: 18. 17. 17.Koo HJ, Lim S, Choe J, Choi SH, Sung H, Do KH. Radiographic and CT Features of Viral Pneumonia. Radiographics 2018; 38(3): 719–39. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1046/j.1360-2276.2003.01133.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F04%2F06%2F2020.04.03.20052068.atom) 18. 18.Chong S, Kim TS, Cho EY. Herpes simplex virus pneumonia: high-resolution CT findings. Br J Radiol 2010; 83(991): 585–9. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYmpyYWRpbyI7czo1OiJyZXNpZCI7czoxMDoiODMvOTkxLzU4NSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzA0LzA2LzIwMjAuMDQuMDMuMjAwNTIwNjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 19. 19.Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015; 350: g7594. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1136/bmj.g7594&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25569120&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F04%2F06%2F2020.04.03.20052068.atom) 20. 20.Claret PG, Bobbia X, de La Coussaye JE. Collinearity and multivariable analysis. Intensive Care Med 2016; 42(11): 1834. 21. 21.Wang Z, Chen X, Lu Y, Chen F, Zhang W. Clinical characteristics and therapeutic procedure for four cases with 2019 novel coronavirus pneumonia receiving combined Chinese and Western medicine treatment. Biosci Trends 2020; 14(1): 64–8. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.5582/bst.2020.01030&link_type=DOI) 22. 22.Jung YJ, Kim JH, Park YJ, et al. Evaluation of cell population data on the UniCel DxH 800 Coulter Cellular Analysis system as a screening for viral infection in children. Int J Lab Hematol 2012; 34(3): 283–9. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22226427&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F04%2F06%2F2020.04.03.20052068.atom) 23. 23.Callegaro D, Miceli R, Mariani L, Raut CP, Gronchi A. Soft tissue sarcoma nomograms and their incorporation into practice. Cancer 2017; 123(15): 2802–20. 24. 24.Chen Z, Lin RM, Bai YK, Zhang Y. Establishment and Verification of Prognostic Nomograms for Patients with Gastrointestinal Stromal Tumors: A SEER-Based Study. Biomed Res Int 2019; 2019: 8293261. 25. 25.Wang Y, Sun K, Shen J, et al. Novel Prognostic Nomograms Based on Inflammation-Related Markers for Patients with Hepatocellular Carcinoma Underwent Hepatectomy. Cancer Res Treat 2019; 51(4): 1464–78. 26. 26.Zhou Z, Mo S, Dai W, et al. Prognostic nomograms for predicting cause-specific survival and overall survival of stage I-III colon cancer patients: a large population-based study. Cancer Cell Int 2019; 19: 355. 27. 27.Hong JY, You JS, Kim MJ, et al. Development and external validation of new nomograms by adding ECG changes (ST depression or tall T wave) and age to conventional scoring systems to improve the predictive capacity in patients with subarachnoid haemorrhage: a retrospective, observational study in Korea. BMJ Open 2019; 9(2): e024007. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOS8yL2UwMjQwMDciO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8wNC8wNi8yMDIwLjA0LjAzLjIwMDUyMDY4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 28. 28.Huebschmann AG, Leavitt IM, Glasgow RE. Making Health Research Matter: A Call to Increase Attention to External Validity. Annu Rev Public Health 2019; 40: 45–63. 29. 29.Huang YQ, Liang CH, He L, et al. Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer. Journal of clinical oncology : official journal of the American Society of Clinical Oncology 2016; 34(18): 2157–64. 30. 30.Talluri R, Shete S. Using the weighted area under the net benefit curve for decision curve analysis. BMC Med Inform Decis Mak 2016; 16: 94. 31. 31.Prabhu V, Rosenkrantz AB, Otazo R, Sodickson DK, Kang SK. Population net benefit of prostate MRI with high spatiotemporal resolution contrast-enhanced imaging: A decision curve analysis. J Magn Reson Imaging 2019; 49(5): 1400–8. 32. 32.Balachandran VP, Gonen M, Smith JJ, DeMatteo RP. Nomograms in oncology: more than meets the eye. Lancet Oncol 2015; 16(4): e173–80. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1470-2045(14)71116-7&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25846097&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F04%2F06%2F2020.04.03.20052068.atom)