PT - JOURNAL ARTICLE AU - He, Xin AU - Wang, Shihao AU - Shi, Shaohuai AU - Chu, Xiaowen AU - Tang, Jiangping AU - Liu, Xin AU - Yan, Chenggang AU - Zhang, Jiyong AU - Ding, Guiguang TI - Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans AID - 10.1101/2020.06.08.20125963 DP - 2020 Jan 01 TA - medRxiv PG - 2020.06.08.20125963 4099 - http://medrxiv.org/content/early/2020/06/17/2020.06.08.20125963.short 4100 - http://medrxiv.org/content/early/2020/06/17/2020.06.08.20125963.full AB - COVID-19 pandemic has spread all over the world for months. As its transmissibility and high pathogenicity seriously threaten people’s lives, the accurate and fast detection of the COVID-19 infection is crucial. Although many recent studies have shown that deep learning based solutions can help detect COVID-19 based on chest CT scans, there lacks a consistent and systematic comparison and evaluation on these techniques. In this paper, we first build a clean and segmented CT dataset called Clean-CC-CCII by fixing the errors and removing some noises in a large CT scan dataset CC-CCII with three classes: novel coronavirus pneumonia (NCP), common pneumonia (CP), and normal controls (Normal). After cleaning, our dataset consists of a total of 340,190 slices of 3,993 scans from 2,698 patients. Then we benchmark and compare the performance of a series of state-of-the-art (SOTA) 3D and 2D convolutional neural networks (CNNs). The results show that 3D CNNs outperform 2D CNNs in general. With extensive effort of hyperparameter tuning, we find that the 3D CNN model DenseNet3D121 achieves the highest accuracy of 88.63% (F1-score is 88.14% and AUC is 0.940), and another 3D CNN model ResNet3D34 achieves the best AUC of 0.959 (accuracy is 87.83% and F1-score is 86.04%). We further demonstrate that the mixup data augmentation technique can largely improve the model performance. At last, we design an automated deep learning methodology to generate a lightweight deep learning model MNas3DNet41 that achieves an accuracy of 87.14%, F1-score of 87.25%, and AUC of 0.957, which are on par with the best models made by AI experts. The automated deep learning design is a promising methodology that can help health-care professionals develop effective deep learning models using their private data sets. Our Clean-CC-CCII dataset and source code are available at:https://github.com/arthursdays/HKBU HPML COVID-19.Competing Interest StatementThe authors have declared no competing interest.Clinical Protocols https://github.com/arthursdays/HKBU_HPML_COVID-19 Funding StatementnilAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:NilAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesOur data is obtained from a publicly available dataset, i.e., http://ncov-ai.big.ac.cn/download?lang=en, which is under a Creative Commons Attribution 3.0 China Mainland License. https://github.com/arthursdays/HKBU_HPML_COVID-19 http://ncov-ai.big.ac.cn/download?lang=en