Abstract
Background To control the spread of Corona Virus Disease (COVID-19), screening large numbers of suspected cases for appropriate quarantine and treatment measures is a priority. Pathogenic laboratory testing is the diagnostic gold standard but it is time consuming with significant false positive results. Fast and accurate diagnostic methods are urgently needed to combat the disease. Based on COVID-19 radiographical changes in CT images, we hypothesized that deep learning methods might be able to extract COVID-19’s graphical features and provide a clinical diagnosis ahead of the pathogenic test, thus saving critical time for disease control.
Methods We collected 453 CT images of pathogen-confirmed COVID-19 cases along with previously diagnosed with typical viral pneumonia. We modified the Inception migration-learning model to establish the algorithm, followed by internal and external validation.
Findings The internal validation achieved a total accuracy of 82·9% with specificity of 80·5% and sensitivity of 84%. The external testing dataset showed a total accuracy of 73·1% with specificity of 67% and sensitivity of 74%.
Interpretation These results demonstrate the proof-of-principle for using artificial intelligence to extract radiological features for timely and accurate COVID-19 diagnosis.
Funding No funding is involved in the execution of the project.
Evidence before this study Fast and accurate screening large numbers of cases with viral pneumonia symptoms for appropriate quarantine and treatment measures is a priority to control the spread of Corona Virus Disease (COVID-19). Pathogenic laboratory testing is the diagnostic gold standard but it is time-consuming with significant false positive results. Computed tomography (CT) is also a major diagnostic tool for COVID-19. Although typical CT images may help early screening of suspected cases, the images of various viral pneumonias are similar and they overlap with other infectious and inflammatory lung diseases. Artificial intelligent technologies such as deep learning might be able to extract COVID-19’s graphical features and provide a clinical diagnosis. However, there is no published work exploring this possibility.
Added value of this study Our study represents the first study to apply artificial intelligence technologies to CT images for effectively screening for COVID-19. We employed a modified Inception migration-learning model to establish the algorithm. The internal validation achieved a total accuracy of 82·9% with specificity of 80·5% and sensitivity of 84%. The external testing dataset showed a total accuracy of 73·1% with specificity of 67% and sensitivity of 74%. The time for each case is about 2 seconds and can be done remotely via a shared public platform.
Implications of all the available evidence To control the outbreak of COVID-19, it is crucial to develop fast, accurate, safe and non-invasive methods for early diagnosis. We present a proof-of-principle of such as method. We believe that with more data included for further optimization and testing, the accuracy, specificity and sensitivity can be improved. Such a platform can be used to assist clinical diagnosis.
Introduction
The outbreak of atypical and person-to-person transmissible pneumonia caused by the severe acute respiratory syndrome coronavirus 2 (SARS-COV-2, also known as 2019-nCov) has caused a global alarm. There have been nearly 70,000 confirmed cases of the Corona Virus Disease (COVID-19) in China, as of February 16, 2020. According to the WHO, 16-21% of people with the virus in China have become severely ill with a 2-3% mortality rate. With the most recent estimated viral reproduction number (R0), the average number of other people that an infected individual will transmit the virus to in a completely non-immune population, stands at about 3·77 1, indicating that a rapid spread of the disease is imminent. Therefore, it is crucial to identify infected individuals as early as possible for quarantine and treatment procedures. The diagnosis of COVID-19 relies on the following criteria: clinical symptoms, epidemiological history and positive CT images, as well as positive pathogenic testing. The clinical characteristics of COVID-19 include respiratory symptoms, fever, cough, dyspna, and pneumonia 2-5. However, these symptoms are nonspecific, as there are isolated cases where, for example, in an asymptomatic infected family a chest CT scan revealed pneumonia and the pathogenic test for the virus came back positive.
Once someone is identified as a PUI (person under investigation), lower respiratory specimens, such as bronchoalveolar lavage, tracheal aspirate or sputum, will be collected for pathogenic testing. This laboratory technology is based on real-time RT-PCR and sequencing of nucleic acid from the virus 6, 7. Since the beginning of the outbreak, the efficiency of nucleic acid testing has been dependent on several rate-limiting factors, including availability and quantity of the testing kits in the affected area. More importantly, the quality, stability and reproducibility of the detection kits are questionable. The impact of methodology, disease development stages, specimen collection methods, nucleic acid extraction methods and the amplification system are all determinant factors for the accuracy of test results. Conservative estimates of the detection rate of nucleic acid are low (between 30-50%) 6,7,15, and tests need to be repeated several times in many cases before they can be confirmed.
Radiological imaging is also a major diagnostic tool for COVID-19. The majority of COVID-19 cases have similar features on CT images including ground-glass opacities in the early stage and pulmonary consolidation in the late stage. There is also sometimes a rounded morphology and a peripheral lung distribution 5, 8. Although typical CT images may help early screening of suspected cases, the images of various viral pneumonias are similar and they overlap with other infectious and inflammatory lung diseases. Therefore, it is difficult for radiologists to distinguish COVID-19 from other viral pneumonias. Artificial Intelligence involving medical imaging deep-learning systems has been developed in image feature extraction, including shape and spatial relation features. Specifically, Convolutional Neural Network (CNN) has been proven in feature extraction and learning. CNN was used to enhance low-light images from high-speed video endoscopy with the limited training data being just 55 videos 9. Also, CNN has been applied to identify the nature of pulmonary nodules via CT images, the diagnosis of pediatric pneumonia via chest X-ray images, automated precising and labeling of polyps during colonoscopic videos, cystoscopic image recognition extraction from videos 10-13.
There are a number of features for identifying viral pathogens on the basis of imaging patterns, which are associated with their specific pathogenesis 14. The hallmarks of COVID-19 are bilateral distribution of patchy shadows and ground glass opacity 2. Based on this, we believed that CNN might help us identify unique features that might be difficult for visual recognition. To test this notion, we retrospectively enrolled 453 CT images of pathogen-confirmed COVID-19 cases along with previously diagnosed typical viral pneumonia. We trained 217 images using the inception migration-learning model in order to establish the algorithm. We achieved a total accuracy of 83% with specificity of 80·5% and sensitivity of 84% for validation. The external testing showed a total accuracy of 73% with specificity of 67% and sensitivity of 74%. These observations demonstrate the proof-of-principle using the deep learning method to extract radiological graphical features for COVID-19 diagnosis.
Methods and Materials
Retrospective collection of datasets
We retrospectively collected CT images from 99 patients, in which the cohort includes 55 cases of typical viral pneumonia and the other 44 cases from three hospitals with confirmed nucleic acid testing of SARS-COV-2. The hospitals providing the images were Xi’an Jiaotong University First Affiliated Hospital, Nanchang University First Hospital and Xi’an No.8 Hospital of Xi’an Medical College. All CT images were de-identified before sending for analysis. This study is in compliance with the Institutional Review Board of each participating institutes. Informed consent was exempted by the IRB because of the retrospective nature of this study.
Delineation of ROIs
We sketched the region of interest (ROI) on the CT images based on the features of pneumonia, such as ground-glass opacity, mosaic sign and interlobular septal thickening. For a ROI, it is sized approximately from 395*223 to 636*533 pixels. An example of ROI is shown in Figure 2.
Deep learning algorithm framework. ROIs were randomly selected for training using a modified Inception network to extract features. The algorithm then makes a prediction.
An example of COVID-19 pneumonia features. The blue arrow points to ground-glass opacity, and the yellow arrow points to the pleural indentation sign
Overview of the proposed architecture
Our systematic pipeline for the prediction architecture is depicted in Figure 1. The architecture consists of three main processes. 1) Randomly selection of ROIs; 2) Training of the CNN model to extract features; 3) Classification model training of fully connected network and prediction of multiple classifiers.
Intercept ROIs from CT images
We randomly selected delineated ROIs for model building. In order to reduce the computation complexity, and based on the signs of characteristic of pneumonia, regions of interest (ROI) were extracted from CT images. We picked 195 ROIs from 44 COVID-19 positive pneumonia patients and 258 ROIs from 50 COVID-19 negative patients. We built a transfer learning neural network based on Inception network. The entire neural network can be roughly divided into two parts: the first part uses a pre-trained inception network to convert image data into one-dimensional feature vectors, and the second part uses a fully connected network and the main role is for classification prediction. We randomly selected 2-3 pictures from each case to form a training dataset. The number of various types of pictures in the training set is approximately equal, with a total number of 236. The remaining CT pictures of each case were used for internal validation. The model training is iterated 15,000 times with a step size of 0·01. In total, 236 ROIs were used to train the model and 217 ROI were extracted for validation.
Feature extraction using transfer learning
We modified the typical Inception network, and fine-tuned the modified Inception (M-Inception) model with pre-trained weights. During the training phase, the original Inception part was not trained, and we only trained the modified part. The architecture of M-Inception is shown in Table 1. The difference between Inception and M-Inception in classification lies in the last fully-connected layers. We reduced the dimension of features before it was sent to the final classification layer. The training dataset made up of all those patches aforementioned. The Inception network is shown in Table1.
The architecture of M-Inception
Prediction
After generating the features, the final step was to classify the pneumonia based on those features. Ensembling of classifiers was used to improve the classification accuracy. In this study, we combined Decision tree and Adaboost to produce the performance.
Performance evaluation metrics
We compared the classification performance using Accuracy, Sensitivity, Specificity, Area Under Curve (AUC), Positive predictive value (PPV), Negative predictive value (NPV), F1 score and Youden Index. TP and TN represent the number of true positive or true negative samples. FP and FN mean the number of false positive or false negative samples. Sensitivity measures the ratio of positives that are correctly discriminated. Specificity measures the ratio of negatives that are correctly discriminated. AUC is an index to measure the performance of the classifier. NPV was used to evaluate the algorithm for screening, and PPV was the probability of getting a disease when the diagnostic index is positive. Youden Index was the determining exponent of the optimal bound. F1 score was a measure of the accuracy of a binary model. Additionally, performance was evaluated with F-measure (F1) to compare the similarity and diversity of performance. 
Results
Algorithm development
In order to develop a deep learning algorithm for the identification of viral pneumonia images, we retrospectively enrolled 99 patients, in which the cohort includes 55 cases of typical viral pneumonia that were diagnosed previously before the COVID-19 outbreak. These patients are termed COVID-19 negative in the cohort. The other 44 cases are from the three hospitals with confirmed nucleic acid testing of SARS-COV-2, therefore termed COVID-19 positive. Two radiologists were asked to review the images and sketched a total of 453 representative images (258 for COVID-19 negative and 195 for COVID-19 positive) for analysis. These images were randomly divided into a training set and a validation set. First, we randomly selected two or three images from all of the patients’ images for training and the remaining images were used for internal validation. The model training has been iterated for 15,000 times with a step size of 0·01. The training loss curve is shown in Figure 3A. To test the stability and generalization of the model, we also randomly selected 237 images (118 images from COVID-19 negative and 119 images from COVID-19 positive) to construct the model. The remaining images were then used for external validation. The model training has been iterated for 15,000 times with a step size of 0·01. The training loss curve is shown in Figure 3B.
Training loss curves of the models on internal (A) and external (B) validation. The loss curve tends to be stable after descending, indicating that the training process converges.
Deep learning performance
The deep learning algorithm yielded an AUC of 0·90 (95% CI, 0.86 to 0.94) on the internal validation and 0·78 (95% CI, 0.71 to 0.84) on the external validation. The AUC is shown in Figure 4. Using the maximized Youden index threshold probability, the sensitivity was 80·5% and 67·1%, specificity 84·2% and 76·4%, the accuracy was 82·9% and 73·1%, the negative prediction values were 0·88 and 0·81, the Youden indexes were 0·69 and 0·44, and the F1 scores were 0·77 and 0·64 for the internal and external datasets, respectively (Table 2). The algorithm was executed at a rate of 2 seconds per case on the graphic processing unit.
Deep learning Algorithm Performance
Receiver operating characteristic plots for COVID-19 identification for the deep learning (Inception) algorithm.
Discussion
Timely diagnosis and triaging of PUIs are crucial for the control of emerging infectious diseases such as the current COVID-19. Due to the limitation of nucleic acid -based laboratory testing, there is an urgent need to look for fast alternative methods that can be used by front-line health care personals for quickly and accurately diagnosing the disease. In the present study, we have developed an AI program by analyzing representative CT images using a deep learning method. This is a retrospective, multicohort, diagnostic study using our modified Inception migration neuro network, which has achieved 82.9% accuracy. Moreover, the high performance of the deep learning model we developed in this study was tested using external samples with 73% accuracy. These findings have demonstrated the proof-of-principle that deep learning can extract CT image features of COVID-19 for diagnostic purposes. The time for each case is about 2 seconds and can be done remotely via a shared public platform. Therefore, further developing this system can significantly shorten the diagnosis time for disease control. Our study represents the first study to apply artificial intelligence technologies to CT images for effectively screening for COVID-19.
The gold standard for COVID-19 diagnosis has been nucleic acid based detection for the existence of specific sequences of the SARS-COV-2 gene. While we still value the importance of nucleic acid detection in the diagnosis of the viral infection, we must also note that the high number of false negatives due to several factors such as methodological disadvantages, disease stages, and methods for specimen collection might delay diagnosis and disease control. Recent data have suggested that the accuracy of nucleic acid testing is only about 30-50% 6,7,15. Using CT imaging feature extraction, we are able to achieve above 83% accuracy, significantly outplaying nucleic acid testing. In addition, this method is non-invasive with minimal cost. Although we are satisfied with the initial results, we believe that with more CT images included in the training, we will achieve higher accuracy. Therefore, further optimizing and testing this system is warranted. To achieve this, we have generated a webpage that licensed healthcare personnel can access to upload CT images for testing and validation. The webpage information is as following: https://ai.nscc-tj.cn/thai/deploy/public/pneumonia_ct.
There are some limitations to our study. CT images present a difficult classification task due to the relatively large number of variable objects, specifically the imaged areas outside the lungs that are irrelevant to the diagnosis of pneumonia 11. In our study, only two radiologists were involved in outlining the ROI area. In addition, the training data set is relatively small. The performance of this system is expected to increase when the training volume is increased. It should also be noted that, the features of the CT images we analyzed are from patients with severe lung lesions at later stages of disease development. A study to associate this with the disease progress and all pathologic stages of COVID-19 is necessary to optimize the diagnostic system.
In future, we intend to link hierarchical features of CT images to features of other factors such as genetic, epidemiological and clinical information for multi-modeling analysis for an enhanced diagnosis. The artificial intelligence system developed in our study should significantly contribute to COVID-19 disease control by reducing the number of PUIs for timely quarantine and treatment.
Contributors
BX and XF designed the study and took responsibility for the integrity of the data and the accuracy of the data analysis. SW, BK, MX and JG contributed to the data analysis. BX, SW, MX and BK contributed to writing of the manuscript. JM, XZ, MC, JY, YL contributed to the collection of data. All authors contributed to data interpretation, and reviewed and approved the final version.
Data Availability
All data referred to in the manuscript is available.
Declaration of interests
All authors declare no competing interests.
Data sharing
The data that support the findings of this study are available from the corresponding authors upon request. Participant data are without names and identifiers, and they will be made available after approval from the corresponding author and National Health Commission. After publication of study findings, the data will be available for others to request. The research team will provide an email address for communication once the data are approved for sharing.
Acknowledgments
We thank all members of the Xu Laboratory (TMUCIH) and Tianhe Supercomputer Team (NSCC-Tianjin) for their help and valuable comments.
Footnotes
↵* Equal contribution
