Abstract
Purpose To validate commercially available general-purpose artificial intelligence (AI)-based software for detecting airspace opacity in chest radiographs (CXRs) of COVID-19 patients.
Materials and Methods We used the ieee8023-covid-chestxray-dataset to validate commercial AI software capable of detecting “Nodule/Mass” and “Airspace opacity” as regions of interest with probability scores. From this dataset, we excluded computed tomography images and CXR images taken using an anteroposterior spine view and analyzed CXR images tagged with “Pneumonia/Viral/COVID-19” and “no findings.” A radiologist then reviewed the images and rated them on a 3-point opacity score for the presence of airspace opacity. The maximum probability score of airspace opacity for each image was calculated using this software. The difference in each maximum probability for each opacity score was evaluated using Wilcoxon’s rank sum test. The threshold of the probability score was determined by receiver operator characteristic curve analysis for the presence or absence of COVID-19, and the true positive rate (TPR) and false positive rate (FPR) were determined for the individual and overall opacity scores.
Results Images from 342 patients with COVID-19 and 15 normal images were included. Opacity scores of 1, 2, and 3 were observed in 44, 70, and 243 images, respectively, of which 33 (75%), 66 (94.2%), and 243 (100%), respectively, were from COVID-19 patients. The overall TPR and FPR were 0.82 and 0.13, respectively, at an area under the curve of 0.88 and a threshold of 0.06, while the FPR for opacity score 1 was 0.18 and the TPR for score 3 was 0.97.
Conclusion Using a public database containing CXR images of COVID-19 patients, commercial AI software was shown to be able to detect airspace opacity in severe pneumonia.
Summary Commercially available AI software was capable of detecting airspace opacity in CXR images of COVID-19 patients in a public database.
Competing Interest Statement
Munemura Suzuki is a funder of Suzuki Medical imaging Lab., and an employee of Plusman LLC. Aruta Niimura is an employee of Plusman LLC. Yusuke Nakamura has 2% share of Plusman LLC and is a managing partner of Plusman LLC. Yujiro Otsuka has 2% share of Plusman LLC and is a managing partner of Plusman LLC.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
We used an anonymized public dataset that has already been widely used for AI research on COVID-19. COVID-19 Radiography Database | Kaggle. Accessed December 5, 2021. https://www.kaggle.com/tawsifurrahman/covid19-radiography-database
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors
