Accuracy of deep learning based computed tomography diagnostic system of COVID-19: a consecutive sampling external validation cohort study

Tatsuyoshi Ikenoue; Yuki Kataoka; Yoshinori Matsuoka; Junichi Matsumoto; Junji Kumasawa; Kentaro Tochitatni; Hiraku Funakoshi; Tomohiro Hosoda; Aiko Kugimiya; Michinori Shirano; Fumiko Hamabe; Sachiyo Iwata; Shingo Fukuma; Japan COVID-19 AI team

doi:10.1101/2020.11.15.20231621

Abstract

Objectives Ali-M3, an artificial intelligence, analyses chest computed tomography (CT) and detects the likelihood of coronavirus disease (COVID-19) in the range of 0 to 1. It demonstrates excellent performance for the detection of COVID-19 patients with a sensitivity and specificity of 98.5 and 99.2%, respectively. However, Ali-M3 has not been externally validated. Our purpose is to evaluate the external validity of Ali-M3 using Japanese sequential sampling data.

Methods In this retrospective cohort study, COVID-19 infection probabilities were calculated using Ali-M3 in 617 symptomatic patients who underwent reverse transcription-polymerase chain reaction (RT-PCR) tests and chest CT for COVID-19 diagnosis at 11 Japanese tertiary care facilities, between January 1 and April 15, 2020.

Results Of 617 patients, 289 patients (46.8%) were RT-PCR-positive. The area under the curve (AUC) of Ali-M3 for predicting a COVID-19 diagnosis was 0.797 (95% confidence intervals [CI]: 0.762-0.833) and goodness-of-fit was P = 0.156. With a cut-off of probability of COVID-19 by Ali-M3 diagnosis set at 0.5, the sensitivity and specificity were 80.6% and 68.3%, respectively, while a cut-off of 0.2 yielded a sensitivity and specificity of 89.2% and 43.2%, respectively. Among 223 patients who required oxygen support, the AUC was 0.825 and sensitivity at a cut-off of 0.5 and 0.2 were 88.7% and 97.9%, respectively. Although the sensitivity was lower when the days from symptom onset were few, sensitivity increased for both cut-off values after 5 days.

Conclusions Ali-M3 was evaluated by external validation and shown to be useful to exclude a diagnosis of COVID-19.

Key Points

The area under the curve (AUC) of Ali-M3, which is an AI system for diagnosis of COVID-19 based on chest CT images, was 0.797 and goodness-of-fit was P = 0.156.
With a cut-off of probability of COVID-19 by Ali-M3 diagnosis set at 0.5, the sensitivity and specificity were 80.6% and 68.3%, respectively, while a cut-off of 0.2 yielded 89.2% and 43.2%.
Although low sensitivity was observed in less number of days from symptoms onset, after 5 days high increasing sensitivity was observed. In patients requiring oxygen support, the AUC was higher that is 0.825.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Hyogo Prefectural Amagasaki General Medical Center, Kyoto University Graduate School of Medicine, Kobe City Medical Center General Hospital, St. Marianna University School of Medicine, Sakai City Medical Center, Kyoto City Hospital, Tokyobay Urayasu Ichikawa Medical Center, Kawasaki Municipal Kawasaki Hospital, Yamanashi Prefectural Central Hospital, Osaka City General Hospital, National Defense Medical College Hospital, Hyogo Prefectural Kakogawa Medical Center

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

The data that support the findings of this study are available from the corresponding author, Tatsuyoshi Ikenoue, upon permission of IRB at Hyogo Prefectural Amagasaki General Medical Center and reasonable request.