PT  - JOURNAL ARTICLE
AU  - Ikenoue, Tatsuyoshi
AU  - Kataoka, Yuki
AU  - Matsuoka, Yoshinori
AU  - Matsumoto, Junichi
AU  - Kumasawa, Junji
AU  - Tochitatni, Kentaro
AU  - Funakoshi, Hiraku
AU  - Hosoda, Tomohiro
AU  - Kugimiya, Aiko
AU  - Shirano, Michinori
AU  - Hamabe, Fumiko
AU  - Iwata, Sachiyo
AU  - Fukuma, Shingo
AU  - ,
TI  - Accuracy of deep learning based computed tomography diagnostic system of COVID-19: a consecutive sampling external validation cohort study
AID  - 10.1101/2020.11.15.20231621
DP  - 2020 Jan 01
TA  - medRxiv
PG  - 2020.11.15.20231621
4099  - http://medrxiv.org/content/early/2020/11/18/2020.11.15.20231621.short
4100  - http://medrxiv.org/content/early/2020/11/18/2020.11.15.20231621.full
AB  - Objectives Ali-M3, an artificial intelligence, analyses chest computed tomography (CT) and detects the likelihood of coronavirus disease (COVID-19) in the range of 0 to 1. It demonstrates excellent performance for the detection of COVID-19 patients with a sensitivity and specificity of 98.5 and 99.2%, respectively. However, Ali-M3 has not been externally validated. Our purpose is to evaluate the external validity of Ali-M3 using Japanese sequential sampling data.Methods In this retrospective cohort study, COVID-19 infection probabilities were calculated using Ali-M3 in 617 symptomatic patients who underwent reverse transcription-polymerase chain reaction (RT-PCR) tests and chest CT for COVID-19 diagnosis at 11 Japanese tertiary care facilities, between January 1 and April 15, 2020.Results Of 617 patients, 289 patients (46.8%) were RT-PCR-positive. The area under the curve (AUC) of Ali-M3 for predicting a COVID-19 diagnosis was 0.797 (95% confidence intervals [CI]: 0.762-0.833) and goodness-of-fit was P = 0.156. With a cut-off of probability of COVID-19 by Ali-M3 diagnosis set at 0.5, the sensitivity and specificity were 80.6% and 68.3%, respectively, while a cut-off of 0.2 yielded a sensitivity and specificity of 89.2% and 43.2%, respectively. Among 223 patients who required oxygen support, the AUC was 0.825 and sensitivity at a cut-off of 0.5 and 0.2 were 88.7% and 97.9%, respectively. Although the sensitivity was lower when the days from symptom onset were few, sensitivity increased for both cut-off values after 5 days.Conclusions Ali-M3 was evaluated by external validation and shown to be useful to exclude a diagnosis of COVID-19.Key PointsThe area under the curve (AUC) of Ali-M3, which is an AI system for diagnosis of COVID-19 based on chest CT images, was 0.797 and goodness-of-fit was P = 0.156.With a cut-off of probability of COVID-19 by Ali-M3 diagnosis set at 0.5, the sensitivity and specificity were 80.6% and 68.3%, respectively, while a cut-off of 0.2 yielded 89.2% and 43.2%.Although low sensitivity was observed in less number of days from symptoms onset, after 5 days high increasing sensitivity was observed. In patients requiring oxygen support, the AUC was higher that is 0.825.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Hyogo Prefectural Amagasaki General Medical Center, Kyoto University Graduate School of Medicine, Kobe City Medical Center General Hospital, St. Marianna University School of Medicine, Sakai City Medical Center, Kyoto City Hospital, Tokyobay Urayasu Ichikawa Medical Center, Kawasaki Municipal Kawasaki Hospital, Yamanashi Prefectural Central Hospital, Osaka City General Hospital, National Defense Medical College Hospital, Hyogo Prefectural Kakogawa Medical CenterAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe data that support the findings of this study are available from the corresponding author, Tatsuyoshi Ikenoue, upon permission of IRB at Hyogo Prefectural Amagasaki General Medical Center and reasonable request.AIartificial intelligenceCOVID-19coronavirus disease 2019CTcomputed tomography