PT - JOURNAL ARTICLE AU - Huang, Yunyou AU - Wang, Nana AU - Tang, Suqin AU - Ma, Li AU - Hao, Tianshu AU - Jiang, Zihan AU - Zhang, Fan AU - Kang, Guoxin AU - Miao, Xiuxia AU - Guan, Xianglong AU - Zhang, Ruchang AU - Zhang, Zhifei AU - Zhan, Jianfeng AU - , TI - OpenClinicalAI: enabling AI to diagnose diseases in real-world clinical settings AID - 10.1101/2021.09.08.21263249 DP - 2021 Jan 01 TA - medRxiv PG - 2021.09.08.21263249 4099 - http://medrxiv.org/content/early/2021/09/12/2021.09.08.21263249.short 4100 - http://medrxiv.org/content/early/2021/09/12/2021.09.08.21263249.full AB - This paper quantitatively reveals the state-of-the-art and state-of-the-practice AI systems only achieve acceptable performance on the stringent conditions that all categories of subjects are known, which we call closed clinical settings, but fail to work in real-world clinical settings. Compared to the diagnosis task in the closed setting, real-world clinical settings pose severe challenges, and we must treat them differently. We build a clinical AI benchmark named Clinical AIBench to set up real-world clinical settings to facilitate researches. We propose an open, dynamic machine learning framework and develop an AI system named OpenClinicalAI to diagnose diseases in real-world clinical settings. The first versions of Clinical AIBench and OpenClinicalAI target Alzheimer’s disease. In the real-world clinical setting, OpenClinicalAI significantly out-performs the state-of-the-art AI system. In addition, OpenClinicalAI develops personalized diagnosis strategies to avoid unnecessary testing and seamlessly collaborates with clinicians. It is promising to be embedded in the current medical systems to improve medical services.One-Sentence Summary We propose a clinical AI benchmark and an open, dynamic machine learning framework to enable AI diagnosis systems to land in real-world clinical settings.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis work is supported by the Project of Guangxi Science and Technology (No. GuiKeAD20297004) and the National Natural Science Foundation of China (No.61967002).Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:All data in this work is public and shared by Alzheimer's Disease Neuroimaging Initiative.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe data from Alzheimer's Disease Neuroimaging Initiative was used under license for the current study. Applications for access to the dataset can be made at http://adni.loni.usc.edu/data-samples/access-data/. All original code has been deposited at the website https://www.benchcouncil.org/ and is publicly available as of the date of publication. http://adni.loni.usc.edu/data-samples/access-data/