Abstract
In recent years, there has been tremendous interest in the use of artificial intelligence (AI) in radiology in order to automate the interpretation. However, uncontrolled and widespread use of AI solutions may have negative consequences. Therefore, before implementing such technologies in healthcare, thorough training of personnel, adaptation of information systems, and standardized datasets for an external validation are required. All this necessitates a formation of a unique unified methodology. The best practices of AI introduction in diagnostic radiology are still subject to debate and require new results of a scientific-practical research with the assessment of implementation conditions.
This work discusses expected issues and potential solutions for the introduction of computer vision-based technologies for automatic analysis of radiological examinations with an emphasis on the real-life experience gained during simultaneous AI implementation into practice of more than a hundred state radiology departments in 2020-2021 in Moscow, Russia (an experiment). The experiment used end-user software testing approaches, quality assurance of AI-based radiological solutions, and accuracy assessment of the AI-empowered diagnostic tools on local data. The methods were adapted and optimized to ensure a successful real-life radiological AI deployment on the extraordinary large scale. The experiment involved in total around thousand diagnostic devices and thousand radiologists. AI deployment was associated with additional options in a routine radiologist’s workflow: triage; additional series formed by AI with indication of pathological findings and their classification; report template prepared by AI in accordance with the target clinical task, user feedback on AI performance.
A multi-stage methodology for implementing AI into radiological practice that was developed and advanced during the experiment is described in this report.
Essentials
A methodology for the AI deployment for non-academic radiological sites excluded more than half of the offered AI solutions that do not fulfill the diagnostic and functional requirements
Quality control of AI should be supported by not only data scientists, IT specialists or engineers, but also by radiologists at all stages of selection and testing.
Radiologists need to understand the capabilities, limitations of AI by getting an additional training.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Independent Ethics Committee (IEC) of Moscow Regional Office (MRO) of the Russian Society of Radiologists and Radiographists (RSRR) gave ethical approval for this work as part of the Experiment on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow. Details of the IEC: Approval Number No.2 (1-II-2020) of the meeting of IEC of RSRR MRO dated February 20, 2020. Address: 28 Srednyaya Kalitnikovskaya St., bld. 1, Moscow. Email: ethics{at}npcmr.ru Present: O.A. Agafonova, E.G. Bahteeva, A.S. Laipan, I.S. Komolov, O.A. Mokienko, I.I. Nadelyaeva, L.A. Nizovtsova, O.V. Omelyanskaya, A.V. Petraikin. Topics: Expert review of the documents for the upcoming study titled the "Experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow." Documents submitted for review: 1. The abstract of the study titled "Experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow" 2. A cover letter to the attention of the Chairman of IEC of RSRR MRO concerning the erhical expert review of the study titled the "Experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow" 3. CV of the Principal Investigator 4. A patient information and informed consent form 5. A project overview Resolution: To approve the upcoming study titled the "Experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow." Due to the conflict of interest O.A. Agafonova, the co-implementer of the project, did not take part in the vote. All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors
List of abbreviations
- AI
- artificial intelligence
- ASPECTS
- The Alberta stroke program early CT score
- AUROC
- area under receiver operating curve
- CT
- computed tomography
- DICOM
- digital imaging and communications in medicine
- DICOM SR
- digital imaging and communications in medicine structured report
- FLG
- fluorography
- LDCT
- low-dose CT
- MMG
- mammography
- PACS
- picture archiving computer system
- QC
- quality control
- RIS
- Radiological Information System
- TT
- technical testing
- URIS
- the Unified Radiological Information Service
