PT  - JOURNAL ARTICLE
AU  - Kim, Su Hwan
AU  - Schramm, Severin
AU  - Riedel, Evamaria Olga
AU  - Schmitzer, Lena
AU  - Rosenkranz, Enrike
AU  - Kertels, Olivia
AU  - Bodden, Jannis
AU  - Paprottka, Karolin
AU  - Sepp, Dominik
AU  - Renz, Martin
AU  - Kirschke, Jan
AU  - Baum, Thomas
AU  - Maegerlein, Christian
AU  - Boeckh-Behrens, Tobias
AU  - Zimmer, Claus
AU  - Wiestler, Benedikt
AU  - Hedderich, Dennis M.
TI  - Automation Bias in AI-Assisted Detection of Cerebral Aneurysms on Time-of-Flight MR-Angiography
AID  - 10.1101/2024.05.31.24308021
DP  - 2024 Jan 01
TA  - medRxiv
PG  - 2024.05.31.24308021
4099  - http://medrxiv.org/content/early/2024/06/03/2024.05.31.24308021.short
4100  - http://medrxiv.org/content/early/2024/06/03/2024.05.31.24308021.full
AB  - Background AI systems have the potential to support in detecting cerebral aneurysms. Yet, the role of automation bias (inclination of humans to overly rely on automated decision-making systems) in AI-assisted cerebral aneurysm detection remains unclear.Purpose To determine how automation bias can affect radiologists with varying experience levels when reading time-of-flight magnetic resonance angiography (TOF-MRA) studies with the assistance of an AI system for cerebral aneurysm detection.Methods In this prospective experiment, nine radiologists with varying levels of experience evaluated twenty TOF-MRA exams for the presence of anterior circulation aneurysms, with each arterial segment rated on a 4-point Likert scale, and provided follow-up recommendations. Every case was evaluated twice (with or without assistance by the AI software © mdbrain), with a washout-period of at least four weeks between the two sessions. Ten out of twenty cases included at least one false-positive AI finding. Aneurysm ratings, follow-up recommendations, and reading times were assessed using the Wilcoxon signed-rank test. A thematic analysis was performed to summarize reader feedback and observations.Results False-positive AI results led to significantly higher suspicion of aneurysm findings (p = 0.01). Inexperienced readers further recommended significantly more aggressive follow-up examinations when presented with false-positive AI findings (p = 0.005). Reading times were significantly shorter with AI assistance in inexperienced (164.1 vs 228.2 seconds; p < 0.001), moderately experienced (126.2 vs 156.5 seconds; p < 0.009), and very experienced (117.9 vs 153.5 seconds; p < 0.001) readers alike.Conclusion Our results demonstrate susceptibility of radiology readers to automation bias in detecting cerebral aneurysms in TOF-MRA studies when encountering false-positive AI findings. In inexperienced readers, this behavior further translated into more aggressive follow-up recommendations. AI assistance resulted in significantly shorter reading times across experience levels. While AI systems for cerebral aneurysm detection can provide benefits, challenges in human-AI interaction need to be mitigated to ensure safe and effective adoption.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis study did not receive any funding Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Ethics committee of the Technical University of Munich gave ethical approval for this workI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesAll data produced in the present study are available upon reasonable request to the authorsCADComputer-assisted diagnosisCTAComputed tomography angiographyTOF-MRATime-of-flight magnetic resonance angiographyDSADigital subtraction angiographyAIArtificial intelligence