Abstract
Background AI systems have the potential to support in detecting cerebral aneurysms. Yet, the role of automation bias (inclination of humans to overly rely on automated decision-making systems) in AI-assisted cerebral aneurysm detection remains unclear.
Purpose To determine how automation bias can affect radiologists with varying experience levels when reading time-of-flight magnetic resonance angiography (TOF-MRA) studies with the assistance of an AI system for cerebral aneurysm detection.
Methods In this prospective experiment, nine radiologists with varying levels of experience evaluated twenty TOF-MRA exams for the presence of anterior circulation aneurysms, with each arterial segment rated on a 4-point Likert scale, and provided follow-up recommendations. Every case was evaluated twice (with or without assistance by the AI software © mdbrain), with a washout-period of at least four weeks between the two sessions. Ten out of twenty cases included at least one false-positive AI finding. Aneurysm ratings, follow-up recommendations, and reading times were assessed using the Wilcoxon signed-rank test. A thematic analysis was performed to summarize reader feedback and observations.
Results False-positive AI results led to significantly higher suspicion of aneurysm findings (p = 0.01). Inexperienced readers further recommended significantly more aggressive follow-up examinations when presented with false-positive AI findings (p = 0.005). Reading times were significantly shorter with AI assistance in inexperienced (164.1 vs 228.2 seconds; p < 0.001), moderately experienced (126.2 vs 156.5 seconds; p < 0.009), and very experienced (117.9 vs 153.5 seconds; p < 0.001) readers alike.
Conclusion Our results demonstrate susceptibility of radiology readers to automation bias in detecting cerebral aneurysms in TOF-MRA studies when encountering false-positive AI findings. In inexperienced readers, this behavior further translated into more aggressive follow-up recommendations. AI assistance resulted in significantly shorter reading times across experience levels. While AI systems for cerebral aneurysm detection can provide benefits, challenges in human-AI interaction need to be mitigated to ensure safe and effective adoption.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Ethics committee of the Technical University of Munich gave ethical approval for this work
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors
Abbreviations
- CAD
- Computer-assisted diagnosis
- CTA
- Computed tomography angiography
- TOF-MRA
- Time-of-flight magnetic resonance angiography
- DSA
- Digital subtraction angiography
- AI
- Artificial intelligence
