Performance of Open-Source LLMs in Challenging Radiological Cases – A Benchmark Study on 1,933 Eurorad Case Reports
View ORCID ProfileSu Hwan Kim, Severin Schramm, Lisa C. Adams, Rickmer Braren, Keno K. Bressem, Matthias Keicher, Claus Zimmer, Dennis M. Hedderich, View ORCID ProfileBenedikt Wiestler
doi: https://doi.org/10.1101/2024.09.04.24313026
Su Hwan Kim
1Department of Diagnostic and Interventional Neuroradiology, Klinikum rechts der Isar, School of Medicine and Health, Technical University of Munich, Munich, Germany
Severin Schramm
1Department of Diagnostic and Interventional Neuroradiology, Klinikum rechts der Isar, School of Medicine and Health, Technical University of Munich, Munich, Germany
Lisa C. Adams
2Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, School of Medicine and Health, Technical University of Munich, Munich, Germany
Rickmer Braren
2Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, School of Medicine and Health, Technical University of Munich, Munich, Germany
Keno K. Bressem
3Department of Cardiovascular Radiology and Nuclear Medicine, German Heart Center Munich, School of Medicine and Health, Technical University of Munich, Munich, Germany
Matthias Keicher
4Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany
Claus Zimmer
1Department of Diagnostic and Interventional Neuroradiology, Klinikum rechts der Isar, School of Medicine and Health, Technical University of Munich, Munich, Germany
Dennis M. Hedderich
1Department of Diagnostic and Interventional Neuroradiology, Klinikum rechts der Isar, School of Medicine and Health, Technical University of Munich, Munich, Germany
Benedikt Wiestler
1Department of Diagnostic and Interventional Neuroradiology, Klinikum rechts der Isar, School of Medicine and Health, Technical University of Munich, Munich, Germany
5AI for Image-Guided Diagnosis and Therapy, School of Medicine and Health, Technical University of Munich, Munich, Germany
Data Availability
The Python code used in the present study is publicly available online at our GitHub repository (https://github.com/ai-idt/os_llm_eurorad).
Posted October 03, 2024.
Performance of Open-Source LLMs in Challenging Radiological Cases – A Benchmark Study on 1,933 Eurorad Case Reports
Su Hwan Kim, Severin Schramm, Lisa C. Adams, Rickmer Braren, Keno K. Bressem, Matthias Keicher, Claus Zimmer, Dennis M. Hedderich, Benedikt Wiestler
medRxiv 2024.09.04.24313026; doi: https://doi.org/10.1101/2024.09.04.24313026
Performance of Open-Source LLMs in Challenging Radiological Cases – A Benchmark Study on 1,933 Eurorad Case Reports
Su Hwan Kim, Severin Schramm, Lisa C. Adams, Rickmer Braren, Keno K. Bressem, Matthias Keicher, Claus Zimmer, Dennis M. Hedderich, Benedikt Wiestler
medRxiv 2024.09.04.24313026; doi: https://doi.org/10.1101/2024.09.04.24313026
Subject Area
Subject Areas
- Addiction Medicine (383)
- Allergy and Immunology (699)
- Anesthesia (192)
- Cardiovascular Medicine (2856)
- Dermatology (244)
- Emergency Medicine (430)
- Epidemiology (12563)
- Forensic Medicine (10)
- Gastroenterology (807)
- Genetic and Genomic Medicine (4437)
- Geriatric Medicine (402)
- Health Economics (716)
- Health Informatics (2852)
- Health Policy (1049)
- Hematology (375)
- HIV/AIDS (893)
- Medical Education (413)
- Medical Ethics (114)
- Nephrology (464)
- Neurology (4196)
- Nursing (222)
- Nutrition (617)
- Oncology (2204)
- Ophthalmology (624)
- Orthopedics (254)
- Otolaryngology (318)
- Pain Medicine (269)
- Palliative Medicine (82)
- Pathology (486)
- Pediatrics (1172)
- Primary Care Research (483)
- Public and Global Health (6784)
- Radiology and Imaging (1490)
- Respiratory Medicine (900)
- Rheumatology (430)
- Sports Medicine (369)
- Surgery (473)
- Toxicology (57)
- Transplantation (202)
- Urology (174)