RT Journal Article
SR Electronic
T1 Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2023.07.31.23293474
DO 10.1101/2023.07.31.23293474
A1 Thirunavukarasu, Arun James
A1 Mahmood, Shathar
A1 Malem, Andrew
A1 Foster, William Paul
A1 Sanghera, Rohan
A1 Hassan, Refaat
A1 Zhou, Sean
A1 Wong, Shiao Wei
A1 Wong, Yee Ling
A1 Chong, Yu Jeat
A1 Shakeel, Abdullah
A1 Chang, Yin-Hsi
A1 Tan, Benjamin Kye Jyn
A1 Jain, Nikhil
A1 Tan, Ting Fang
A1 Rauz, Saaeha
A1 Ting, Daniel Shu Wei
A1 Ting, Darren Shu Jeng
YR 2023
UL http://medrxiv.org/content/early/2023/08/06/2023.07.31.23293474.abstract
AB Objective To evaluate the clinical potential of large language models (LLMs) in ophthalmology using a more robust benchmark than raw examination scores.Materials and methods GPT-3.5 and GPT-4 were trialled on 347 questions before GPT-3.5, GPT-4, PaLM 2, LLaMA, expert ophthalmologists, and doctors in training were trialled on a mock examination of 87 questions. Performance was analysed with respect to question subject and type (first order recall and higher order reasoning). Masked ophthalmologists graded the accuracy, relevance, and overall preference of GPT-3.5 and GPT-4 responses to the same questions.Results The performance of GPT-4 (69%) was superior to GPT-3.5 (48%), LLaMA (32%), and PaLM 2 (56%). GPT-4 compared favourably with expert ophthalmologists (median 76%, range 64-90%), ophthalmology trainees (median 59%, range 57-63%), and unspecialised junior doctors (median 43%, range 41-44%). Low agreement between LLMs and doctors reflected idiosyncratic differences in knowledge and reasoning with overall consistency across subjects and types (p>0.05). All ophthalmologists preferred GPT-4 responses over GPT-3.5 and rated the accuracy and relevance of GPT-4 as higher (p<0.05).Discussion In view of the comparable or superior performance to trainee-grade ophthalmologists and unspecialised junior doctors, state-of-the-art LLMs such as GPT-4 may provide useful medical advice and assistance where access to expert ophthalmologists is limited. Clinical benchmarks provide useful assays of LLM capabilities in healthcare before clinical trials can be designed and conducted.Conclusion LLMs are approaching expert-level knowledge and reasoning skills in ophthalmology. Further research is required to develop and validate clinical applications to improve eye health outcomes.Competing Interest StatementAM is a member of the Panel of Examiners of the Royal College of Ophthalmologists and performs unpaid work as an FRCOphth examiner. DSWT holds a patent on a deep learning system to detect retinal disease. DSJT authored the book used in the study and receives royalty from its sales. The other authors have no competing interests to declare.Funding StatementDSWT is supported by the National Medical Research Council, Singapore (NMCR/HSRG/0087/2018 MOH-000655-00 MOH-001014-00), Duke-NUS Medical School (Duke-NUS/RSF/2021/0018 05/FY2020/EX/15-A58), and Agency for Science, Technology and Research (A20H4g2141 H20C6a0032). DSJT is supported by a Medical Research Council / Fight for Sight Clinical Research Fellowship (MR/T001674/1). These funders were not involved in the conception, execution, or reporting of this review.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:No ethical approval required as no human or animal subjects involved.I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesData are available upon request, excluding copyrighted material from the textbook used for experiments.AIartificial intelligenceFRCOphthFellowship of the Royal College of OphthalmologistsGPTGenerative Pretrained TransformerLLaMALarge Language Model Meta AILLMlarge language modelOKAPOphthalmology Knowledge Assessment ProgramPaLMPathways Language Model