General purpose large language models match human performance on gastroenterology board exam self-assessments

Introduction While general-purpose large language models(LLMs) were able to pass USMLE-style examinations, their ability to perform in a specialized context, like gastroenterology, is unclear. In this study, we assessed the performance of three widely available LLMs: PaLM-2, GPT-3.5, and GPT-4 on the most recent ACG self-assessment(2022), utilizing both a basic and a prompt-engineered technique.

Methods We interacted with the chat interfaces of PaLM-2, GPT-3.5, and GPT-4. We first applied a basic prompt approach, providing each exam question and answer text with minimalist text descriptions of any images. For the engineered approach, we added additional context and instructions. We assessed each model-prompt combination in terms of overall and difficulty-stratified performance and compared this to average human performance. We also evaluated each model’s self-assessed uncertainty. The highest scoring model-prompt combination was further assessed on the 2021 exam. We also assessed the impact of image descriptions on our findings.

Results Using a basic prompt, PaLM-2, GPT-3.5, and GPT-4 achieved scores of 32.6%, 55.3%, and 68.9% respectively. With the engineered prompt, scores improved to 42.7%, 65.2%, and 76.3% respectively. Testing GPT-4 on the ACG-2021 exam yielded a similar score(75.3%). GPT-4 scores matched the average score for human test-takers reported by ACG(75.7%). GPT-4 showed a capability to self-assess its confidence accurately in the context of a multiple-choice exam with its confidence estimates falling within 5% of its actual performance. Excluding image-based questions didn’t change the primary findings.

Discussion Our study highlights the capability of GPT-4 to answer subspecialty board-exam questions at a level commensurate with the average human test-taker. The results confirm that prompt-engineering can enhance LLMs’ performance on medical reasoning tasks. We also show GPT-4 can provide insightful measures of uncertainty in the setting of board-style multiple-choice questions, alerting users to low-quality answers. Future studies of LLMs in gastroenterology should incorporate prompt-engineering to maximize model capabilities.

WHAT IS KNOWN

1. State of the Art large language models like GPT-4 and PaLM-Med 2 have achieved above average performance on USMLE board examinations.

2. In a previous study using basic model prompt instructions, GPT 3.5 and GPT 4 did not pass the 2021 and 2022 ACG self-assessment exams.

WHAT IS NEW HERE

1. Optimizing large language model prompt instructions improved the performance of chat-based GPT-3.5, GPT-4, and PaLM 2 on the ACG self-assessment exams.

2. With optimized prompt instructions, chat-based GPT-4 performed at the level of average human test takers on ACG-self assessment examinations and achieved a passing score.

3. Chat-based GPT-4 self-reported confidence levels correlated with correct answer rates on the ACG-self assessment examinations.