PT  - JOURNAL ARTICLE
AU  - Han, Tianyu
AU  - Adams, Lisa C
AU  - Bressem, Keno
AU  - Busch, Felix
AU  - Huck, Luisa
AU  - Nebelung, Sven
AU  - Truhn, Daniel
TI  - Comparative Analysis of GPT-4Vision, GPT-4 and Open Source LLMs in Clinical Diagnostic Accuracy: A Benchmark Against Human Expertise
AID  - 10.1101/2023.11.03.23297957
DP  - 2023 Jan 01
TA  - medRxiv
PG  - 2023.11.03.23297957
4099  - http://medrxiv.org/content/early/2023/11/08/2023.11.03.23297957.short
4100  - http://medrxiv.org/content/early/2023/11/08/2023.11.03.23297957.full
AB  - Importance Medicine is poised for transformation with artificial general intelligence becoming integral to almost all clinical environments. Currently, the performance of multimodal AI, specifically one powered by GPT-4, in real clinical cases remains uncharted.Objective To ascertain whether GPT-4V can consistently comprehend complex diagnostic scenarios through both imagery and textual data.Design A selection of 140 clinical cases from the JAMA Clinical Challenge and 348 from the NEJM Image Challenge were used. Each case, comprising a clinical image and corresponding question, was processed by GPT-4V, and responses were documented. The significance of imaging information was assessed by comparing GPT-4V’s performance with that of four other leading-edge large language models (LLMs).Main Outcomes and Measures The accuracy of responses was gauged by juxtaposing the model’s answers with the established ground truths of the challenges. The confidence interval for the model’s performance was calculated using bootstrapping methods. Additionally, human performance on the NEJM Image Challenge was chronicled, reflected by the choice percentage selected by challenge participants.Results GPT-4V demonstrated superior accuracy in analyses of both sources, achieving 73.3% for JAMA and 88.7% for NEJM, notably outperforming text- only LLMs such as GPT-4, GPT-3.5, Llama2, and Med-42. Remarkably, both GPT-4V and GPT-4 exceeded average human participants’ performance at all complexity levels within the NEJM Image Challenge.Conclusions and Relevance GPT-4V has exhibited considerable promise in clinical diagnostic tasks, surpassing the capabilities of its predecessors as well as those of human experts. However, while its proficiency in identification tasks is commendable, it requires further refinement in decision-making and strategic planning. Despite these encouraging results, such models should be adopted with prudence in clinical settings, serving to augment rather than replace human discretion. Continual research is imperative to fully evaluate the potential impact on patient care.Competing Interest StatementThe authors have declared no competing interest.Funding StatementN/AAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesThe data that support the findings of this study are openly available (https://jamanetwork.com/collections/44038/clinical-challenge and https://www.nejm.org/case-challenges). Specific data related to AI model responses can be accessed freely, ensuring transparency and reproducibility of the research.