RT Journal Article
SR Electronic
T1 Evaluating General Vision-Language Models for Clinical Medicine
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2024.04.12.24305744
DO 10.1101/2024.04.12.24305744
A1 Jiang, Yixing
A1 Omiye, Jesutofunmi A.
A1 Zakka, Cyril
A1 Moor, Michael
A1 Gui, Haiwen
A1 Alipour, Shayan
A1 Mousavi, Seyed Shahabeddin
A1 Chen, Jonathan H.
A1 Rajpurkar, Pranav
A1 Daneshjou, Roxana
YR 2024
UL http://medrxiv.org/content/early/2024/04/18/2024.04.12.24305744.abstract
AB Recently emerging large multimodal models (LMMs) utilize various types of data modalities, including text and visual inputs to generate outputs. The incorporation of LMMs into clinical medicine presents unique challenges, including accuracy, reliability, and clinical relevance. Here, we explore clinical applications of GPT-4V, an LMM that has been proposed for use in medicine, in gastroenterology, radiology, dermatology, and United States Medical Licensing Examination (USMLE) test questions. We used standardized robust datasets with thousands of endoscopy images, chest x-ray, and skin lesions to benchmark GPT-4V’s ability to predict diagnoses. To assess bias, we also explored GPT-4V’s ability to determine Fitzpatrick skin tones with dermatology images. We found that GPT-4V is limited in performance across all four domains, resulting in decreased performance compared to previously published baseline models. The macro-average precision, recall, and F1-score for gastroenterology were 11.2%, 9.1% and 6.8% respectively. For radiology, the best performing task of identifying cardiomegaly had precision, recall, and F1-score of 28%, 94%, and 43% respectively. In dermatology, GPT-4V had an overall top-1 and top-3 diagnostic accuracy of 6.2% and 21% respectively. There was a significant accuracy drop when predicting images of darker skin tones (p<0.001). GPT-4V accurately identified Fitzpatrick skin tones for 56.5% of images. For the multiple-choice-styled USMLE image-based test questions, GPT-4V had an accuracy of 59%. Our findings demonstrate that the current version of GPT-4V is limited in its diagnostic abilities across multiple image-based medical specialties. Future work should be done to explore LMM’s sensitivity to prompting as well as hybrid models that can combine LMM’s capabilities with other robust models.Competing Interest StatementR.D. has served as an advisor to MDAlgorithms and Revea and received consulting fees from Pfizer, L'Oreal, Frazier Healthcare Partners, and DWA, and research funding from UCB. All other authors declare no competing interests.Funding StatementThis study did not receive any funding.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.Yes