Does ChatGPT Provide Appropriate and Equitable Medical Advice?: A Vignette-Based, Clinical Evaluation Across Care Contexts

MAIN

Large language models (LLMs) are statistical models trained on large texts that can be used to support human-like chat applications. The recently released ChatGPT application is based on a LLM trained using large text samples from the world wide web, Wikipedia, and book text, among other sources, and reinforced with human-supervised questions and answers.¹ ChatGPT can engage in conversations with human-like responses to prompts like writing research papers, poetry, and computer programs. Just as Internet searches have become common for people seeking health information, ChatGPT may also become an efficient and accessible tool for people seeking online medical advice.²

Some preliminary work in the medical domain highlighted ChatGPT’s ability to write realistic scientific abstracts,³ pass medical licensing exams,⁴ and accurately determine appropriate radiology studies.⁵ Although ChatGPT can triage medical cases,⁶ answer clinical questions consistent with the judgment of practicing physicians,⁷ and provide medical advice that is perceived as human-like by non-clinicians,⁸ its ability to provide appropriate and equitable advice to patients across a range of clinical contexts remain unknown. These knowledge gaps are important because the underlying training data and approach for ChatGPT have not been released,⁹ and there are substantive concerns about the safety, fairness, and regulation of LLMs and clinical AI systems.^10-12

Therefore, we sought to (1) assess the clinical appropriateness of ChatGPT’s responses to advice-seeking questions across the clinical spectrum, including prevention, management of acute illness, and end-of-life decision-making, and (2) assess the equity of the responses by evaluating whether they differ by patient race, gender, or insurance status.

We presented ChatGPT with text-based vignettes using all permutations of three advice-seeking clinical scenarios while varying age (25 or 65 years old), race (Black or white), gender (man or woman), and insurance status (good or no insurance). Each scenario also included two variations of the patient’s medical history. The first clinical scenario inquired about lipid testing appropriateness in patients with different medical histories (healthy patient or prior heart attack). The second scenario inquired about triage for a case of acute chest pain (likely dyspepsia or acute coronary syndrome). The third scenario requested advice about pursuing palliative care in a patient with end-stage heart failure (good or poor prognosis) (Extended Data Fig. 1). Thus, the factorial design resulted in 96 unique vignettes based on 3 scenarios x 2 clinical histories x 2 ages x 2 races x 2 genders x 2 insurance statuses. Responses were assessed for clinical appropriateness, acknowledgement of uncertainty, appropriate follow-up reasoning, recommendation type, and differences by demographic characteristics. These outcomes were dual coded by two physicians (AJN and GEW) and disagreement was resolved through consensus discussion.

• Download figure
• Open in new tab

Extended Data Figure 1. Clinical vignettes used in ChatGPT queries with associated clinical scenarios, demographic variables, and ascertained outcomes. Each vignette-based query was followed by the question: “Can you explain your answer further?”.

Three (3%) responses contained clinically inappropriate advice that was clearly inconsistent with established care guidelines. One response in scenario 1 recommended every adult undergo regular lipid screening, one in scenario 2 recommended always emergently seeking medical attention for any chest pain, and another in scenario 2 advised an uninsured 25-year-old with crushing left-sided chest pain to present either to a community health clinic or the emergency department (ED). Although technically appropriate, some responses were overly cautious and over-recommended ED referral for low-risk chest pain in scenario 2. Many responses lacked a specific recommendation and simply provided explanatory information such as the definition of palliative care while also recommending discussion with a clinician in scenario 3. Ninety-three (97%) responses appropriately acknowledged clinical uncertainty through the mention of a differential diagnosis or dependence of a recommendation on additional clinical or personal factors. The three responses that did not account for clinical uncertainty were in scenario 2 and did not provide any differential diagnosis or alternative possibilities for acute chest pain other than potentially dangerous cardiac etiologies. Ninety-five (99%) responses provided appropriate follow-up reasoning. The one response that provided faulty medical reasoning was from scenario 2 and reasoned that because the chest pain was happening after eating spicy foods it was more likely from a serious etiology (Table 1).

ChatGPT provided either no recommendation or suggested further discussion with a clinician 34 (35%) times. Of these, 2 (2%) were from scenario 1, 0 from scenario 2, 32 (33%) from scenario 3. Eighteen (19%) responses provided a general recommendation, all from scenario 1 and referred to what a typical patient in a given age range might do according to the American Heart Association (AHA) guidelines for lipid screening.¹³ Forty-four (46%) provided a specific recommendation, 12 (13%) from scenario 1 where ChatGPT specifically recommended the patient to get their lipids checked, 32 (33%) from scenario 2, with a specific recommendation to seek care in the ED, and 0 from scenario 3, as scenario 3 responses uniformly described palliative care in broad terms, sometimes differentiating it from hospice, and always recommended a discussion with a clinician without a specific recommendation to pursue palliative or aggressive care (Table 1). Five (5%) responses in scenario 3 began with a disclaimer about being an AI language model not being able to provide medical advice.

Nine (9%) responses accounted for race, often simply prefacing the reply with the patient’s race and gender. Eight (8%) race-tailored responses were from scenario 1, 1 (1%) from scenario 2 which mentioned increased cardiovascular disease risk in black men, and none were from scenario 3. 37 (39%) responses acknowledged the insurance status and, in doing so, often suggested less costly treatment venues such as community health centers. One case of high-risk chest pain in an uninsured patient in scenario 2 was inappropriately recommended to present to either a community health center or the ED despite only recommending ED presentation to the same patient with insurance. Eleven (12%) insurance-tailored responses were from scenario 1, 21 (22%) from scenario 2, and 5 (5%) from scenario 3. Twenty-eight (29%) incorporated gender into the response. 19 (20%) gender-tailored responses were from scenario 1, 7 (7%) from scenario 2 where one response described atypical presentations of acute coronary syndrome in women, and 2 (2%) from scenario 3.

There were no associations between race or gender with the type of recommendation or with a tailored response (Table 2). Only the mention of “no insurance” in the vignette was consistently associated with a specific response related to healthcare costs and access. ChatGPT never asked any follow up questions.

Overall, we found that ChatGPT usually provided appropriate medical advice in response to advice-seeking questions. The types of responses ranged from providing explanations, such as in scenario 3 where ChatGPT described palliative care, to decisive medical advice, such as in scenario 2 where it provided urgent, patient-specific recommendations to seek immediate care in the ED. Importantly, the responses lacked personalized nuance or follow-up questions that would be expected of a clinician when providing medical advice.¹⁴ For example, in scenario 1, the response referenced the AHA guidelines to support lipid screening recommendations but ignored other established guidelines with divergent recommendations.¹³ In scenario 2, ChatGPT suboptimally triaged a case of high-risk chest pain and over-cautiously recommended ED presentation. The responses rarely provided a more tailored approach that considered pain quality, duration, and associated symptoms or contextual clinical factors that are standard of practice when evaluating cases of chest pain. The potential implications of following such advice without nuance or further information gathering include over-presentation to already overflowing emergency departments, over-utilization of medical resources, and unnecessary patient financial strain.

ChatGPT’s responses accounted for social factors including race, insurance status, and gender in varied ways with important clinical implications. Most notably, the content of the medical advice varied when ChatGPT recommended evaluation at a community health clinic for the uninsured patient and the ED for the same patient with good insurance, even when the ED was the safer place of initial evaluation. This difference, without a clinical basis, raises the concern that ChatGPT’s medical advice could exacerbate health disparities if followed.

The content and type of responses varied widely and seemingly arbitrarily. Changing one social characteristic while keeping the clinical history fixed sometimes resulted in a reply that changed from a confident recommendation to a disclaimer about being an artificial intelligence tool with limitations necessitating discussion with a clinician. This finding highlights a lack of reliability in ChatGPT’s responses and the unanswered question of the optimal balance among personalization, consistency, and human-like spontaneity when providing medical advice in a digital chat environment.

Several limitations should be considered in the interpretation of our findings. First, we tested three specific clinical scenarios and our analysis of ChatGPT’s responses may not generalize to other clinical contexts. Second, our study design did not assess within-vignette variation and thus could not detect or measure potential randomness in the responses.

This study provides important evidence contextualizing the ability of ChatGPT to offer appropriate and equitable advice to patients across the care continuum. We found that ChatGPT’s medical advice was usually safe but often lacked specificity or nuance. The responses maintained an inconsistent awareness of ChatGPT’s inherent limitations and clinical uncertainty. We also found that ChatGPT often tailored responses to a patient’s insurance status in ways that were clinically inappropriate. Thus, following such advice could lead to over-utilization, increased spending, and exacerbation of inequities in care. Based on these findings, ChatGPT is currently useful for providing background knowledge on general clinical topics but cannot reliably provide personalized or appropriate medical advice. Future training on medical corpora, clinician-supervised feedback, and augmenting awareness of uncertainty and information seeking may offer improvements to the medical advice provided by future LLMs.

METHODS

Personal Acknowledgements

Not applicable.

Funding

GEW received support from NIH K23HL141639. KRC received support from NIH K23143181 and R01AG073384.

Disclosure

The authors declare that they have no conflicts of interest relevant to the analysis to report.