Abstract
Background Vasospastic angina is sometimes suspected from patients’ medical history. It is essential to appropriately distinguish vasospastic angina from acute coronary syndrome because its standard treatment is pharmacotherapy, not catheter intervention. Large language models have recently been developed and are currently widely accessible. In this study, we aimed to use large language models to distinguish between vasospastic angina and acute coronary syndrome from patient information and compare the accuracies of these models.
Method We searched for cases of vasospastic angina and acute coronary syndrome which were written in Japanese and published in online-accessible abstracts and journals, and randomly selected 66 cases as a test dataset. In addition, we selected another ten cases as data for few-shot learning. We used generative pre-trained transformer-3.5 and 4, and Bard, with zero- and few-shot learning. We evaluated the accuracies of the models using the test dataset.
Results Generative pre-trained transformer-3.5 with zero-shot learning achieved an accuracy of 52%, sensitivity of 68%, and specificity of 29%; with few-shot learning, it achieved an accuracy of 52%, sensitivity of 26%, and specificity of 86%. Generative pre-trained transformer-4 with zero-shot learning achieved an accuracy of 58%, sensitivity of 29%, and specificity of 96%; with few-shot learning, it achieved an accuracy of 61%, sensitivity of 63%, and specificity of 57%. Bard with zero-shot learning achieved an accuracy of 47%, sensitivity of 16%, and specificity of 89%; with few-shot learning, this model could not be assessed because it failed to produce output.
Conclusion Generative pre-trained transformer-4 with few-shot learning was the best of all the models. The accuracies of models with zero- and few-shot learning were almost the same. In the future, models could be made more accurate by combining text data with other modalities.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was supported by JSPS KAKENHI (grant no. JP 23H03491).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
N/A
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All relevant data are within the manuscript and its Supporting Information files.
