Abstract
BACKGROUND Accurate interpretation of Cardiotocography (CTG) is a critical tool for monitoring fetal well-being during pregnancy and labor, providing crucial insights into fetal heart rate and uterine contractions. Advanced artificial intelligence (AI) tools such as AI-Large Language Models (AI-LLMs) may enhance the accuracy of CTG interpretation, leading to better clinical outcomes. However, this potential has not yet been examined and reported yet.
OBJECTIVE This study aimed to evaluate the performance of three AI-LLMs (ChatGPT-4o, Gemini Advance, and Copilot) in interpreting CTG images, comparing their performance to junior and senior human doctors, and assessing their reliability in assisting clinical decisions. STUDY DESIGN: Seven CTG images were evaluated by three AI-LLMs, five senior doctors (SHD), and five junior doctors (JHD) and rated by five maternal-fetal medicine (MFM) experts (raters) using five parameters (relevance, clarity, depth, focus, and coherence). The raters were blinded to the source of interpretations, and a Likert scale was used to score the performance of each system. Statistical analysis assessed the homogeneity of expert ratings and the comparative performance of AI-LLMs and doctors.
RESULTS ChatGPT-4o outperformed the other AI models with a score of 77.86, much higher than Gemini Advance (57.14) and Copilot (47.29), as well as the junior doctors (JHD; 61.57). CG4o’s performance (77.86) was only slightly below that of the senior doctor (SHD; 80.43), with no statistically significant differences between CG4o and SHD (p>0.05). Meanwhile, CG4o had the greatest score in the “depth” category, while the other four parameters were only marginally behind SHD.
CONCLUSION CG4o demonstrated outstanding performance in CTG interpretation, surpassing junior doctors and other AI-LLMs, while senior doctors remain superior in all groups. AI-LLMs, particularly CG4o, showed promising potential as valuable tools in clinical practice to assist obstetricians, enhance diagnostic accuracy, and improve patient care.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This research was partly funded by the China Medical University Ying-Tsai Scholar Fund CMU109-YT-04 and CMU internal fund CMU112-IP-01 and CMU113-MF-56 (to MT). KEG is a recipient of an Elite Program Scholarship from the Taiwan Ministry of Education.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Ethical Approval from The Research Ethics Committe of Rumah Sakit Universitas Airlangga, Surabaya, Indonesia no. 156/KEP/2024 protocl number: UA-02-24216
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
We have ensured that all important data required has been included in the Supplementary file. The exception is the raw values provided by individual doctors, which can be provided upon request.
