Abstract
This study evaluated the accuracy and reproducibility of ChatGPT models, specifically GPT-4 and GPT-4o, by reviewing Japanese-language clinical research protocols and informed consent forms using Japanese prompts. The integration of generative AI technologies into clinical research ethics reviews has the potential to enhance consistency, reduce human error, and decrease the manual effort required to assess complex documents. This study primarily aimed to assess and compare the ability of these models to accurately extract and summarize key elements such as research objectives, study design, and ethical considerations, which are critical for ethical review processes. We developed and optimized custom prompts to improve the performance of the models, focusing on the essential aspects of the protocol and informed consent review. The results showed that GPT-4o achieved an 80% accuracy rate in identifying research objectives and a 100% accuracy rate for research design, indicating superior consistency compared with GPT-4, which, despite being slightly less accurate, still showed significant potential for application in ethics reviews. Furthermore, a comparison between customized GPTs and standard prompts revealed that customized GPTs provided significantly higher reproducibility and accuracy, underscoring the value of fine-tuning and Retrieval-Augmented Generation techniques for enhancing AI-assisted review processes. Additionally, challenges in parsing complex PDF documents were identified, highlighting the importance of standardized document formatting to ensure accurate AI analysis. These findings demonstrate the potential of AI-driven systems to improve the efficiency, accuracy, and standardization of research ethics evaluations, potentially setting new standards for AI integration in clinical research practice.
Author summary This study examined the use of ChatGPT models, specifically GPT-4 and GPT-4o, to review clinical research protocols and informed consent forms written in Japanese, focusing on their ability to improve the review process in clinical research ethics. Our objective was to determine whether these AI models can reliably extract and summarize critical elements, such as research objectives, study design, and ethical considerations, from research documents. The GPT-4o model demonstrated high accuracy and reproducibility, particularly in evaluating research designs, with 80% and 100% accuracy rates for research objectives and research designs, respectively. GPT-4 also exhibited good performance, albeit with a slightly lower accuracy. The findings revealed that tailored prompts significantly enhanced model performance compared to standard prompts, highlighting the necessity for precise customization in AI-driven analyses. Furthermore, we discovered that well-structured documents are crucial for enabling accurate AI-assisted reviews. These results suggest that, with further refinement, ChatGPT could become a valuable tool for enhancing the consistency and efficiency of ethics committee evaluations, potentially reducing manual workload and setting new standards for AI integration in clinical research.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
The author(s) received no specific funding for this work.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study used publicly available mock review materials, including a research protocol and ICF, provided on the official website of the Ministry of Health, Labour and Welfare in Japan. The study did not involve any patient or personal information therefore, IRB approval was not required.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All relevant data are included in the manuscript and its supplementary files, specifically in Supplementary file 1, which has been submitted along with the main text.
