Abstract
Background ChatGPT received recognition for medical writing. Our objective was to evaluate whether ChatGPT 4.0 could improve the quality of abstracts submitted to a medical conference by clinical researchers.
Methods This was an experimental study involving 24 international researchers who provided one original abstract intended for submission at the 2024 Pediatric Academic Society (PAS) conference. We created a prompt asking ChatGPT-4 to improve the quality of the abstract while adhering PAS submission guidelines. Researchers received the revised version and were tasked with creating a final abstract. The quality of each version (original, ChatGPT and final) was evaluated by the researchers themselves using a numeric scale (0-100). Additionally, three co-investigators assessed abstracts blinded to the version. The primary analysis focused on the mean difference in scores between the final and original abstracts.
Results Abstract quality varied between the three versions with mean scores of 82, 65 and 90 for the original, ChatGPT and final versions, respectively. Overall, the final version displayed significantly improved quality compared to the original (mean difference 8.0 points; 95% CI: 5.6-10.3). Independent ratings by the co-investigator confirmed statistical improvements (mean difference 1.10 points; 95% CI: 0.54-1.66). Researchers identified minor (n=10) and major (n=3) factual errors in ChatGPT’s abstracts.
Conclusion While ChatGPT 4.0 does not produce abstracts of better quality then the one crafted by researchers, it serves as a valuable tool for researchers to enhance the quality of their own abstracts. The utilization of such tools is a potential strategy for researchers seeking to improve their abstracts.
Funding None
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Declaration of interests: The authors have no conflict of interest relevant to this article to disclose.
Financial Disclosure Statement: The authors have no financial relationships relevant to this article to disclose. This study was conducted without financial support.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
Abbreviations
- (ANOVA)
- Analysis of Variance
- (CI)
- Confidence Interval
- (LLM)
- Large Language Models
- (NLP)
- Natural Language Processing
- (PAS)
- Pediatric Academic Society
