Granularly, Precisely, and Timely: Leveraging Large Language Models for Safety and Efficacy Extraction in Oncology Clinical Trial Abstracts (SEETrials)

Objective Conference abstracts provide preliminary evidence for clinical trial outcomes. This study aims to develop an automatic extraction system to precisely extract and convert granular safety and efficacy information from abstracts into a computable format for timely downstream analysis.

Materials and Methods We collected multiple myeloma clinical trial abstracts from ASCO, ASH, and PubMed (2012-2023) to develop SEETrials, a GPT-4 based system. Qualitative and quantitative evaluations were conducted. Descriptive data analysis on efficacy and safety entities was performed. The generalizability of the system was tested in three other cancer trial studies.

Results SEETrials achieved precision, recall, and F1 of 95.5%, 93.9%, and 94.7% across 70 data elements. Analysis of 245 multiple myeloma clinical trial abstracts revealed variations in safety and efficacy entity distribution across different modalities and phases. Application to other cancer trial studies demonstrated consistent performance with precision, recall, and F1 of 96.9%, 95.4%, and 96.1%, respectively.

Discussion Qualitative error analysis identified a common source of inaccuracy, notably in cohort identification and categorization. Phase1/2 studies prioritize early indications of treatment efficacy, while phase 2/3 studies emphasize prolonged effects. Adverse event distribution is consistent across phases, with noticeable increases in fatal events in later phases, suggesting longer follow-up or more patients may reveal severe adverse events missed in phase 1 safety studies.

Conclusion SEETrials displayed high accuracy and generalizability to diverse drug modalities and disease domains. Its capacity to streamline large-scale dataset analysis is crucial for advancing clinical trial research, ensuring timely and accurate data extraction, and facilitating efficient dissemination.