RT Journal Article
SR Electronic
T1 Data Extraction for Evidence Synthesis Using a Large Language Model: A Proof-of-Concept Study
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2023.10.02.23296415
DO 10.1101/2023.10.02.23296415
A1 Gartlehner, G
A1 Kahwati, L
A1 Hilscher, R
A1 Thomas, I
A1 Kugley, S
A1 Crotty, K
A1 Viswanathan, M
A1 Nussbaumer-Streit, B
A1 Booth, G
A1 Erskine, N
A1 Konet, A
A1 Chew, R
YR 2023
UL http://medrxiv.org/content/early/2023/10/03/2023.10.02.23296415.abstract
AB Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the advent of Large Language Models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n=4) were missed data items. Importantly, Claude 2’s ease of use was high; it required no technical expertise or training data for effective operation. Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.Competing Interest StatementThe authors have declared no competing interest.Clinical Protocols https://osf.io/search?q=osf.io%2F2546n Funding StatementThis study was funded by RTI International through the Innovation Fund.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesData supporting the findings of this study are available from the corresponding author upon reasonable request.