PT  - JOURNAL ARTICLE
AU  - Salem, Alexandra C.
AU  - Gale, Robert C.
AU  - Fleegle, Mikala
AU  - Fergadiotis, Gerasimos
AU  - Bedrick, Steven
TI  - Automating intended target identification for paraphasias in discourse using a large language model
AID  - 10.1101/2023.06.18.23291555
DP  - 2023 Jan 01
TA  - medRxiv
PG  - 2023.06.18.23291555
4099  - http://medrxiv.org/content/early/2023/06/22/2023.06.18.23291555.short
4100  - http://medrxiv.org/content/early/2023/06/22/2023.06.18.23291555.full
AB  - Purpose To date there are no automated tools for the identification and fine-grained classification of paraphasias within discourse, the production of which is the hallmark characteristic of most people with aphasia (PWA). In this work we fine-tune a large language model (LLM) to automatically predict paraphasia targets in Cinderella story retellings.Method Data consisted of 353 Cinderella story retellings containing 2,489 paraphasias from PWA, for which research assistants identified their intended targets. We supplemented this training data with 256 sessions from control participants, to which we added 2,427 synthetic paraphasias. We conducted four experiments using different training data configurations to fine-tune the LLM to automatically “fill in the blank” of the paraphasia with a predicted target, given the context of the rest of the story retelling. We tested the experiments’ predictions against our human-identified targets and stratified our results by ambiguity of the targets and clinical factors.Results The model trained on controls and PWA achieved 46.8% accuracy at exactly matching the human-identified target. Fine-tuning on PWA data, with or without controls, led to comparable performance. The model performed better on targets with less human ambiguity, and on paraphasias from participants with less severe or fluent aphasia.Conclusion We were able to automatically identify the intended target of paraphasias in discourse using just the surrounding language about half of the time. These findings take us a step closer to automatic aphasic discourse analysis. In future work, we will incorporate phonological information from the paraphasia to further improve predictive utility.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis work was supported by National Institute on Deafness and Other Communication Disorders Grant R01DC015999 (Principal Investigators: Steven Bedrick and Gerasimos Fergadiotis).Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Data is available from AphasiaBank to all members of the AphasiaBank consortium group (https://aphasia.talkbank.org/).I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesData is available from AphasiaBank to all members of the AphasiaBank consortium group (https://aphasia.talkbank.org/). https://aphasia.talkbank.org/