Self-Logical Consistency Assessment of Large Language Models for Patient Feedback Classification : Algorithm Development and Validation Study

Abstract

Background : Patient satisfaction feedback is crucial for hospital service quality, but manual reviews are time-consuming, and traditional natural language processing methods remain inadequate. Large Language Models (LLMs) show promise but are prone to extrinsic faithfulness hallucinations - fabricated or illogical outputs that limit their reliability in healthcare. Objective: This study aimed to evaluate the Self-Logical Consistency Assessment (SLCA), an original method designed to enhance LLM feedback classification reliability by enforcing a logically-structured chain of thought. Methods: SLCA uses two validation steps: self-consistency (identifying the most coherent response) and logical consistency (ensuring alignment with the original statement and expert classifications). We evaluated SLCA using GPT-4 and Llama-3.1 405B on 12,600 classifications from 100 patient feedback samples to assess hallucinations, and tested its performance on a 49,140-classification benchmark derived from 1,170 feedbacks. Results: SLCA reduced hallucinations among detected categories from 15.80% (168/1063) to 0.51% (4/786) with GPT-4 and from 7.17% (51/711) to 1.67% (10/599) with Llama-3.1, with residual errors confined to the emergency feedback category. On the benchmark, SLCA achieved precision-recall scores of 0.86-0.78 for GPT-4 and 0.84-0.58 for Llama-3.1. These results demonstrate SLCA s ability to achieve human-level performance across LLMs. Conclusions: SLCA offers a scalable, explainable solution for improving LLM classification reliability in healthcare. Its capacity to enhance performance without fine-tuning or additional training data positions it as a valuable tool for analyzing patient feedback and supporting hospital service quality improvement.