ABSTRACT
Background Increased use of electronic health records (EHR) and patient-initiated messaging has led to inefficiency, staff shortages, and provider burnout. We evaluated the impact of a natural language processing (NLP) algorithm for message routing on communication dynamics.
Methods We developed an NLP model to accurately label inbound EHR messages from patients using a pre-trained classifier with fine-tuning based on clinician feedback. In a real-world study, the model was deployed to prospectively label and route messages sent to participating physicians at an integrated health system. A parallel control group of unrouted messages was generated for comparative analysis. The primary endpoints for assessing model performance were the time to first message interaction, the time to conversation resolution, and the total number of message interactions by staff, compared with the control group. Secondary endpoints were the precision, recall, F1 score (a measure of positive predictive value and sensitivity), and accuracy for correct message classification.
Results The model prospectively labeled and routed 469 unique conversations over 14 days from the inbaskets of participating physicians. Compared to a control group of 402 unrouted conversations from the same time period, conversations in the routed group had an 83.3% reduction in the time to first interaction (median difference [MD], −1 hour; P<0.001), an 84.4% reduction in the time to conversation resolution (MD, −22.5 hours; P<0.001), and a 40% reduction in the total number of staff interactions after application of intervention (MD, −2.0 interactions; P<0.001). The model demonstrated high precision (>97.6%), recall (>95%), and F1 scores (>96.5%) for accurate prediction of all five message classes, with a total accuracy of 97.8%.
Conclusions Real-time message routing using advanced NLP was associated with significantly reduced message response and resolution times.
Competing Interest Statement
All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; AH and AMH have received in-kind research support from Switchboard MD; MZ and ADC are consultants for Switchboard MD; BA is an employee of Switchboard MD; no other relationships or activities that could appear to have influenced the submitted work.
Funding Statement
The study was funded by Switchboard MD (Atlanta, GA), an artificial intelligence and data science company, and performed in partnership with Emory University.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This was a prospective, open-label, pragmatic, feasibility study conducted at Emory Healthcare, an integrated health system in Atlanta, Georgia. The study was deemed exempt from human subjects research by the institutional review board at Emory University and received a waiver of informed consent.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
The participants of this study have not provided written consent for their data to be shared publicly, therefore supporting data is not available due to the sensitive nature of the research
