RT Journal Article
SR Electronic
T1 Evaluating the impact on clinical task efficiency of a natural language processing algorithm for searching medical documents: Prospective crossover study
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2022.05.24.22275490
DO 10.1101/2022.05.24.22275490
A1 Park, Eunsoo H
A1 Watson, Hannah I
A1 Mehendale, Felicity V
A1 O’Neil, Alison Q
YR 2022
UL http://medrxiv.org/content/early/2022/06/01/2022.05.24.22275490.abstract
AB Background Information retrieval (IR) from the free text within Electronic Health Records (EHRs) is time-consuming and complex. We hypothesise that Natural Language Processing (NLP)-enhanced search functionality for EHRs can make clinical workflows more efficient and reduce cognitive load for clinicians.Objective To evaluate the efficacy of three levels of search functionality (no search, string search, and NLP-enhanced search) in supporting IR for clinical users from the free text of EHR documents in a simulated clinical environment.Methods A clinical environment was simulated by uploading three sets of patient notes into an EHR research software application and presenting these alongside three corresponding IR tasks. Tasks contained a mixture of multiple choice and free text questions.A prospective crossover study design was used, for which three groups of evaluators were recruited, comprised of doctors (n=19) and medical students (n=16). Evaluators performed the three tasks using each of the search functionalities in an order according to their randomly assigned group. The speed and accuracy of task completion was measured and analysed, and user perceptions of NLP-enhanced search were reviewed in a feedback survey.Results NLP-enhanced search facilitated significantly more accurate task completion than both string search (5.26%, p=0.01) and no search (7.44%, p=0.05). NLP-enhanced search and string search facilitated similar task speeds, both showing an increase in speed over no search function (15.9%/11.6%, p=0.05). 93% of evaluators agreed that NLP-enhanced search would make clinical workflows more efficient than string search, with qualitative feedback reporting that NLP-enhanced search reduced cognitive load.Conclusions To the best of our knowledge, this study is the largest evaluation to date of different search functionalities for supporting target clinical users in realistic clinical workflows, with a 3-way prospective crossover study design. NLP-enhanced search improved both accuracy and speed of clinical EHR IR tasks compared to browsing clinical notes without search. NLP-enhanced search improved accuracy and reduced the number of searches required for clinical EHR IR tasks compared to direct search term matching.Competing Interest StatementHI Watson and AQ O'Neil are employees of Canon Medical Research Europe, who provided the software and algorithms for this evaluation. EH Park was sponsored by Canon Medical Research Europe during her Spring 2021 BSc research project at the University of Edinburgh ("Evaluation of a natural language processing algorithm for searching medical documents") which was the basis for this evaluation. EH Park had previously performed paid annotation work for the development of the NLP-enhanced search tool.Funding StatementThis work is part of the Industrial Centre for AI Research in digital Diagnostics (iCAIRD), which is funded by Innovate UK on behalf of UK Research and Innovation (UKRI) project number 104690. FV Mehendale's research at the University of Edinburgh is supported by the Caledonian Heritable Foundation.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesAll data used is non-publicly available data.EHRElectronic Health RecordIRInformation RetrievalNLPNatural Language ProcessingUMLSUnified Medical Language System