Evaluating the impact on clinical task efficiency of a natural language processing algorithm for searching medical documents: Prospective crossover study

Introduction

The benefits of the transition from storing patient information in paper notes to Electronic Health Records (EHRs) has been a topic of debate among healthcare professionals [1-4]. Many clinicians have expressed dissatisfaction with their current hospital systems and EHR use is consistently cited as a contributor to clinician burnout [5-7]. Approximately 40% of doctors’ time is spent documenting patient information, with evidence showing that this work burden has increased following EHR implementation [8,9]. However, difficulties in quickly and accurately retrieving relevant information from these documents means this wealth of collected information is often not fully utilised [10,11]. Navigating EHR documents is challenging due to the complexity of medical text, which tends to include frequent misspellings, abbreviations, specialty-specific acronyms, and clinical shorthand [12-15]. Time-consuming and inaccurate information gathering from EHRs limits the efficiency of wider clinical workflows [16], with some doctors believing that difficulties in retrieving patient information significantly impact face-to-face patient care [17].

Despite the increasing sophistication of general search engines, there remain relatively limited search options within medical record software. One barrier is the need for patient data to be held securely; therefore, access to computing power and shared resources may be limited. To have clinical utility, search facilities must be fast and intuitive for use by time-pressured clinicians, including relatively junior members of staff to whom the task of searching through complex notes is frequently delegated. In addition, the search must handle high variability of text expression as mentioned above. Clinical text is error-prone; unlike journals and other publications, there is no editorial control to check for errors. Medical terminology, acronyms and abbreviations vary between regions, hospitals and even across different specialties; for instance, “CHD” may be related to chronic heart disease (Cardiology), congenital heart disease (Paediatrics), or congenital hip dislocation (Orthopaedics). Since clinical care is a high stakes environment, failure to find relevant information potentially has great implications; to effectively save the time of clinicians, search tools should ideally go beyond document-level results to locate and highlight all relevant sentences or even words within a document. Efforts to achieve easier information retrieval (IR) have included the integration of string search in some EHRs, similar to the “Ctrl-F” or “Find” function now frequently available on everyday platforms [18]. However, the effectiveness of string search is limited for heterogeneous clinical text; therefore, studies have also considered semantic search algorithms [19-22]. A large-scale retrospective analysis of searches performed in an EHR found that the use of search varied considerably across and within user roles, with physicians and pharmacists being the most active user groups [19]. A review of the use of search within EHRs found that few articles focussed on the impact of search within clinical workflows [23]; one study with 7 diabetes experts found that content-based search was both faster and more accurate than conventional search for finding relevant information [20], another study with 10 family and internal medicine physicians found that semantic search gave faster medical notes navigation for IR tasks [21], and a final study with 4 students found that a semantic search tool enabled faster clinical note summarisation [22]. Only one of the described studies [20] used a crossover study design. In this paper, a larger study is reported (n=35 valid task completions, n=42 qualitative responses), in which a 3-way prospective crossover study was conducted, comparing standard string search with no search and with Natural Language Processing (NLP)-enhanced search. The custom NLP-enhanced search tool combines ontologies with fuzzy matching to offer search functionality which captures not only semantically related terms (e.g., synonyms and hyponyms) but also linguistic alternative (miss)spellings and word forms of the search term. A simulated clinical environment was used alongside target-user feedback to determine whether search tools could make clinical workflows more efficient and reduce clinicians’ cognitive burdens when attempting to find information.

2 Materials and Methods

2.1 Search tools

The string search function is an open-source JavaScript library implementation^a. NLP-enhanced search is a proprietary rule-based algorithm (developed at Canon Medical Research Europe) that leverages NLP techniques such as edit distance and stemming in conjunction with medical knowledge bases, notably the Unified Medical Language System (UMLS) semantic web, Metathesaurus [24], and the Wikipedia [25] and OpenMD medical abbreviation lists [26]. These sources are used to expand the original search term into a list of equivalent terms which are then located in the text. The tool was designed to locate linguistic variants such as misspellings and alternative spellings, wordforms, and abbreviations, as well as additional semantic synonyms.

Search tools were integrated into a patient-centric viewer (EHR research software), which allowed the user to type in a search term and view the highlighted findings within the retrieved subset of documents, which the user could scroll through. In the case of no search, the user was expected to scroll through the patient’s EHR to find the relevant information. Figure 1 illustrates the difference between the two search tools in the patient-centric viewer.

• Download figure
• Open in new tab

Figure 1:

Example results for string search (A) and NLP-enhanced search (B) for the search term “heart”. String search returns only direct matches to “heart” (green highlights) whereas NLP-enhanced search returns also semantically related terms (yellow highlights), such as: “coronary”, the misspelling of atrial (fibrillation) as “atriall”, and the appearance of “heart” within the abbreviation of heart failure, “HF”.

2.2 Simulating a Clinical Workflow

Free text medical documents were synthesised for three fictional patients. These materials were paired with corresponding sets of ten IR questions for each patient, grounded in relevant and realistic clinical scenarios. Patient documents were uploaded into the patient-centric viewer. Questions were uploaded onto a custom Evaluation Platform built using Vue.js that also displayed the clinical scenarios and task-specific instructions for the evaluator. Below we describe the document synthesis and question generation in more detail.

Patient document synthesis

Three patient profiles were created with varying age, sex, ethnic background, social history, and medical history. The three patients were assigned primary medical specialties of respiratory, neurology, and oncology. For each patient, 20 documents were created by selecting and augmenting publicly available anonymised medical documents [27], as well as manually synthesising additional documents, to provide a patient EHR with a coherent chronological sequence of clinical events. Documents were varied and included discharge letters, outpatient clinic letters, operation notes, and general practice (GP) referral letters. To imitate real-world medical text, common misspellings, abbreviations and acronyms were included in the text, using investigator clinical experience (HW) and reference papers [13].

Clinical scenarios and question generation

For each task, clinical scenarios were designed to focus on real-world situations where information can be extracted from patient notes. To ensure the tasks were comparable across patients (and therefore interventions), a master template of 10 questions prompting IR was created, which was then tailored to fit each patient scenario. Questions were inspired by past medical exam questions [28] and investigators’ clinical experience (HW, FM). Requested information resembled that required in typical clinical workflows to support clinical decision-making. Care was taken to ensure task questions tested search function and not clinical knowledge or judgement, therefore all answers could be found by searching the respective patient’s notes. Questions required a mixture of multiple choice and free text responses. Examples of scenarios and corresponding questions for each patient can be seen in Figure 2.

• Download figure
• Open in new tab

Figure 2:

Examples of clinical scenarios for each patient and their corresponding question-answer options. Scenarios aimed to simulate a standard clinical workflow, providing context for the questions.

2.5 Data Collection

Evaluators were provided with secure remote access to the evaluation environment (Figure 3), allowing the evaluation to be performed remotely from personal machines. Using this setup, evaluators could view the patient-centric viewer and the Evaluation Platform. Answers had to be inputted sequentially on the Evaluation Platform, which did not allow evaluators to return to a question once they had submitted an answer.

• Download figure
• Open in new tab

Figure 3:

Screenshot of evaluation environment during a task. Evaluators only had permission to view the two relevant sites: the patient-centric viewer (left) and Evaluation Platform (right). The patient-centric viewer contains the synthetic patient documents for a given patient (in this case “Joseph Williams”) with “hba1c” as the search term. The Evaluation Platform detailed the clinical scenarios, task-specific instructions, along with the question-and-answer sections.

During each task, the evaluators submitted answers to the task questions through the Evaluation Platform. To ensure accurate recording of task times, evaluators were asked to do each task in one go, and to take breaks between tasks rather than during tasks. Evaluators were free to spend as long as they needed on each task. In addition, a search log was kept that recorded the search terms entered by the evaluator along with the search functionality used, plus the time spent on each question.

3 Results

4 Discussion

Our results showed a significant increase in accuracy when NLP-enhanced search was used compared to string search and no search, whilst both NLP-enhanced search and string search offered time savings. There was a perception of easier navigation from evaluators and a measured decrease in required interactivity in the case of NLP-enhanced search (lower number of search terms compared to string search). We caveat this conclusion with the observation that the strict parameter settings of string search meant search terms matched only to whole words, not substrings; this increased the number of terms that evaluators used and potentially reduced the search accuracy, compared to a string search version that matches also to substrings.

There is limited literature on the potential impact of EHR search tools on day-to-day clinical care [29]. Our results support those of previous studies [20-22] which find that semantic search tools allow faster and more accurate EHR task completion in simulated clinical workflows. A related study showed Artificial Intelligence-optimised patient records to give speed improvements in answering clinical questions whilst maintaining the same accuracy [30]. Interestingly, the impact of the patient record search engine MorphoSaurus has been measured in a real clinical setting [31], albeit with user surveys only. This method would have had the benefit of involving real-world stresses such as task interruptions and time pressure, as well as the key element of patient interaction. Importantly, however, our method of using a controlled simulated clinical environment enabled us to control for variables such as distractions or interruptions, as well as variation in the complexity and length of medical records. Additionally, our crossover design controlled for individual participants’ ability, experience and diligence. This enabled robust comparison of quantitative and qualitative data for each search type, while minimising the impact of confounding factors.

Overall, evaluator feedback suggested that the optimum approach to navigating clinical notes is a hybrid of manual browsing and search, depending on the context. In the real-world, NLP-enhanced search is likely best employed as a complementary tool to aid clinical users in navigating clinical notes, with the ability to manually parse and ingest relevant facts from a complex medical history remaining important.

5 Conclusion

In conclusion, this study suggests that search tools have a positive effect on both the measured and perceived accuracy and ease of clinical IR. Search tools that can leverage NLP techniques are more effective for retrieving all relevant terms from heterogeneous medical free text. There is potential to reduce clinicians’ cognitive burden and make clinical workflows more efficient. A critical direction for future research is to assess the use of search tools in real-world clinical practice.

Authors’ Contributions

• Eunsoo H Park co-designed the study, co-designed the patient histories, reviewed the synthetic patient notes, designed the tasks, designed the clinical feedback survey, organised evaluator recruitment, recorded training materials for evaluators, performed preliminary analysis of the findings, and contributed to the paper draft.

• Hannah I Watson co-designed the study, co-designed the patient histories, created the synthetic patient notes, reviewed the tasks, reviewed the clinical feedback survey, supported evaluator recruitment, organised the infrastructure for the practical evaluation, contributed to & reviewed the analysis, and contributed to & reviewed the paper draft.

• Felicity V Mehendale co-designed the study, reviewed the patient histories, reviewed the synthetic patient notes, reviewed the tasks, reviewed the clinical feedback survey, reviewed the analysis, and contributed to & reviewed the paper draft.

• Alison Q O’Neil co-designed the study, organised provision of the NLP-enhanced search, reviewed the tasks, reviewed the clinical feedback survey, contributed to & reviewed the analysis, and contributed to & reviewed the paper draft.

Funding

This work is part of the Industrial Centre for AI Research in digital Diagnostics (iCAIRD), which is funded by Innovate UK on behalf of UK Research and Innovation (UKRI) project number 104690. FV Mehendale’s research at the University of Edinburgh is supported by the Caledonian Heritable Foundation.

Conflicts of Interest

HI Watson and AQ O’Neil are employees of Canon Medical Research Europe, who provided the software and algorithms for this evaluation. EH Park was sponsored by Canon Medical Research Europe during her Spring 2021 BSc research project at the University of Edinburgh (“Evaluation of a natural language processing algorithm for searching medical documents”) which was the basis for this evaluation. EH Park had previously performed paid annotation work for the development of the NLP-enhanced search tool.