Evaluation of Patient-Level Retrieval from Electronic Health Record Data for a Cohort Discovery Task

Objective Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well-understood. The objective of this research was to assess patient-level information retrieval (IR) methods using electronic health records (EHR), and to investigate the interplay between commonly used IR approaches and the cohort definition structure.

Materials and Methods Using the Cranfield IR evaluation methodology, we developed a test collection based on 56 test topics characterizing patient cohort requests for various clinical studies. Test collection data was derived from patient records originating from OHSU’s EHR data warehouse. Automated IR tasks were performed, varying four different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created 56 structured Boolean queries for the 56 topics for performance comparisons. Finally, we designed 59 taxonomy characteristics to classify the structure of the 56 topics. Six topic complexity measures were derived from these characteristics for further evaluation using a beta regression simulation.

Results The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision, but were still not able to recall all relevant patients found by the automated queries. We also found strong performance associations with the six complexity measures created from the topic taxonomy, and interactions with automated query parameter settings.

Conclusion While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in the methods tested for this study. Some of the characteristics derived from a query taxonomy could lead to improved selection of approaches based on the structure of the topic of interest. Insights gained here will help guide future work to develop new methods for patient-level cohort discovery with EHR data.