Use of Recommended Real-World Methods for Electronic Health Record Data Analysis Has Not Improved Over 10 Years

Chenyu Li, Abdulrahman M. Alsheikh, Karen A. Robinson, Harold P. Lehmann
doi: https://doi.org/10.1101/2023.06.21.23291706

ABSTRACT

Background and Purpose To document the use of recommended Real-World Methods (RWM) in Electronic Health Record (EHR)-based analysis in biomedical research over 10 years.

Methods Sampled-article scoping review of methods used in EHR-based biomedical research. We developed a search strategy to identify reports of biomedical research based on EHR data and systematically sampled articles from different ranges of years (epochs) between 2010 and 2019 to establish a trajectory of use of recommended RWM. Methods were classified by 3 phases of research: pre-analytic (missing data), analytic (specific methods), and post-analytic (sensitivity analysis). The primary outcome was the proportion of studies using recommended RWM within each epoch. Meta-regressions were performed to examine trends.

Data Synthesis Five epochs were defined between 2010 and 2019 with 35 studies selected per epoch as pre-defined by a sample size calculation. Of the 175 articles reviewed, 70 (40.%) reported recommended RWM in any of the 3 phases of research. The breakdown for the most recent year in the dataset, 2019, was 14.% (95% confidence interval 2.7%, 26.%), 14.% (2.7%, 26.%), and 11.% (0.89%, 22.%), for assessing missing data, using specific methods, and performing sensitivity analysis, respectively. Only 3.4 % of studies used appropriate methods for each phase of research. Meta-regression slopes for each of the three phases were statistically 0.

Limitation and Conclusions The underuse of recommended Real-World Methods (RWM) in EHR-based biomedical research remains a concern, with less than 50% of reports using these methods in any phase of research over the last decade. This lack of use indicates a continued risk of bias in the EHR-based literature.

INTRODUCTION

Use of Real-World Data (RWD) to generate Real-World Evidence (RWE) is playing an increasing role in health care decisions worldwide.[1] RWD refers to information acquired from a range of sources generated in the process of clinical care.[1] Electronic health records (EHRs), claims databases, patient registries, and other types of data repositories may be among these sources.[1,2] Turning RWD into RWE can provide useful insights into the safety, effectiveness, and value of medical products and treatments in real-world clinical settings.[3] There is a growing interest in using RWE by stakeholders, including policymakers, biomedical researchers, clinicians, and medical product developers.[4–10] Despite potential broad utility, the quality of RWE, compared with evidence generated from Randomized Controlled Trials (RCTs) — still the gold-standard of clinical research[3]—is found wanting. The methods recommended to derive RWE from RWD are intended to reduce bias (especially confounding) and to allow associational and even causal inference when using observational data. Not using the appropriate methods may result in inaccurate, biased, or even inequitable algorithms.[11] Ultimately, applying such results or algorithms in real-world practice may result in the improper treatment of patients.[12]

EHRs are essential RWD sources[1]. The 21st Century Cures Act in the U.S. was signed into law in 2016 and calls on Congress to “evaluate the potential use of real-world evidence.”[13] FDA has developed guidelines on the various uses of RWE, for example.[2,14,15] The guidelines for data analysis and RWE generation methodology remain under discussion. [1,2,14–16] The COVID-19 pandemic has surfaced the importance of RWE for addressing new diseases,[17] while, at the same time, methodological concerns about the use of RWE continue to be raised.[18–20]

Discussions about Real World Data analysis address 3 phases: pre-analytic (missing data), analytic (specific methods), and post-analytic (sensitivity analysis). During phase 1, the pre-analytic phase (the preparation of data for analysis), addressing missing data is of special concern, because researchers have no control of what data are collected or recorded, and so traditional approaches may not apply, necessitating, for instance, greater attention to data missing not at random (MNAR).[21,22] During phase 2, the analytic phase (the statistical modeling and analysis of the data), many Analytic Real World Methods (ARWM) have been developed to analyze RWD, from the statistical, epidemiological, and computer-science communities.[19,23–25] During phase 3, the post-analytic phase, sensitivity analyses establish the robustness of conclusions to reasonable uncertainty in assumptions.[24–27] In the current paper, we call the entire set of methods in these 3 phases of research “recommended Real World Methods (recommended RWM).”

Our objective was to assess use of recommended RWM for each of the 3 phases in biomedical research assessing hypothesized associations and treatment effectiveness, based on EHR data, as documented in the peer-reviewed literature and to assess whether this use has changed over the past 10 years. We focused on this research domain, because it is the focus of much desire for using EHRs for clinical research.[28]

Because our goal was to identify proportions, we did not need to identify every relevant article published in the past 10 years as would be required for a standard systematic review. Instead, we sampled from papers identified through a thorough literature search and applied a systematic-review approach to the sample papers. We call this approach a sampled-article scoping review.

METHODS

Our scoping review was based on randomly-sampled papers, from which we aimed (1) to define the articles that would be eligible for review (i.e., operationalize the intention of “papers that used EHR as the main data source for biomedical research”), (2) to operationalize the list of recommended RWM; and (3) to establish the proportion of use of the these methods for time epochs over the past 10 years. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Review (PRISMA-ScR) checklist in reporting this scoping review. [29]

Given the novelty of our method, for items (1) and (2), we conducted a pilot review[30] to derive the search strategy, to refine our inclusion and exclusion criteria, to establish lists of keywords for identifying recommended RWM, to estimate inter-rater reliability, and to generate data needed for the sample-size calculation for the study.

Data Sources and Searches

We sought studies that used EHR data for traditional biomedical-research aims of assessing hypothesized associations. We used an NLM-supplied search strategy for keyword “Electronic Health Records”[31] to search PubMed. We modified the NLM supplied strategy based on results from pilot; see Supplement (Appendix Table 1 Search Strategies Details) for details. We limited the publication date to 2010/01/01-2019/12/31. The search was conducted on 2020/11/09. The detailed search strategy is presented in Appendix Table 1.

We developed the list of eligibility criteria iteratively, see Table 1. Because our focus was on associational and causal studies, we excluded review papers, research reports based on data from claims, genomic analysis, manually collected registries, or prospective clinical trials. Research that was not patient focused or biomedical, such as those focused on physician behavior, information-system evaluation, health-services evaluation, and new information technology in healthcare, were also excluded. Finally, research based on unstructured and semi-structured data which needed natural language processing or text mining was excluded.

View this table:
Table 1

Inclusion Exclusion Criteria

Study Selection

We divided the 10-year period, 2010−2019 into 5 epochs: 2010−2013, 2014−2016, 2017, 2018, and 2019. We hypothesized that the proportions would rise, because of guidance that came out during the decade, specifically in the years 2013[14], 2016[32], 2018[2,15,16,33], and 2019.[34] Our goal was to get a sense of change over time, so we grouped older years together, giving more recent years finer precision. Older years had fewer articles, so we wanted our sampling to be fair to each epoch. By equalizing denominators (approximately), equal numerators made sense. We ended at 2019, as research published during the COVID-19 pandemic is likely qualitatively different.

Our statistical goal was to distinguish proportions between 2 epochs. The proportion of recommended RWM use from our pilot review was 10% (Pilot study results see Appendix Table 3, Appendix Table 4); that percentage, coupled with a desired confidence level (α) of 0.95, was used to calculate the sample size [35] for our scoping review of 35 eligible papers per epoch. We sampled 300 citations within each epoch, and then screened papers based on inclusion/exclusion criteria, until we reached 35 included papers per epoch. Random selection was performed by generating random numbers via Python (package Random) within the set of numbers representing all papers retrieved for an epoch.

Citations were downloaded and titles and abstracts were screened by one person (CL). We calculated the inter-rater reliability of CL and AA in the pilot study, the Cohen’s Kappa was used to measure inter-rater reliability, κ=0.95, and the agreement between the two reviewers was 98%. If there was any uncertainty, the article was included for full-article screening, and any remaining questions were discussed among 3 of us (HL, CL, AA).

Data Extraction and Quality Assessment

For each randomly selected article, we recorded study type, study design and methods used. The key outcome variable was whether recommended RWM were used for each of the three phases of research. We documented the RWM methods used in included papers, then matched these to the recommended RWMs we identified from regulator guidelines, books, and RWD meeting recommendations.[2,14,15,25,26,34,36,37]

We then scored each article for whether use of a recommended RWM was reported; details are provided in Appendix Table 2. For phase 1, studies that even simply deleted records with missing data were tagged as “Addressed missing data.” Papers that reported performing any of the following were tagged as “assessed missing data”: deletion methods with examining the sensitivity of results to the missing completely at random (MCAR) and missing at random (MAR) assumptions,[26] single imputation methods, and model based methods.[26,38] For phase 2, we looked for words listed in the “Analytic Real World Method” section of Appendix Table. To broaden the definition of RWM as broadly as possible, any machine-learning methods combined with causal inference also were considered as RWM. [34] (See Appendix Table 2)

For phase 3, we looked for the words, “sensitivity” or “sensitivity analysis.”

Where judgment was needed, we biased in favor of ascribing use of RWM. Therefore, our final proportions are upper bounds within each epoch of the use of such methods. The full list of 35 RWM and 45 traditional methods is provided in Appendix Table 2.

Paper characteristics details (see Appendix Table 9 available in GitHub, due to size:1) were extracted by a single reader (CL), discussing ambiguous results with the senior author (HL). Beyond citation data, data collected include research design (primary objective), study design type, association type, key variables (exposure, outcome, confounding), information pertaining to the 3 phases, and use of reporting checklists (such as STROBE[39] or RECORD[40]). We also searched the supplementary materials of all the papers, looking for mentions of our target RWM terms. To supplement the manual reading of an article, we searched for keywords through text processing of the article and supplement full text, using the vocabulary assembled in the pilot.

Data Synthesis and Analysis

The percentages of papers within each epoch using recommended RWM were calculated, along with 95% confidence intervals, and graphed over time. To assess the impact of any article or study characteristics on rates, we conducted a mixed-effects meta-regression using restricted maximum-likelihood (ReML) using PyMARE, in Python 3.7 using Jupyter Notebook. [41] Three meta-regressions, one for each phase of research, were done with time as the independent variable. Since the epoch lengths were unequal, we used the midpoint of each epoch in the analysis. To make the intercept meaningful, we subtracted the epoch mid-point (2015) from the epoch year values.

Sensitivity Analysis

We performed sensitivity analyses for the current study by loosening our categories for phases 1 and 2, to assess the impact of changed definitions. For phase 1 missingness and phase 2 ARWM, we further calculated estimated percentages and their confidence intervals of methods used, using maximal definitions of “assessing missing data” (i.e., giving credit to articles that simply mentioned “missingness” or missing data), and phase 2 ARWM (i.e., adding methods that were not listed by the FDA but still not typical analytical methods; see Appendix Table 2), which defined Combined Missingness and Combined ARWM categories, respectively.

For phase 3, we found no other synonyms for the concept of sensitivity analysis, the third phase of research and thus had no further calculations to perform.

Results

Study Selection Flow

The study selection flow is summarized using a PRISMA 2020 flow diagram (Figure 1).[42] The literature search retrieved 5885 citations (for retrieved articles’ distribution, see Appendix Figure 1) We generated a random list of article identifiers for full-text review. We screened the full text of 392 articles total to achieve our target of 35 articles for each epoch. This process resulted in 175 total articles for analysis (for included paper list see Appendix Table 5). Reasons for, and numbers of, exclusions are given in Figure 1.

Figure 1
Figure 1 PRISMA Flow Diagram

The characteristics of the 175 included papers (for distribution, see Appendix Figure 2) are provided in Table 2. Of the included papers, 101 (57.7%) were conducted in the North America countries, 37 (21.%) in European countries, and 32 (18.%) in Asia (detailed country distribution see Appendix Table 6). Regarding intention, 142 of the papers aimed at association, and 33, at assessing interventions, with the proportion comparable in each epoch. In terms of design, we characterized 120 (68.6%) as retrospective cohort studies, 14 (8.0%) as retrospective cross-sectional studies, 14 (8.0%) as retrospective chart reviews, and 12 (6.9%) as prospective cohort studies. While a wide variety of statistical tools was reported, 28 (16.0%) reports did not provide such details (detailed statistics tools used see Appendix Table 7). Whereas 4 (2.%) out of 175 papers reported that the research was conducted followed the STROBE[39] check list, no other papers reported using any reporting checklist.

View this table:
Table 2.

Included Paper Characteristics, by Epoch

Analysis

Only 1 supplement added the mention of assessed missing data beyond what was found in the main articles. This paper was included in our figures and tables. Thus, for phase 1, the total percentage of articles that assessed missing data was 22%.

Regarding phase 3, the number of papers that reported sensitivity analysis in 2019 is 4 (11%), with a confidence-interval upper bound of 22.%. The range over the 10 years was 12.% to 23.%.

Regarding changes over the years, the raw percentage of recommended methods in each phase, assessing missing data, ARWM, and sensitivity analysis, varied between 10% and 40%. (Figure 2 and Appendix Table 8,) While the proportions seem to rise by 2017, they return to low values in 2018 and 2019.

Figure 2
Figure 2 Estimated Percentage of Methods Used in RWD Research, by Epoch

Regarding phase 2, the number of papers that used ARWM in 2019 is 5 (14.%), (with a confidence-interval upper bound of 26.%). Regarding specific traditional methods, 96 (55.%) papers used linear or logistic regressions, 41 papers used survival analysis, with 26 (63. %) not adjusting for baseline or time-varying confounders. [43]

Only 6 papers (3.4%), with a confidence-interval upper bounds of less than 6%, overall addressed recommended RWM in all 3 phases.

Meta-regressions

The meta-regressions modeled the percentage of method used in each of the three phases as a function of an epoch. The coefficients for the meta-regressions are shown in Table 3. As an example, the coefficient for assessed missing data is −1%, meaning an average drop of 1 percent per epoch after the baseline intercept value of 22% (in 2015) assessing missingness. However, for all other methodology coefficients, the coefficients were close to and statistically equivalent to 0. The percentage of studies using ARWM or using sensitivity analysis did not increase over time.

View this table:
Table 3.

Meta-regression for Three Phases

Sensitivity Analysis

In the case that our categories were too restrictive, we “gave credit” to articles using broader criteria for missingness and RWM definitions. Figure 2 shows the estimated percentages and confidence intervals using these broader criteria (“Combined Missingness” and “Combined ARWM”). The estimates for missingness go as high as 60%, and for ARWM, as high as 29%. As in the meta-regression, there was statistically no improvement over time. Because we found no other synonyms for the concept of sensitivity analysis, we did not pursue the third phase.

DISCUSSION

We took a novel random-sampling approach for this scoping review to assess the proportion of articles that reported methods recommended for analyzing RWD to generate RWE, over the last 10 years. We divided the RWD-analysis process into 3 phases. The number of papers reporting recommended RWM in all 3 phases was only 6 (3.4%). The proportion of studies using recommended methods within each phase has not changed substantively between 2010 and 2019, whether looking at assessing or just mentioning missing data (phase 1; between 14 and 40%), reporting ARWM (phase 2; between 20 and 40%), or using sensitivity analysis (phase 3; between 9 and 29%). These low rates occurred despite multiple calls by methodology and funding organizations for just such methods throughout this time period.[2,14–16,32,34] While there seems to have been a peak in 2017, the later 2018 and 2019 rates of use of ARWM are no better than in 2010−2013. We biased our inclusion and tagging criteria to be as generous towards use of RWM as possible, so these percentages are probably upper bounds to the true percentages.

The low rates we documented are accompanied by a low documented rate of using methodology checklists. A recent new checklist for reporting RWE—STarT-RWE—includes the requirement to describe how missing data are addressed throughout the analysis and subgroup analyses.[44,45] Of our included papers, 81 (46.%) mentioned a missing-data issue in their data cleaning or limitation section. Of the 81 that did report on missing data, only 39 (48.%) took action beyond mentioning the issue. While statisticians often impute missing data to make maximal use of data that are available, this imputation presumes that missing data are missing at random. However, missed EHR data is generally missing not at random. [21] [46] Methods for missing not at random (MNAR) are more complex and difficult, yet appropriate.[5,47] None of the studies discussed this key concern of EHR data. There are many weaknesses in using EHR data that go beyond the statistical approaches listed in this paper. For instance, improving data quality and accuracy would be paramount, before attempting advanced analytic methods.[48,49] We submit that attention to missing data, ARWM, and sensitivity analysis comprise the minimum of what is needed.

The percentage of papers that used ARWM is low, and remained so for the entire time period, with less than a quarter doing so (although the confidence interval goes as high as 37%). Taking the case of a single, relatively common analysis—survival curves—only 37% used the appropriate method for RWD[26,43]. In addition, the meta regressions suggest that the assessment of missing data, the use of, or the deployment of sensitivity analysis had not changed over 10 years. One could argue that many of the methods listed in Supplementary Table 2 are too recent to expect to appear in our sample. However, “bias,” “adjusting,” “confounding” and “non-adherence” are concerns in epidemiology and evidence-based medicine for decades.[50,51] The low percentage of use of recommended RWM has important implications for the results of the studies: Improper attention to missing data and to selection bias within EHR data means that results from one dataset cannot be applied to other populations in a predictable (and safe) way.[52–54] Improper analytic methods limit generalizability. Lack of sensitivity analysis or other robustness approaches may hide the fragility of the conclusions to contexts outside the RWD dataset.

While we used the FDA meeting[1,2,14–16] as a rich list of modern methods, the sentiment of the need for specific methods to deal with RWD is well established. A standard textbook of Epidemiology calls for bias assessment in “secondary data analysis” (using the definition of RWD). [36] The National Academy of Medicine raises bias issues and results calibration in their 2013 Observational Studies in a Learning Health System workshop summary.[55] The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) in 2017 developed recommendations for “Good practices for real-world data studies of treatment and/or comparative effectiveness.” In their discussion, they write:

Threats to the validity of RWD studies of the effects of interventions include unmeasured confounding, measurement error, missing data, model misspecification, selection bias, and fraud. Scholarly journals publish many observational studies, often without requiring the authors to report a thorough exploration of these threats to validity. Unless the authors identify these threats and provide guidance about how they could bias study results, consumers of observational studies must trust the study authors to conduct the study with integrity and report it transparently.[37]

The low percentages we have documented also have important implications for the credibility of EHR-based research. Use of EHRs as a source of data for biomedical research has increased linearly over the past 10 years, rising from 228 biomedical-research articles published based on such data in 2010 to 1104 in 2019 (see Appendix Figure 1). Certainly, such use will only grow, as multi-center efforts, like The National Patient-Centered Clinical Research Network (PCORnet) [56], Observational Health Data Sciences and Informatics (OHDSI)[57] and National COVID Cohort Collaborative (N3C)[58] mature, and as the research community becomes more comfortable with the use of these data. This increase in attention means that it is even more important that proper methods are used for analyzing these data. Furthermore, if the larger corpus of EHR-based research is deemed non-generalizable, biased, and fragile, then the efforts by these organizations may be impugned by the very audience they aim to convince.

One could argue that our study ended too early to see a change a research methodology as a results of guidelines published in 2010s. After all, a 2011 study determined that translating health research takes 17 years.[59] Our results, then, serve as a baseline for RWD research to monitor changes in methodology to satisfy already-published research guidelines.

Because of the complexity of RWD, it is not appropriate to use traditional data processing methods with these large datasets.[60–62] In order to ensure internal and external validity in EHR-based research, researchers must determine whether the data are accurately extracted, adequately adjusted, correctly analyzed and cogently presented.[26] To analyze RWD properly requires multidisciplinary teams comprising clinicians, informaticians, epidemiologists, biostatisticians and data scientists.

The percentage of published studies using recommended RWM has not risen over the past 10 years, despite guidance being issued in multiple years by a variety of sources. Alternative approaches are needed. OHDSI has developed tools to conduct real-world evidence generation.[63] However, at any one institution, a small group of investigators may lack the ability to implement such a sophisticated toolset. Beyond promoting the use of RWM through reporting checklists [64], we suggest a third alternative is to provide analytic decision support, where the analyst receives suggestions from the software in the process of developing an analysis. Examples could include libraries of methods (articles), appropriately tagged; libraries of software code that implements those methods; and metrics to assess successful implementation of the methods. Using the Agency for Healthcare Research and Quality (AHRQ) principle,[65] we suggest that such decision support could do for RWE what EHR-based decision support has done for guideline adherence.[1,66]

LIMITATIONS

There are several limitations in this research. First, the main data collection was conducted by a single interpreter, which could inject subjectivity into the selection of articles and extraction of significant data. To minimize this potential bias, a uniform approach was adhered to throughout the review process, we had a pilot study that included two reviewers collected data that had a high inter-rater reliability, and a senior reviewer (HL) was consulted in cases of doubt. In addition, the included papers were randomly selected per epoch, providing a representative sample of the literature. Notwithstanding these precautions, it is likely that some important publications were overlooked, or that the sole reviewer impacted the interpretation of the data. Future research may choose to employ several reviewers to boost the validity and applicability of their findings.

Second, we did not include all eligible articles, because our goal was to assess the proportion of such articles using appropriate methods; random sampling is the basis of much biomedical research, our sampling method ensured a representative subset of the literature, allowing for a sound interpretation of the prevalence of recommended RWM usage in the RWD analysis process over the past decade.

Third, we did not include articles from 2020 or later because the COVID-19 pandemic was such a disruption to research practice. Although removing articles published in 2020 or later may limit the scope of our analysis, we made this decision because the COVID-19 pandemic has produced an unprecedented disturbance in research practice. Concerns have also been raised over the quality of research published during the epidemic, as several papers were hurried to publication and lacked sufficient peer review.[67–69] To ensure that our analysis included a more typical sample of studies conducted under regular research conditions, we omitted articles published in 2020 or later. We note, however, that this omission may limit the applicability of our results to more recent studies on RWE approaches. Future research may include analysis of more recent papers and evaluate any modifications to the use of approved RWE procedures considering the pandemic.

Fourth, some EHR-based studies were excluded, such as implementation studies and studies of physician behavior. The analyses of such data are often even more complicated than the more traditional studies of association or medical-intervention effectiveness.[70,71] Certainly, a review of such studies along the lines of the current article would provide a view into how EHR data are used in those contexts as well. We excluded NLP because use of such methods requires informatics expertise. Of 392 articles reviewed, only 4 were rejected because of NLP.

Last but not least, it is possible that not all methodological details were included in the Methods sections which would have led to low observed percentage. However, we reviewed all supplement sections and found only a single additional mention.

CONCLUSION

The proportion of published clinical research papers based on EHR data using recommended real-world evidence methods is low and has not increased significantly over a 10-year interval. The low percentages we documented may reduce the credibility of EHR-based research, if the larger corpus of EHR-based research is deemed non-generalizable, biased, and fragile. New approaches are needed to promote the use of these methods to improve the validity of studies, to reduce biases, to improve RWE quality, and to better support healthcare decisions. Not doing so risks another 10 years of no improvement in the derivation of evidence from RWD.

FUNDING STATEMENT

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

COMPETING INTERESTS STATEMENT

Authors have no competing interests to declare for this research.

CONTRIBUTORSHIP STATEMENT

HL, CL, and KR designed the study. CL conducted the study including data curation, analysis and interpretation. AA contributed to data curation as a second reviewer. HL reviewed data analysis and interpretation. CL drafted the manuscript that reviewed and revised by HL and KR. implementation.

Data Availability Statement

The analytic data used for this scoping review was collected during the reviewing process. The retrieved EHR based articles, reviewed article details, and analytic code are available on GitHub at https://github.com/ChenyuL/RWE-in-EHR-Data-Analysis. These resources are provided to facilitate replication and further investigation of the findings presented in this manuscript.

Appendix and Supplement

Search Strategy Details (combined)

The following table provides details on the pilot study and the main study search strategies, showing the broader search (all journals) used in the main study.

View this table:
Appendix Table 1 Search Strategies

During the pilot review, we identified several issues with that search strategy, for instance it was limited to 8 informatics journals, while its terms for “Biomedical Quantitative Study” were too broad12. In the main study, we removed the journal restrictions on NLM-supplied Electronic Health Records search strategy, details see Appendix Table 1. “Biomedical Research” was operationalized as the PubMed Publication Type, “Study Characteristics.” To limit the result to quantitative biomedical studies with data analytic methods, we added keywords “data”[All Fields] AND “analy*”[All Fields] to the search. We used PubMed clinical filters to focus on diagnosis, etiology, prognosis studies, the broad definition for searching diagnosis, etiology, and prognosis has sensitivity of 90%, 93%, and 90%, respectively.3

The explanation provided in Appendix 2 of our manuscript refers to the process we used to categorize Real-World Methods (RWM) and Traditional Methods for dealing with missing data in EHR-based biomedical research.

Specifically, when we reviewed the articles, we looked for any mention of a method being used for RWD analysis. If a method was specifically mentioned in the context of RWD analysis, we considered it to be an RWM and placed it on the left side of the table in Appendix 2.

However, if a method was not explicitly mentioned in the context of RWD analysis, we considered it to be a Traditional Method and placed it on the right side of the table.

For example, in the case of logistic regression, while it is often used in RWD analysis, it can also be used in traditional data analysis. Therefore, if it was not explicitly mentioned in the context of RWD analysis, we considered it to be a Traditional Method and placed it on the right side of the table.

It is important to note that our categorization of methods as RWM or Traditional Methods was based on our interpretation of the articles and the information provided by the authors. It is possible that other researchers may categorize these methods differently, based on their own interpretation of the literature or other criteria.

Overall, the purpose of Appendix 2 was to provide a comprehensive list of methods used in EHR-based biomedical research, categorized according to whether they were RWM or Traditional Methods. This information can be used by researchers to select appropriate methods for their own studies and to compare the methods used in different studies.

View this table:
Appendix Table 2 Real World Methods List

Pilot Study Results

We sampled 300 papers in order to gain a stable estimate of our target proportions

View this table:
Appendix Table 3 Number of articles retrieved, by year
View this table:
Appendix Table 2 Pilot Study Estimated Proportions and Confidence Interval of Methods used in EHRs Based Research

Main Study Results

Appendix Figure 1:
Appendix Figure 1:

Published Papers Extracted from PubMed, by Year

Appendix Figure 1
Appendix Figure 1

Included Papers, by Year

View this table:
Appendix Table 5 Reviewed Paper List4
View this table:
Appendix Table 6 Main Study Included Paper Countries Details, by Epoch
View this table:
Appendix Table 7 Main Study Included Papers Statistics Tools Used, by Epoch
View this table:
Appendix Table 8 Percentage of Key Methodologies, by Epoch5

ACKNOWLEDGEMENTS

We gratefully acknowledge the help of Claire Twose, MLIS, Associate Director for Research Services at Johns Hopkins University, in refining the search strategy and reviewing search implementation. Chenyu Li is supported by University of Pittsburgh School of Medicine Department of Biomedical Informatics Dean’s scholarship.

Footnotes

REFERENCES

  1. 1.↵
  2. 2.↵
    FDA. Framework for FDA’s Real-World Evidence Program. Sec Dec. 2018 2018. https://www.fda.gov/media/120060/download.
  3. 3.↵
    Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-World Evidence - What Is It and What Can It Tell Us? N Engl J Med 2016;375(23):2293–97 doi: 10.1056/NEJMsb1609216 published Online First: 2016/12/14].
    OpenUrlCrossRefPubMed
  4. 4.↵
    Bartlett VL, Dhruva SS, Shah ND, Ryan P, Ross JS. Feasibility of Using Real-World Data to Replicate Clinical Trial Evidence. JAMA Netw Open 2019;2(10):e1912869 doi: 10.1001/jamanetworkopen.2019.12869 published Online First: 2019/10/10].
    OpenUrlCrossRef
  5. 5.↵
    Baumfeld Andre E, Reynolds R, Caubel P, Azoulay L, Dreyer NA. Trial designs using real-world data: The changing landscape of the regulatory approval process. Pharmacoepidemiol Drug Saf 2019 doi: 10.1002/pds.4932 published Online First: 2019/12/12].
    OpenUrlCrossRef
  6. 6.
    Blonde L, Khunti K, Harris SB, Meizinger C, Skolnik NS. Interpretation and Impact of Real-World Clinical Data for the Practicing Clinician. Adv Ther 2018;35(11):1763–74 doi: 10.1007/s12325-018-0805-y published Online First: 2018/10/26].
    OpenUrlCrossRefPubMed
  7. 7.
    Corrigan-Curay J, Sacks L, Woodcock J. Real-World Evidence and Real-World Data for Evaluating Drug Safety and Effectiveness. JAMA 2018;320(9):867–68 doi: 10.1001/jama.2018.10136 published Online First: 2018/08/15].
    OpenUrlCrossRefPubMed
  8. 8.
    Franklin JM, Schneeweiss S. When and How Can Real World Data Analyses Substitute for Randomized Controlled Trials? Clin Pharmacol Ther 2017;102(6):924–33 doi: 10.1002/cpt.857 published Online First: 2017/08/25].
    OpenUrlCrossRefPubMed
  9. 9.
    Makady A, de Boer A, Hillege H, Klungel O, Goettsch W. What Is Real-World Data? A Review of Definitions Based on Literature and Stakeholder Interviews. Value Health 2017;20(7):858–65 doi: 10.1016/j.jval.2017.03.008 published Online First: 2017/07/18].
    OpenUrlCrossRefPubMed
  10. 10.↵
    Miksad RA, Abernethy AP. Harnessing the Power of Real-World Evidence (RWE): A Checklist to Ensure Regulatory-Grade Data Quality. Clin Pharmacol Ther 2018;103(2):202–05 doi: 10.1002/cpt.946 published Online First: 2017/12/08].
    OpenUrlCrossRefPubMed
  11. 11.↵
    Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A. A Survey on Bias and Fairness in Machine Learning. arXiv pre-print server 2019 doi: None arxiv:1908.09635 published Online First.
  12. 12.↵
    Griffith GJ, Morris TT, Tudball MJ, et al. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nature Communications 2020;11(1) doi: 10.1038/s41467-020-19478-2 published Online First.
    OpenUrlCrossRefPubMed
  13. 13.↵
    Congress t. 21st Century Cures Act Sec 2016. https://www.congress.gov/bill/114th-congress/house-bill/34.
  14. 14.↵
    FDA-2011-D-0057. Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data. Sec May 2013. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/best-practices-conducting-and-reporting-pharmacoepidemiologic-safety-studies-using-electronic.
  15. 15.↵
    FDA-2016-D-1224. Use of Electronic Health Record Data in Clinical Investigations Guidance for Industry. Sec [Regulation] July 2018 2018. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-electronic-health-record-data-clinical-investigations-guidance-industry.
  16. 16.↵
    FDA. Submitting Documents Using Real-World Data and Real-World Evidence to FDA for Drugs and Biologics Guidance for Industry. Sec 2019. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/submitting-documents-using-real-world-data-and-real-world-evidence-fda-drugs-and-biologics-guidance.
  17. 17.↵
    Naidoo P, Bouharati C, Rambiritch V, et al. Real-world evidence and product development: Opportunities, challenges and risk mitigation. Wiener klinische Wochenschrift 2021;133(15-16):840–46 doi: 10.1007/s00508-021-01851-w published Online First.
    OpenUrlCrossRef
  18. 18.↵
    Khosla S, White R, Medina J, et al. Real world evidence (RWE) – a disruptive innovation or the quiet evolution of medical evidence generation? F1000Research 2018;7:111 doi: 10.12688/f1000research.13585.2 published Online First.
    OpenUrlCrossRef
  19. 19.↵
    Crown WH. Real-World Evidence, Causal Inference, and Machine Learning. Value in Health 2019;22(5):587–92 doi: 10.1016/j.jval.2019.03.001 published Online First.
    OpenUrlCrossRef
  20. 20.↵
    National Academies of Sciences EM, Division HM, Policy BHS, et al. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series: National Academies Press, 2019. 9780309488297;
  21. 21.↵
    Haneuse S, Arterburn D, Daniels MJ. Assessing Missing Data Assumptions in EHR-Based Studies: A Complex and Underappreciated Task. JAMA Network Open 2021;4(2):e210184 doi: 10.1001/jamanetworkopen.2021.0184 published Online First.
    OpenUrlCrossRef
  22. 22.↵
    Moore L, Hanley JA, Lavoie A, Turgeon A. Evaluating the validity of multiple imputation for missing physiological data in the national trauma data bank. J Emerg Trauma Shock 2009;2(2):73–9 doi: 10.4103/0974-2700.44774 published Online First: 2009/06/30].
    OpenUrlCrossRefPubMed
  23. 23.↵
    Yerrapragada G, Siadimas A, Babaeian A, Sharma V, O’Neill TJ. Machine Learning to Predict Tamoxifen Nonadherence Among US Commercially Insured Patients With Metastatic Breast Cancer. JCO Clinical Cancer Informatics 2021(5):814–25 doi: 10.1200/cci.20.00102 published Online First.
    OpenUrlCrossRef
  24. 24.↵
    Fang Y, Wang H, He W. A Statistical Roadmap for Journey from Real-World Data to Real-World Evidence. Therapeutic Innovation & Regulatory Science 2020;54(4):749–57 doi: 10.1007/s43441-019-00008-2 published Online First.
    OpenUrlCrossRef
  25. 25.↵
    Sturmer T, Wang T, Golightly YM, Keil A, Lund JL, Jonsson Funk M. Methodological considerations when analysing and interpreting real-world data. Rheumatology (Oxford) 2020;59(1):14–25 doi: 10.1093/rheumatology/kez320 published Online First: 2019/12/14].
    OpenUrlCrossRefPubMed
  26. 26.↵
    MIT-Critical-Data. Secondary Analysis of Electronic Health Records, 2016. ISBN-13: 978-3-319-43742-2; doi:10.1007/978-3-319-43742-2
  27. 27.↵
    Schneeweiss S. Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics. Pharmacoepidemiology and Drug Safety 2006;15(5):291–303 doi: https://doi.org/10.1002/pds.1200 published Online First.
    OpenUrlCrossRefPubMedWeb of Science
  28. 28.↵
    Casey JA, Schwartz BS, Stewart WF, Adler NE. Using Electronic Health Records for Population Health Research: A Review of Methods and Applications. Annu Rev Public Health 2016;37:61–81 doi: 10.1146/annurev-publhealth-032315-021353 published Online First: 2015/12/17].
    OpenUrlCrossRefPubMed
  29. 29.↵
    PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine 2018;169(7):467–73 doi: 10.7326/m18-0850 %m 30178033 published Online First.
    OpenUrlCrossRefPubMed
  30. 30.↵
    Li C. ANALYTIC METHODS USED IN REAL WORLD DATA BASED BIOMEDICAL RESEARCH-A SCOPING REVIEW. Johns Hopkins University, 2020.
  31. 31.↵
    NLM. MEDLINE / PubMed Search Strategy & Electronic Health Record Information Resources. Sec 2020/03/23 2020. https://www.nlm.nih.gov/services/queries/ehr_details.html.
  32. 32.↵
    Public Policy Committee ISoP. Guidelines for good pharmacoepidemiology practice (GPP). Pharmacoepidemiol Drug Saf 2016;25(1):2–10 doi: 10.1002/pds.3891 published Online First: 2015/11/06].
    OpenUrlCrossRefPubMed
  33. 33.↵
    FDA. FDA’s Sentinel Initiative. Sec 2008. https://www.fda.gov/safety/fdas-sentinel-initiative.
  34. 34.↵
  35. 35.↵
    Pourhoseingholi MA, Vahedi M, Rahimzadeh M. Sample size calculation in medical studies. Gastroenterol Hepatol Bed Bench 2013;6(1):14–17 Online First.
    OpenUrlPubMed
  36. 36.↵
    Rothman KJ, Greenland S, Lash TL. Modern epidemiology: Wolters Kluwer Health/Lippincott Williams & Wilkins Philadelphia, 2008
  37. 37.↵
    Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Value in Health 2017;20(8):1003–08 Online First.
    OpenUrl
  38. 38.↵
    Little RJ, Rubin DB. Statistical analysis with missing data: John Wiley & Sons, 2019. 0470526793;
  39. 39.↵
    von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. Annals of Internal Medicine 2007;147(8):573–77 doi: 10.7326/0003-4819-147-8-200710160-00010 published Online First.
    OpenUrlCrossRefPubMedWeb of Science
  40. 40.↵
    Benchimol EI, Smeeth L, Guttmann A, et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS medicine 2015;12(10):e1001885 Online First.
    OpenUrl
  41. 41.↵
    PyMARE: Python Meta-Analysis & Regression Engine [program], 2020.
  42. 42.↵
    Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Medicine 2009;6(7):e1000097 doi: 10.1371/journal.pmed.1000097 published Online First.
    OpenUrlCrossRefPubMed
  43. 43.↵
    Collaboration TCR-wEfViCC. Developing a framework for incorporating real-world evidence into drug funding decisions: CanREValue Collaboration Methods Working Group Progress Report on Real World Survival Data: Canada, 2020.
  44. 44.↵
    Bayley KB, Belnap T, Savitz L, Masica AL, Shah N, Fleming NS. Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied. Med Care 2013;51(8 Suppl 3):S80–6 doi: 10.1097/MLR.0b013e31829b1d48 published Online First: 2013/06/19].
    OpenUrlCrossRefPubMed
  45. 45.↵
    Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol 2017;106(1):1–9 doi: 10.1007/s00392-016-1025-6 published Online First: 2016/08/26].
    OpenUrlCrossRefPubMed
  46. 46.↵
    Beaulieu-Jones BK, Lavage DR, Snyder JW, Moore JH, Pendergrass SA, Bauer CR. Characterizing and Managing Missing Structured Data in Electronic Health Records: Data Analysis. JMIR Medical Informatics 2018;6(1):e11 doi: 10.2196/medinform.8960 published Online First.
    OpenUrlCrossRef
  47. 47.↵
    Olson MS. Developing an integrated strategy for evidence generation. J Comp Eff Res 2018;7(1):5–9 doi: 10.2217/cer-2017-0073 published Online First: 2017/10/21].
    OpenUrlCrossRef
  48. 48.↵
    Yadav S, Kazanji N, KC N, et al. Comparison of accuracy of physical examination findings in initial progress notes between paper charts and a newly implemented electronic health record. Journal of the American Medical Informatics Association 2017;24(1):140–44 Online First.
    OpenUrlCrossRefPubMed
  49. 49.↵
    Pfaff ER, Girvin AT, Gabriel DL, et al. Synergies between centralized and federated approaches to data quality: a report from the national COVID cohort collaborative. Journal of the American Medical Informatics Association 2022;29(4):609–18 Online First.
    OpenUrlPubMed
  50. 50.↵
    Delgado-Rodríguez M, Llorca J. Bias. J Epidemiol Community Health 2004;58(8):635–41 doi: 10.1136/jech.2003.008466 published Online First: 2004/07/15].
    OpenUrlAbstract/FREE Full Text
  51. 51.↵
    Sackett DL. Bias in analytic research. J Chronic Dis 1979;32(1-2):51–63 doi: 10.1016/0021-9681(79)90012-2 published Online First: 1979/01/01].
    OpenUrlCrossRefPubMedWeb of Science
  52. 52.↵
    Naemi A, Schmidt T, Mansourvar M, Ebrahimi A, Wiil UK. Quantifying the impact of addressing data challenges in prediction of length of stay. BMC Med Inform Decis Mak 2021;21(1):298 doi: 10.1186/s12911-021-01660-1 published Online First: 2021/11/10].
    OpenUrlCrossRef
  53. 53.
    Schuemie MJ, Cepeda MS, Suchard MA, et al. How Confident Are We about Observational Findings in Healthcare: A Benchmark Study. Harv Data Sci Rev 2020;2(1) doi: 10.1162/99608f92.147cc28e published Online First: 2020/12/29].
    OpenUrlCrossRef
  54. 54.↵
    Chan PY, Zhao Y, Lim S, Perlman SE, McVeigh KH. Using Calibration to Reduce Measurement Error in Prevalence Estimates Based on Electronic Health Records. Prev Chronic Dis 2018;15:E155 doi: 10.5888/pcd15.180371 published Online First: 2018/12/24].
    OpenUrlCrossRef
  55. 55.↵
    on Value R, Care S-DH. Observational Studies in a Learning Health System: Workshop Summary. 2013 Online First.
  56. 56.↵
    Forrest CB, McTigue KM, Hernandez AF, et al. PCORnet® 2020: current state, accomplishments, and future directions. Journal of Clinical Epidemiology 2021;129:60–67 doi: 10.1016/j.jclinepi.2020.09.036 published Online First.
    OpenUrlCrossRefPubMed
  57. 57.↵
    OHDSI. The Book of OHDSI: Observational Health Data Sciences and Informatics: OHDSI, 2019. 9781088855195;
  58. 58.↵
    Haendel MA, Chute CG, Bennett TD, et al. The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment. Journal of the American Medical Informatics Association 2021;28(3):427–43 doi: 10.1093/jamia/ocaa196 published Online First.
    OpenUrlCrossRefPubMed
  59. 59.↵
    Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med 2011;104(12):510–20 doi: 10.1258/jrsm.2011.110180 published Online First: 2011/12/20].
    OpenUrlCrossRefPubMed
  60. 60.↵
    Madigan D, Stang PE, Berlin JA, et al. A Systematic Statistical Approach to Evaluating Evidence from Observational Studies. Annual Review of Statistics and Its Application 2014;1(1):11–39 doi: 10.1146/annurev-statistics-022513-115645 published Online First.
    OpenUrlCrossRef
  61. 61.
    Hersh WR, Weiner MG, Embi PJ, et al. Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research. Medical Care 2013;51(Supplement 8Supp):S30–S37 doi: 10.1097/mlr.0b013e31829b1dbd published Online First.
    OpenUrlCrossRefPubMedWeb of Science
  62. 62.↵
    Maclure M, Schneeweiss S. Causation of Bias: The Episcope. Epidemiology 2001;12(1):114–22 Online First.
    OpenUrlCrossRefPubMedWeb of Science
  63. 63.↵
    Ryan PDJ. The Book of OHDSI. The book of OHDSI-Observational Health Data Sciences and Informatics, 2020
  64. 64.↵
    Wang SV, Pinheiro S, Hua W, et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ 2021;372:m4856 doi: 10.1136/bmj.m4856 published Online First: 2021/01/14].
    OpenUrlFREE Full Text
  65. 65.↵
  66. 66.↵
    ISPE. RWE for Regulatory Decision Making. Sec 2021. https://www.pharmacoepi.org/strategic-initiatives/rwe-for-regulatory-decision-making/. 10/18/2021
  67. 67.↵
    Alexander PE, Debono VB, Mammen MJ, et al. COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine. Journal of Clinical Epidemiology 2020;123:120–26 doi: 10.1016/j.jclinepi.2020.04.016 published Online First.
    OpenUrlCrossRefPubMed
  68. 68.
    Celi LA, Charpignon M-L, Ebner DK, et al. Gender Balance and Readability of COVID-19 Scientific Publishing: A Quantitative Analysis of 90,000 Preprint Manuscripts: Cold Spring Harbor Laboratory, 2021.
  69. 69.↵
    Dobler CC. Poor quality research and clinical practice during COVID-19. Breathe 2020;16(2):200112 doi: 10.1183/20734735.0112-2020 published Online First.
    OpenUrlAbstract/FREE Full Text
  70. 70.↵
    Chan W-K, Saravanan A, Manikam J, Goh K-L, Mahadeva S. Appointment waiting times and education level influence the quality of bowel preparation in adult patients undergoing colonoscopy. BMC gastroenterology 2011;11:1–9 Online First.
    OpenUrlCrossRefPubMed
  71. 71.↵
    Babushkina EA, Belokopytova LV, Grachev AM, Meko DM, Vaganov EA. Variation of the hydrological regime of Bele-Shira closed basin in Southern Siberia and its reflection in the radial growth of Larix sibirica. Regional Environmental Change 2017;17:1725–37 Online First.
    OpenUrl
Back to top
Posted June 22, 2023.
Download PDF
Data/Code
Email
Use of Recommended Real-World Methods for Electronic Health Record Data Analysis Has Not Improved Over 10 Years
Chenyu Li, Abdulrahman M. Alsheikh, Karen A. Robinson, Harold P. Lehmann
medRxiv 2023.06.21.23291706; doi: https://doi.org/10.1101/2023.06.21.23291706
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools

Subject Area