Abstract
Background The integration of real-world evidence (RWE) from real-world data (RWD) in clinical research is crucial for bridging the gap between clinical trial results and real-world outcomes. Analyzing routinely collected data to generate clinical evidence faces methodological concerns like confounding and bias, similar to prospectively documented observational studies. This study focuses on additional limitations frequently reported in the literature, providing an overview of the challenges and biases inherent to analyzing routine clinical care data, including health claims data (hereafter: routine data).
Methods We conducted a literature search on routine data studies in four high-impact journals based on the Journal Citation Reports (JCR) category “Medicine, General & Internal” as of 2022 and three oncology journals, covering articles published from January 2018 to October 2023. Articles were screened and categorized into three scenarios based on their potential to provide meaningful RWE: (1) Burden of Disease, (2) Safety and Risk Group Analysis, and (3) Treatment Comparison. Limitations of this type of data cited in the discussion sections were extracted and classified according to different bias types: main bias categories in non-randomized studies (information bias, reporting bias, selection bias, confounding) and additional routine data-specific challenges (i.e., operationalization, coding, follow-up, missing data, validation, and data quality). These classifications were then ranked by relevance in a focus group meeting of methodological experts. The search was pre-specified and registered in PROSPERO (CRD42023477616).
Results In October 2023, 227 articles were identified, 69 were assessed for eligibility, and 39 were included in the review: 11 on the burden of disease, 17 on safety and risk group analysis, and 11 on treatment comparison. Besides typical biases in observational studies, we identified additional challenges specific to RWE frequently mentioned in the discussion sections. The focus group had varied opinions on the limitations of Safety and Risk Group Analysis and Treatment Comparison but agreed on the essential limitations for the Burden of Disease category.
Conclusion This review provides a comprehensive overview of potential limitations and biases in analyzing routine data reported in recent high-impact journals. We highlighted key challenges that significantly impact analysis results, emphasizing the need for thorough consideration and discussion for meaningful inferences.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was funded by the Federal Ministry of Education and Research (BMBF) in Germany in the framework of the EVA4MII project (FKZ 01ZZ2308A, 01ZZ2308B, 01ZZ2308C). The funding agency had no role in the design, data collection, analyses, interpretation, and reporting of the study. The work of HB, MB, and NB has also been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - Project-ID 499552394 - SFB 1597.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Small Changes in the authors affiliation, Supplementary Table labeling and more detailed author's contribution.
Data Availability
All data produced in the present work are contained in the manuscript
Abbreviations
- COPD
- Chronic Obstructive Pulmonary Disease
- EHR
- Electronic Health Records
- ICD
- International Classification of Diseases and Related Health Problems
- JCR
- Journal Citation Report
- OPS
- Operation and Procedure Classification System
- RCT
- Randomized Controlled Trial
- RWD
- Real-World Data
- RWE
- Real-World Evidence
