Real-world data: A systematic literature review on the barriers, challenges, and opportunities associated with their inclusion in the health technology assessment process

2.1 Study design, inclusion and exclusion criteria

A PRISMA-based systematic review [10, 11] was conducted to identify articles assessed by the researchers, employing inclusion criteria to ascertain study eligibility aligned with the review’s objectives. The search strategy, as detailed in section 2.2, was utilized to encompass these criteria. Inclusion criteria were as follows:

• ⮚ Population: There were no restrictions on populations, and studies were included from populations and sub-populations from all countries around the world and without any unique chracteristics.

• ⮚ Intervention: RWD and evidence that arise from the use and analysis of RWD.

• ⮚ Comparator: No comparator.

• ⮚ Outcomes: Data regarding the current use of RWD in HTA, the barriers, challenges, weaknesses of their integration in the process, opportunities, as well as the perspectives of stakeholders regarding the use of RWD in the HTA process were included. Additionally, data related to the views of stakeholders regarding RWD in the HTA process were also included.

• ⮚ Types of studies: All types of studies were included, such as reviews, policy texts, primary research, RCTs, qualitative research studies.

• ⮚ Language: Only studies written in English were included.

• ⮚ Timeline: No time restrictions were specified for the publication of studies and policy reports.

The exclusion criteria for studies in this analysis were as follows:

• ⮚ Study Types: Abstracts (oral and posters) that did not include at least one of the above outcome criteria were excluded.

• ⮚ Language: Any other written language apart from English was not included in this review.

2.2 Search strategy

The detailed search strategy, which was performed on July 2022, is provided in S1 Appendix.

Search strategy was implemented to the following databases: PubMed/Medline, Scopus, IDEAS-RePEc, International HTA database, Centre for Reviews and Dissemination. In addition, supplementary ad-hoc searches for relevant information were performed on Google Scholar, as well as various international organizations such as the World Health Organization (WHO) and Organisation for Economic Co-operation and Development (OECD), and specific health technology assessment organizations such as National Institute for Health and Care Excellence (NICE), Haute Autorité de santé (HAS), and Institute for Clinical & Economic Review (ICER) to identify relevant texts and references related to the study objectives.

2.3 Study selection methods

The literature discovered through the search was archived in a bibliographic database (EndNote), with duplicate entries subsequently removed. A pilot training check process was conducted initially to ensure consistency in selection and identify areas for modifications in the inclusion criteria to provide a more comprehensive and explicit list of study types that would be considered eligible for this review. Two researchers independently checked a random sample of approximately fifty (50) titles and abstracts for eligibility, and a high level of agreement was achieved which indicates that the two researchers largely agreed on whether each of these documents met the inclusion criteria established for the study. After this, a single researcher checked the remaining titles and abstracts for eligibility. Later, the studies resulting from the removal of duplicate entries were uploaded into Abstrackr [12], a specialized software developed by Brown University and the Center for Evidence Synthesis in Health. All abstracts were examined, and full-text documents were retrieved for the files that were flagged for inclusion. The retrieved articles were then analyzed in detail based on the full text.

2.4 Data extraction and synthesis methods

The study data was meticulously extracted and organized into four tables, a process undertaken to streamline and enhance the subsequent analysis and synthesis of the information. The design of these tables was thoughtfully structured to systematically capture pertinent information derived from the selected studies. The first table contained details relevant to the characteristicts including author, year of study, country, study type, objectives, health technology studied, population and therapeutic category, and subcategory of real-world data. The second table was dedicated to encompassing data concerning the contemporary utilization and reception of RWD and RWE. In contrast, the third table comprehensively addressed the hurdles, challenges, and complexities encountered when integrating RWD-RWE into HTA. Meanwhile, the fourth table was designed to encompass the potential advantages, opportunities, and viability associated with the adoption of RWD-RWE within the realm of HTA. To ensure consistency and pinpoint any potential adjustments required for the data extraction model, two researchers initially conducted an independent pilot test on a random sample of ten (10) studies. During this process, an appropriate level of agreement was observed, denoting that there was a satisfactory degree of consensus or concurrence among the researchers involved in the extraction of data from the selected studies. The extraction of the remaining studies was conducted by a primary researcher, supported by a secondary researcher who remained readily available to offer assistance in clarifying information or in situations where the primary researcher encountered challenges or uncertainties during the extraction process.

2.5 Appraisal of methodological quality

The methodological quality of the studies included in this review was assessed using several critical appraisal tools, namely the Critical Appraisal Skills Programme (CASP) tool for qualitative research [13], the Joanna Briggs Institute (JBI) checklist for systematic reviews and evidence syntheses [14], and Joanna Briggs Institute (JBI) checklist for text and opinions [15]. Each tool evaluated different aspects of study quality by one reviewer, including the study design, data collection methods, data analysis, and reporting of results. For the assessment of each study done using the CASP tool, the reviewer assessed the quality of the study design, data collection methods, data analysis, and interpretation of findings. The Critical Appraisal Skills Programme (CASP) tool is the most used tool for quality appraisal in health-related qualitative evidence syntheses [16]. Meanwhile, the JBI checklist was used to evaluate the relevance of the studies to the review question, study design, sample size, data collection methods, data analysis, and reporting of results.

Quality appraisal, in detail, of eligible studies can be found in the S2 Appendix. In the overarching context, it is pertinent to elucidate that the quality of the incorporated studies exhibits a discernible spectrum, wherein, a number of studies may be aptly delineated as demonstrating a standard of moderate quality, while the preponderance of the corpus can be distinguished as manifesting a commendable standard of good quality.

3.1 Description of study characteristics

Overall, 29 studies were included in this review, among which, most of them were refering to European countries. Most of these 29 studies were referring to multiple countries within their analyses while few of them assessed information related to RWD/RWE for HTA in continents. In particular, England (n=6), Germany (n=5), UK (n=4), Sweden (n=4), Netherlands (n=3), Scotland (n=3), France (n=3), Norway (n=2), Italy (n=2), Spain (n=1), Austria (n=1), Denmark (n=1), and Belgium (n=1). In addition, Europe was referred in three studies, while one study included European Union countries and another one study included Central and Eastern Europe. On the other hand, several included studies referred to North America, and particularly United States (n=2) and Canada (n=2) and one study referred to South America countries and particularly to Argentina, Brazil, Colombia, and Chile. The review also included one study referring to Asian countries (Bhutan, China, India, Indonesia, Japan, Malaysia, Philippines, Singapore, South Korea, Taiwan, Thailand). Last, but not least, Middle East and North Africa (n=1) as well as Saudi Arabia (n=1) were part of the final studies while international scope was referred in five studies.

3.2 Current use and acceptance of real-world data/real-world evidence in HTA

Out of the twenty-nine studies (n=29) reviewed, twenty-four studies (n=24) highlighted the current use or acceptance of RWD and/or RWE in the process of HTA. Hagen et al. (2021) [17] outlined the Norwegian HTA process and highlighted the use of RWD from observational studies when RCTs are lacking. Guidelines in Norway govern the collection and utilization of RWD, including data from Norwegian patient registries. Fasseeh et al. (2020) [18] found that the use of local data in HTA processes is not obligatory in the Middle East and North Africa region, suggesting that reliance on local data may not be a prerequisite in the region’s HTA processes. The utilization and acceptance of evidence from primary care databases in HTA conducted by the NICE was investigated by Leahy et al. (2020) [19]. These databases were used to depict disease epidemiology, describe treatments, and provide clinical inputs for cost-effectiveness models. Fuchs et al. (2016) [20] conducted a systematic review on the utilization of real-world data/evidence (RWD/RWE) in HTA for medical devices in European HTA organizations. The review found that while RCTs are preferred, alternative study designs such as observational studies are acknowledged in certain circumstances. On the other hand, Patel et al. (2021) [21] found that the use of RWD/RWE in single-arm based HTA submissions was relatively low, accounting for only 5% of total submissions. Oncological and hematological-oncological diseases were the most common areas of study. The use of external comparators derived from real-world data increased over time, with varying acceptance rates among HTA organizations. Furthermore, the clinical evidence used in the evaluation of new pharmaceutical technologies in the UK’s HTA system was analyzed by Tolley (2010) [22]. In particular, the study highlighted challenges related to evidence requirements, cost-effectiveness assessments, and stakeholder involvement. NICE emphasized the importance of including pragmatic trials for real-world documentation. Kent et al. (2021) [23] provided recommendations on the use of evidence from non-randomized studies on treatment effects in the HTA process. Most HTA bodies prefer RCT data due to lower bias risk, but there are variations in the acceptance of non-randomized trials among organizations. Furtheremore, the consideration and acceptance of external control arms (ECAs) using real-world data in regulatory and HTA evaluations for specific drugs in oncology was reviewed by Jaksa et al. (2022) [24]. Varying levels of acceptance were observed among different HTA organizations, including NICE, G-BA, HAS, pCODR, and PBAC. Additional included studies showed the significance of RWE in decision-making processes, variations in its utilization across different countries, and the prevailing preference for RCTs as a primary source of evidence in HTA. Justo et al.’s (2019) [25] systematic review and qualitative research across Argentina, Brazil, Colombia, and Chile explored RWD sources, attributes, and applications for health technologies. Findings indicate RWD’s growing role in Health Technology Assessments (HTA) and reimbursement, driven by anticipated increases and pharmaceutical industry investment. Colombia emphasizes RWD in economic evaluations, while Chile anticipates wider use for HTA and cost data. These insights inform future legislation and RWD management strategies. Timbie et al.’s (2021) [26] qualitative thematic analysis, conducted internationally, explores the contemporary application of RWE in the realm of medical devices. The research advances comprehension of RWE’s role in regulatory and reimbursement decision-making, despite prevailing payer preferences for RCTs and concerns about potential bias in RWE. Notably, RWE has yet to assume the principal role in pre-market approval processes. Bullement et al. (2020) [27] conducted a systematic review showing extensive incorporation of RWE in the cost-effectiveness analysis of cancer drugs evaluated by NICE in England, emphasizing its significant role in decision-making. Al-Omar et al. (2021) [28] identified differing stakeholder perspectives in Saudi Arabia regarding the mandatory integration of RWE and RCT data in HTA, indicating the need for further discussions in developing HTA processes for pharmaceuticals. The utilization of RWD in HTA was also observed by Makady et al. (2018) [29] and partocularly in melanoma medicine assessments across five European countries, noting variations in the types and extent of RWD incorporation, with an increasing trend over time. The importance of RWE in reimbursement decision-making process of specific technologies related to genomics was investigated by Deverka et al. (2020) [30]. They found limited utilization of RWE in coverage decision-making for next-generation sequencing (NGS) tests by US payers, with a primary reliance on clinical guidelines and RCTs. Lou et al. (2020) [31] reported positive acceptance of RWD and RWE for clinical efficacy assessment by HTA organizations in multiple Asian countries, but with varying opinions on the requirement to justify the rationale for using RWD. The frequent use of RWE in decision analysis and modeling for HTA was highlighted by Bowrin et al. (2019) [32], with RWE influencing various aspects of decision-making, drug access, and licensing. In addition, Brogaard et al. (2021) [33] observed the inclusion and utilization of RWD in HTA assessments and reimbursement decisions for entrectinib and larotrectinib, demonstrating its varying impact and benefits in different countries. Hogervorst et al.’s (2022) [34] primary qualitative research in European countries examined the acceptance challenges surrounding RWD for complex health technologies. Survey results indicated that patient registry data received high acceptance (86%), while unpublished data and expert opinions had lower acceptance rates (approximately 25% to 28%). Sievers et al. (2021) [35] conducted qualitative research on stakeholder perceptions of post-marketing RWE and RWD collection requirements in several European countries. The study highlighted a prevailing preference for RCT data and limited utilization of RWE in the evaluation of health technologies, particularly in the German context.

On the other hand, Hampson et al. (2018) [36] conducted research in the United States, finding that initial assessments of health technologies relied on data from RCTs by organizations such as the United States ICER and payers. The use of RWE for comparative clinical effectiveness posed challenges, but RWE was considered more relevant for re-evaluations by payers. Difficulties in collecting similar data hindered the widespread adoption of RWE. George (2016) [37] emphasized the common use of non-randomized clinical trial data in health technology assessments by the NICE in the United Kingdom. Non-randomized clinical trial data was valuable for assessing medical devices due to limitations in RCTs. NICE recognized the importance of including non-randomized clinical trial data to comprehensively evaluate health technologies. Makady et al. (2017) [38] examined the policies of six HTA organizations in Europe and found that they accepted various types of clinical evidence, including RWD, for initial reimbursement discussions. RWD was particularly requested for pharmacoeconomic assessments, and local RWD sources such as databases and registries were recommended. The use of RWD had a positive impact on the decision-making process. Pongiglione et al. (2021) [39] investigated RWD sources for HTA documentation on medical devices in Europe, focusing on procedures such as hip and knee arthroplasty, percutaneous aortic valve implantation (TAVI), mitral valve repair (TMVR), and da Vinci robotic surgery. Four categories of RWD sources were identified: administrative data, registry data, other observational studies, and other RWD sources with health outcomes data. Data availability varied across cases, and some sources lacked comparable data for HTA. Ciminata (2019) [40] explored the use of RWD to support HTA in Scotland, specifically for anticoagulant drugs in atrial fibrillation. RWD studies consistently showed that direct oral anticoagulants (DOACs) were at least as effective as warfarin in preventing stroke and systemic embolism, with a lower risk of major bleeding.

3.3 Barriers, challenges, and difficulties encountered in incorporating RWD/RWE within HTA

The scoping review by Hagen et al. (2021) [17] emphasized the growing recognition of RWD from observational studies in Norwegian HTA. Real-world data contributes significantly to verifying clinical efficacy across different therapeutic categories within the general population. The editorial by Gonçalves (2020) [41] highlighted the importance of integrating ethical assessment into the HTA process for advanced therapy medicinal products (ATMPs). The inclusion of real-world data in HTA processes for ATMPs is crucial to provide a more comprehensive understanding of their real-world effectiveness but there is absence of satisfactory data from both clinical studies or real-world sources. The study by Fasseeh et al. (2020) [18] focused on HTA implementation in the Middle East and North Africa (MENA) region. The study revealed limitations in the availability and transferability of local real-world data, emphasizing the challenges faced in accessing and utilizing comprehensive local data sources for HTA processes in the region. The systematic review by Leahy et al. (2020) [19] investigated the use of documentation derived from primary care databases in health technology assessments conducted by NICE in the UK. Tailored analyses using data from primary care databases were better received by evaluation committees, highlighting the importance of generating specific analyses for NICE submissions. Another systematic review conducted by Fuchs et al. (2016) [20] focused on HTA activities for medical devices among European HTA organizations. The review highlighted the differences in the interpretation of MDs and variations in the willingness to embrace alternative study designs. This from another perspective could pose challenges in the pursuit of standardizing the integration of RWD across institutions. The qualitative study by Dai et al. (2021) [42] evaluated a proposed re-evaluation process for cancer drugs in Canada. The study emphasized the importance of reliable RWE, challenges in establishing an incremental cost-effectiveness ratio (ICER) for reassessments, and the importance of rigorous evaluation criteria in the re-evaluation process for cancer drugs. The retrospective review by Patel et al. (2021) [21] assessed the use of external comparators (ECs) based on single-arm test data in HTA submissions. Variability in the acceptance of RWD and RWE among HTA stakeholders was found, with challenges identified in study design, analysis, publication bias, and limited information. Tolley’s (2010) [22] commentary examined challenges within the UK’s HTA system, including clinical evidence requirements, cost-effectiveness evaluations, and stakeholder involvement. The commentary emphasized the specific requirements of UK HTA agencies for demonstrating clinical effectiveness and cost-effectiveness compared to placebo. Kent et al. (2021) [23] provided recommendations on the use of evidence from non-randomized trials in HTA. The study identified challenges in study planning, design, analysis, reporting, and systematic protocols in using non-randomized clinical evidence in HTA. Jaksa et al. (2022) [24] reviewed regulatory and HTA agencies’ critiques of external control arms (ECAs) with real-world data. The study emphasized the need for future recommendations in ECA design and production based on different critiques raised by HTA agencies. Justo et al. (2019) [25] explored RWD sources, characteristics, and uses in South America. The study identified problems with RWD quality, consistency, collection methods, lack of expertise, and confidence in observational research. Kamusheva et al. (2022) [43] conducted a scoping review and qualitative research in Central and Eastern European countries to identify barriers to implementing RWE in health technology assessment. The study found technical, regulatory, clinical, and scientific barriers, as well as other barriers, that hinder the adoption and integration of RWE into healthcare decision-making processes in the region. Timbie et al. (2021) [26] conducted a survey and interviews to examine the challenges faced by manufacturers in utilizing RWE for regulatory and reimbursement decisions in the medical device industry. The study identified barriers such as uncertain return on investment, difficulty in accessing high-quality data, and lack of RWE expertise, which hinder the widespread use of RWE. In addition, the systematic review conducted by Bullement et al. (2020) [27] on the use of RWE in single technology assessments (STAs) of cancer medicines by the NICE in England, found key criticisms regarding the sources of RWE, such as the comparability of real patient cohorts with clinical trial patients and the relevance of RWE patient cohorts to current clinical practice. Al-Omar et al. (2021) [28] conducted a scoping review and primary qualitative research in Saudi Arabia to explore the perspectives of local experts on potential evidence relevant to HTA processes and methods for pharmaceutical products. The study identified barriers such as limited availability and quality of real-world data, concerns about data privacy and confidentiality, and lack of standardization in data collection and reporting practices. What’s more, Makady et al. (2018) [29] conducted a retrospective study on the use of RWD in the HTA process for melanoma drugs in Europe. The study found that RWD inclusion was more common in CEAs compared to REAs, but there was a lack of consistent evaluation of RWD and variability across different HTA organizations. Deverka et al. (2020) [30] conducted a systematic review identifying barriers and issues related to the use of RWE in payer decision-making. The study found concerns about the relevance and timeliness of RWE studies, perceived inadequacy of RWE methods, limited influence of genomic studies on coverage decisions, and lack of expertise among payers and clinicians in using RWE. Lou et al. (2020) [31] conducted a qualitative study in multiple Asian countries on the use of RWD and RWE in all health technologies. The study identified challenges such as lack of representative patient samples, incomplete patient information, inadequate infrastructure and skills for RWD analysis, and lack of support for data sharing. Facey et al. (2020) [44] conducted a qualitative study in European Union countries to explore the use of RWD in decision-making processes for highly innovative technologies. The study identified challenges such as fragmentation, lack of clarity and harmonization, isolated approaches, and a lack of collaboration among payers and HTA organizations. An additional systematic review by Bowrin et al. (2019) [32] with the aim to explore the limitations of using RWE in decision analysis and modeling, identified limitations such as confounding bias, lack of accuracy in drug exposure and outcome data, challenges in data protection and confidentiality, and insufficient patient numbers. Brogaard et al.’s (2021) [33] analysis encompassed England, Germany, France, Canada, Denmark, Sweden, and Scotland, focusing on HTA agency reviews and reimbursement decisions for entrectinib and larotrectinib. The study aimed to compare assessments and reimbursement outcomes for these medications across countries, shedding light on similarities and differences in their evaluation and acceptance. Key findings included variation in evidence criteria, payer concerns about clinical data, especially non-traditional datasets like RWD, and limited integration of RWD in HTA evaluations, suggesting the need for guidance on RWD acceptance and indirect comparison methods. Hogervorst et al. (2022) [34] conducted a qualitative research study in European countries to assess challenges related to the acceptance of RWD in the context of complex health technologies. The study identified obstacles such as a lack of necessary data sources, inadequate policy structures, difficulties in interpretation and verification, absence of relevant variables, and time constraints in accessing data. Sievers, et al. (2021) [35] conducted a qualitative research study in multiple European countries to explore stakeholder perceptions of post-marketing RWE. The study identified challenges including methodological issues, low data quality, limited infrastructure for RWD collection, and ensuring data representativeness. Stakeholders emphasized the value of RWE but stressed the need to address these challenges for optimal use in decision-making processes. Hampson et al. (2018) [36] investigated RWE utilization by conducting a literature review and interviews with RWE experts revealed challenges: bias, incomplete data, accessibility issues, methodological disparities, and a shortage of qualified researchers. Enhancing RWE’s role in reimbursement decisions requires addressing these concerns and fostering consensus on methodologies. George E (2016) [37] provided a UK perspective on the use of non-RCT data within the NICE. The commentary highlighted challenges associated with non-RCT data, such as confusion, biases, incomplete data, limited use of quality-of-life data, and uncertainty regarding data accuracy. The discussion shed light on considerations for utilizing real-world data in evaluating health technologies across various therapeutic categories. Husereau et al. (2019) [45] conducted a qualitative research study in Canada on the use of RWD for drug pricing and reimbursement decisions. The study identified barriers to increasing RWD utilization, including lack of trust, difficulty in accessing data, lack of standardization, insufficient infrastructure, perceived threats to stakeholders’ interests, lack of cooperation, and challenges in translating data into action. Pongiglione et al.’s (2021) [39] systematic review encompassed 15 European countries, examining the availability and quality of real-world data pertinent to Life Cycle Information Systems (LIS) for medical devices. The study critically evaluated real-world data sources, emphasizing their relevance to medical device-associated ATHENA (Assessment of Therapeutic Effectiveness by National Authorities). Focused on hip and knee arthroplasty, TAVI, TMVR, and da Vinci robotic surgery, the research identified barriers, including data accessibility challenges, lack of outcome standardization, limited comparable data, data quality concerns, and demographic and epidemiological data gaps. The latest study by Ciminata (2019) [40] focusing on the use of RWE for assessing anticoagulant medications in patients with atrial fibrillation, explored the opportunities and challenges of using RWD in the decision-making process. It highlighted challenges such as limited availability of high-quality data, data quality issues, methodological challenges, and data access barriers. The thesis stressed the need for standardization in RWD collection, management, and analysis to improve the quality and comparability of results.

3.4 Potential Benefits, Opportunities, and Feasibility of Utilizing RWD/RWE in the HTA Process

The first study conducted by Hagen et al. (2021) [17] focused on the Norwegian HTA system and highlighted the importance of patient registries as valuable sources of evidence. The study emphasized that well-organized registries have the potential to provide crucial data on health utilities, resource use, and costs, enhancing the assessment of health technologies and informing decision-making processes. Patient registries were found to offer insights into treatment patterns, real-world treatment adherence, and the natural history of diseases, contributing to accurate economic evaluations and modeling. In the editorial by Gonçalves (2020) [41], the integration of ethical evaluation into the HTA process for advanced therapy medicinal products (ATMPs) is discussed, with a specific focus on the use of RWD. The editorial emphasizes the importance of collecting and analyzing evidence from RWD to complement the assessment of ATMPs, particularly in the context of rare diseases. It highlights the need for long-term efficacy and comparative effectiveness data, supplemented by real-world evidence, to ensure comprehensive evaluations of these advanced therapies. Fasseeh et al. (2020) [18] conducted a Middle East and North Africa Primary Survey to assess the implementation of HTA in the region. The study revealed a willingness among respondents to invest in patient registries and make payer databases accessible to the HTA sector. It underscored the importance of collecting and utilizing local data to enhance the quality of HTA practices. Furthermore, Leahy et al. (2020) [19] conducted a UK Systematic Review that explored the utilization of evidence from primary care databases in NICE HTA assessments. The study identified primary care databases as a key source of RWD for informing NICE submissions and aiding healthcare decision-making. The acceptance and positive feedback from review committees and the Evidence Review Group (ERG) indicate the reliability and accuracy of primary care databases as a data source. Another systematic review by Fuchs et al. (2016) [20] with the aim to review and compare the activities of HTA for medical devices among European organizations involved in Mutual Recognition Agreements (MRAs), highlighted various initiatives to address the challenges in medical device assessment, such as standardized data elements, outcome measures, and data exchange platforms. These initiatives aim to enhance the consistency and quality of MD assessment processes across European HTA organizations. Dai et al. (2021) [42] evaluated the proposed re-evaluation process for cancer medicines in Canada, focusing on the use of RWD and indicated that stakeholders expressed confidence in the results, citing the availability of RWE on clinical benefit and cost-effectiveness from various Canadian provinces. The study recommended involving patients in the reassessment process and leveraging RWE for studying rare cancers and targeted therapies. Patel et al. (2021) [21] conducted an international retrospective study reviewing HTA submissions from various countries. The study highlighted the need for a well-designed strategy and robust data collection methods for indirect treatment comparison. It emphasized the increasing use of RWD in HTA submissions and the importance of monitoring and developing guidelines for RWD submissions. Tolley (2010) [22] offered a commentary on the evaluation of new pharmaceutical technologies within the UK’s HTA system. The commentary highlighted the importance of observational studies, including registers, for assessing the real-life efficacy and cost-effectiveness of medicines during the post-marketing phase. It emphasized the need for conditional drug reimbursement schemes and the adaptation of research and development strategies to meet the demand for evidence from unproven clinical trials. On the other hand, the author discusses challenges in evaluating new pharmaceutical technologies within the UK’s HTA system and highlights the importance of real-world data during the post-marketing phase. Kent et al. (2021) [23] provides recommendations for using evidence from non-randomized trials in HTA processes, emphasizing study design, reporting potential errors, and strengthening scientific advice processes. Jaksa et al. (2022) [24] review critiques of ECAs utilizing real-world data, highlighting selection bias and confusion as key issues. The expansion of healthcare coverage and increasing number of diseases and interventions covered put pressure on healthcare budgets as mentioned in the systematic review and qualitative research through workshops with stakeholders conducted by Justo et al. (2019) [25]. RWE can help monitor outcomes and cost-effectiveness, but careful data recording and evidence-based policy planning are crucial. Kamusheva et al. (2022) [43] conducted a scoping review and qualitative research by having internal discussions and a webinar with stakeholders from Central and Eastern European countries to identify barriers to the use of RWE in healthcare. The study focused on Central and Eastern European countries. Those countries face challenges in collecting RWE due to limited IT infrastructure and frequently changing legislation. Collaborative initiatives and infrastructure development are needed. Timbie et al. (2021) [26] explored the use of RWE for medical devices, recommending clearer guidance, rigorous study design, engagement in RWE-related activities, and alternative reimbursement strategies. Bullement et al. (2020) [27] conducted a systematic review with the aim of how RWE has been used to inform Single Technology Assessments (STAs) of cancer drugs conducted by the NICE. Developing best practice guidelines and addressing common criticisms can enhance the use of RWE in submissions to NICE. Al-Omar et al. (2021) [28] identify challenges in utilizing RWD in HTA processes in Saudi Arabia, including data availability, acceptance by stakeholders, and expertise and resources. In Makady’s, et al. (2018) [29] literature review focusing on England, Scotland, Netherlands, France, and Germany, it was examined whether RWDs are included in the REAs and CEAs of melanoma drugs, and the evaluation of RWDs for their intended purposes by five HTA organisations in Europe. Alignment with policies, quantitative methods, and the role of RWD in re-evaluations are discussed. RWDs could play a critical role in drug reassessment by confirming previous efficacy estimates, cost-effectiveness ratios (ICERs) and budget implications. Comprehensive re-evaluation reports under conditional reimbursement systems (CRS) could benefit from the use of RWDs. Deverka et al. (2020) [30] examined the use of RWE in coverage decision-making for NGS-based tests. Opportunities for training and development incentives in RWE and NGS were identified, including providing training on observational study methods and pragmatic trials for stakeholders, particularly payers. Additionally, encouraging funding for RWE studies that demonstrate the benefits and harms of NGS testing was recommended. Improving the methods of studies that generate RWE was also highlighted. Several groups have developed RWE assessment tools specifically for payers, and it was suggested that multi-stakeholder groups should adapt these best practices and evaluation tools to NGS. Furthermore, transparent engagement processes should be established to incorporate payer information needs. Lastly, the study emphasized the potential of artificial intelligence (AI)-based methods, such as natural language processing and machine learning, in processing and analyzing unstructured data from electronic health records and patient-generated data, reducing the need for manual curation.

The first study conducted by Hagen et al. (2021) [17] focused on the Norwegian HTA system and Lou et al. (2020) [31] conducted a study across multiple Asian countries, examining the use of real-world data and evidence in the assessment of health technologies in the general population. The study identified challenges and proposed the development of a guidance document specifically tailored for Asia to address these challenges. Collaboration and the adoption of good practices among stakeholders were emphasized as crucial strategies to overcome these challenges and effectively utilize real-world data and evidence in health technology assessment processes across Asia. Facey et al. (2020) [44] investigated the use of RWD in decision-making by payers and HTA organizations in European Union countries. The study employed a mixed-methods approach and identified several opportunities and applications for leveraging RWD, including collaboration with academia, engaging industry in multilateral dialogues, promoting governance and transparency in RWD production, establishing an EU multilateral learning network for RWD, and seeking regulatory support and guidance. These actions have the potential to improve the use of RWD in decision-making processes, enhance the reliability and relevance of RWD, and foster knowledge exchange in the health sector across European Union countries. Bowrin et al. (2019) [32] conducted a systematic review to investigate the limitations of using RWE in decision analysis, particularly in modeling, and to identify existing recommendations on RWD-based modeling. The review highlighted the need for a practical guide on utilizing registry data for cost-effectiveness decisions and proposed a checklist to assess statistical issues when evaluating cost-effectiveness models with observational data. It also revealed opportunities for incorporating RWE in decision-making, as several agency guidelines recognized its usefulness in providing evidence on clinical practice, treatment pathways, resource utilization, disease natural history, and intervention efficacy and safety. Brogaard et al. (2021) [33] analyzed HTA agency assessments and reimbursement decisions for tumor-agnostic therapies in multiple countries. The study focused on the role of RWD in these assessments and highlighted the importance of monitoring future evaluations and potential re-evaluations to enhance the acceptability of RWD as a data source. Conditional reimbursement procedures and the need for clear guidance on indirect comparisons and acceptable use of RWD were identified as opportunities. The study also emphasized the lack of guidance on structuring HTA assessments and developing economic models for future tumor-agnostic therapies, calling for further guidance and integration of RWD into the assessment process. Hogervorst et al. (2022) [34] conducted qualitative research in European countries to assess the challenges associated with the acceptance of RWD in the evaluation of new health technologies. The study identified circumstances where RWD could be deemed acceptable, such as high disease burden, severe indications, outdated or inconsistent RCT findings, uncertainty regarding resource utilization, innovative technologies, limited treatment options, and clinical practice comparisons. The study also highlighted the importance of reliable documentation, clinical uncertainty, high uncertainty in cost-effectiveness analyses, and well-established use according to European legislation as factors that could warrant the use of RWD. Sievers et al. (2021) [35] conducted qualitative research in several European countries to explore stakeholder perceptions of post-marketing RWE and its value in healthcare decision-making. The study identified key challenges and opportunities related to RWE utilization, emphasizing its potential to complement RCT data, address gaps, and reduce uncertainty. Opportunities included harmonizing documentation requirements, fostering greater acceptance, and facilitating dialogue among stakeholders through platforms like the Joint Scientific Advice and EUnetHTA. The study highlighted the importance of coordinating post-marketing RWE requirements between regulatory agencies and HTA organizations. Hampson et al. (2018) [36] explored the use of RWE in the healthcare system in the USA. They identified opportunities for improvement, including the establishment of a national mandatory register for observational studies, national data repositories, and better quality electronic medical records. The study recommended rigorous protocols and consensus on good practice guidelines for generating high-quality RWE. Effective governance arrangements for data sharing and pragmatic trials that bridge the gap between RWE and TC were also suggested. The study proposed fast-track licensing processes to utilize RWE for decision-making on innovative therapies with limited documentation. George (2016) [37] focused on using RWD in health technology assessments by the NICE in the UK. The article emphasized the need for improvements in non-RCT data, including better case identification, data source linkage, and standardized recording of health-related quality of life and patient preferences. Transnational participation in projects such as IMI Get Real and ADAPT SMART was highlighted to develop frameworks and methodologies for integrating RWD into NICE assessments. The article suggested that using both RCTs and RWD can provide a more comprehensive understanding of healthcare interventions’ effectiveness and safety. Makady et al. (2017) [38] reviewed policies of six health technology assessment organizations in Europe regarding the use of RWD in assessments of medicines. The study found a lack of alignment and guidance on practical aspects of RWD collection and analysis, which may discourage investment in RWD production. The researchers recommended aligning policies and providing guidance on RWD collection and analysis to incentivize stakeholders. Harmonization of policies can support the generation of additional or alternative data for medicines where RCTs are not feasible, especially for rare diseases or orphan drugs. Husereau et al. (2019) [45] conducted qualitative research in Canada to explore stakeholders’ views on using RWD for drug pricing and reimbursement decisions. The study highlighted the need for investments in data infrastructure, standardization, analytical tools, and staff training to fully leverage the potential of RWD. Collaboration among researchers, regulators, payers, and industry stakeholders was emphasized to ensure the quality and reliability of RWD. Data standardization and harmonization, along with cooperation and partnerships, were identified as important for effective utilization of RWD in decision-making processes. Pongiglione et al. (2021) [39] conducted a systematic review on the use of RWD in HTA of medical devices in Europe. The study emphasized the need for standardization in the selection, measurement, use, and reporting of RWD to address biases and ensure consistency in clinical research and practice. Stronger coordination at the EU level and a coordinated approach involving stakeholders and industry were recommended to enhance the use of RWD in HTA processes and inform policy decisions. Ciminata (2019) [40] explored the opportunities and challenges of using RWE to support the Authorization for Temporary Use (ATF) decision-making process in Scotland, specifically focusing on anticoagulant medication in patients with atrial fibrillation. The study emphasized the need for standardization in RWD collection, management, and analysis to improve the quality and comparability of results.