Abstract
Background Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of experts to use when formulating recommendations or making decisions. While extensively used for clinical and public health recommendations, EtD frameworks are not in widespread use in environmental health.
Objectives This review sought to identify, compare and contrast key EtD frameworks for decisions or recommendations on interventions used in clinical medicine, public health or environmental health. Our goal was to identify best practices and guidance which will be used to inform the development of an EtD framework for formulating recommendations regarding interventions to prevent or mitigate the harmful effects of exposure to substances in the environment.
Methods We identified a convenience sample of EtD frameworks used by a range of organizations. We searched Medline for systematic reviews of EtD frameworks used in clinical medicine, and public or environmental health. In a qualitative manner, we summarized the decision criteria in the selected frameworks and in the reviews.
Results Fourteen key organizations provided 18 EtD frameworks; most frameworks focused on clinical medicine or public health interventions; four focused on environmental health and three on economic considerations. Only one framework was based on an underlying conceptual model, and rarely was a systematic review of potential criteria performed during the frameworks development. GRADE encompasses a set of closely related frameworks for different types of decisions. Harms of interventions were examined in all frameworks and benefits in all but one. Other criteria included certainty of the body of evidence (15 frameworks), resource considerations (15), feasibility (13), equity (12), values (11), acceptability (11), and human rights (2). There was variation in how specific criteria were defined. The five identified systematic reviews reported a similar spectrum of EtD criteria.
Discussion The EtD frameworks examined encompassed similar criteria, with tailoring to specific audience needs. However, there is variation in development processes, terminology, level of detail provided and presentation of the criteria. Existing frameworks are a useful starting point for development of one tailored to decision-making in environmental health.
Introduction
It is widely recognized that environmental pollution is an important determinant of health and that interventions, in particular policy recommendations, are a key means by which population health can be improved. The formulation of trustworthy and impactful recommendations and policies on environmental health interventions is a complex task, which requires the identification of all relevant data and evidence, their critical appraisal and synthesis and translation into a recommendation or policy. The processes and methods for hazard identification and risk assessment of environmental substances based on systematic reviews of the evidence have advanced considerably in the last decade.1–4 These approaches have been largely based on methods developed for clinical medicine,5 which have been expanded to encompass public health interventions,6 diagnostic test accuracy and impact,7 coverage decisions,8, 9 and health technology assessments (HTA),10 among others.
The final step in the process of formulating evidence-informed recommendations and policies involves the translation of data and evidence on various decision factors into an explicit recommendation or policy. Evidence-to-decision (EtD) frameworks provide a structured and transparent approach for groups of technical experts or policy-makers to accomplish this step.6, 11 These frameworks include explicit criteria which the group considers individually and in aggregate focused on the relative benefits and harms, as well as other considerations. Such frameworks can facilitate: i) consideration of all relevant criteria in the decision-making process; ii) examination of the pros and cons of each intervention option; iii) presentation of relevant evidence for each criterion; iv) identification of the reasons for any disagreement within the expert group; v) transparent reporting of the decision-making process; and vi) crafting of the rationale statement for each recommendation. Populated EtD frameworks can also facilitate implementation in the local context, by assisting the end-user in understanding how and why specific recommendations were made, and by providing data and evidence on each decision criterion which may help to facilitate local adoption or inform adaptation.
In environmental health, once hazards are identified and risks assessed, organizations may want to examine mitigating and prevention interventions and make recommendations and policies based on systematic reviews of research evidence and other data and information. An EtD framework suitable for environmental health interventions will facilitate this process. However, such frameworks have not been widely used in this field.
The objectives of this review were to identify, compare and contrast key EtD frameworks for interventions in clinical medicine, public health and environmental health and to summarize the key decision criteria across these frameworks. The identification of these key elements will inform the development of an EtD framework for use in formulating recommendations regarding mitigating and prevention interventions related to exposure to harmful substances in the environment.
Methods
In order to identify relevant, existing EtD frameworks, we took two approaches: a search of the peer-reviewed literature for systematic reviews of EtD frameworks, and identification of frameworks used by a range of organizations which make decisions or formulate recommendations in clinical medicine, or public or environmental health.
1. Scope
The focus of this paper is on the substantive criteria for decision-making with respect to interventions, including both normative criteria (what should be done) and feasibility criteria (what can be done).12 Criteria related to the process (versus substance) of decision-making are beyond the scope of this work. While criteria in healthcare decision-making can be either qualitative or quantitative,10 this paper focuses on the former, although quantitative criteria (commonly used in HTA) were not excluded.
There are an increasing number of organizations which use systematic and transparent approaches to synthesize evidence in environmental health for hazard identification,13 risk assessment,14 and for the synthesis of the benefits and harms of interventions.15 These organizations were not included in the current analysis because they do not make recommendations on interventions to mitigate the effects of harmful exposures.
2. Bibliographic database search and article screening
To identify relevant EtD frameworks, we searched Medline via PubMed for systematic reviews of EtD frameworks, templates or tools published in English and indexed in the 10-year period up to 7 January 2021. The complete search strategy is found in Annex 1. We also solicited the advice of guideline development experts for any additional potential citations. Publications were included if they were systematic reviews of frameworks, tools or templates for formulating decisions or recommendations, or for priority setting related to the use of interventions (including diagnostic tests) in clinical medicine, public health, or environmental health. The included frameworks focused on population-level interventions; decision tools for the provider-patient interaction and patient decision aids were excluded. Frameworks which focused exclusively on economic considerations were also excluded. The setting for both decision-making and implementation of recommendations was not restricted.
Two persons (SLN and MTA) screened the titles and abstracts, and full-text versions were retrieved for studies which potentially fulfilled inclusion criteria. Consensus was achieved between the two reviewers for final inclusion in the review.
3. Search for EtD frameworks used by key organizations
in addition to the systematic review described above, we examined a broad range of organizations including clinical and public health guideline development groups in academia and the private sector, healthcare provider professional organizations, governmental agencies and international organizations. Given the purpose of this review, we focused particularly on organizations that work in the field of environmental health. Organizations were identified based on author knowledge of prominent guidelines in public health and clinical medicine, by snowballing (reviewing the origin of or basis for identified frameworks), and by conferring with a broad network of guideline developers internationally based on our experiences and contacts.
We used this pragmatic approach for several reasons. First, because of the dominance of the GRADE (Grading of Recommendations Assessment, Development and Evaluation) EtD framework over the last 15 years,5, 11, 16 many organizations either use GRADE or a modification thereof. Thus, a more exhaustive search was unlikely to yield novel or unique frameworks. Second, because the methods used by guideline development organizations are infrequently published in the peer-reviewed literature, an extensive hand-search of the grey literature including organizational web-sites would have been required, which was infeasible.
4. Data extraction and synthesis
For each systematic review identified in our search, a single author (SLN) extracted key information describing the focus of the review, time period searched, and the main findings, including an overview of the criteria identified across the included frameworks. We did not extract data on each individual framework included in each review.
For the EtD frameworks used by key organizations, we extracted data on organizational characteristics, the methods used to develop the framework, the funders and any declared interests of the developers of the framework, how evidence should be used to inform each criterion in the framework, quality assessment of individual studies and of the body of evidence, the specific EtD criteria, and the types of conclusions or recommendations formulated. One author extracted these data (SLN) and a second checked them (MTA); disagreements were discussed and consensus reached. Data were extracted into a template in Excel (Microsoft Corporation, Redmond WA, USA)
For the key organizations, the primary decision criteria for each EtD framework were extracted from the main published reports, and specifically from the identified template or list of criteria. For organizations for which there are significant background materials or publications such as for the GRADE framework, the decision criteria that were extracted into the tables and figure include those that are explicitly presented as part of the framework. Other concepts or criteria which are mentioned only in the narrative text accompanying a framework were not extracted unless the text suggested that they were important and were consistently applied in decision-making. Where a clear set of criteria could not be identified (e.g., criteria were presented in narrative text without clear demarcation of the main decision criteria, or when criteria appeared to vary across publications), the authors of the framework were contacted for clarification.
Data were summarized in a narrative, qualitative manner. Given that the organizations and frameworks discussed were a convenience sample, descriptive statistics and statistical comparisons are not meaningful. Quality assessment of the identified EtD frameworks was not performed as there is no standard for such an assessment.
Results
1. Identified systematic reviews of EtD frameworks
The bibliographic database search for systematic reviews of EtD frameworks yielded 399 citations, of which four fulfilled inclusion criteria7, 10, 17, 18 (Table 1, Annex 2 and 3). One additional study which fulfilled inclusion criteria was identified by the co-authors.8 Each of these reviews included a cohort of EtD frameworks which the authors had systematically identified, with a particular focus: multicriteria decision analysis in HTA,10 vaccine adoption into national programs,17 diagnostic tests,7 decision-making in local low-income settings,18 and frameworks for informing health system coverage decisions.8 Each review summarized the main EtD criteria identified across their included, individual frameworks, and these main criteria are presented in Table 1. All five reviews included the balance of benefits and harms and consideration of resource use or cost-effectiveness. Only the review by Mustafa and colleagues7 included an assessment of certainty of evidence. Considerations of equity, acceptability and feasibility were included only in the reviews by Baltussen et al.10 and Burchett et al.17 The EtD criteria outlined in these five reviews generally corresponded to the main criteria in the EtD frameworks of the selected key organizations (see Results section 2). One additional publication, excluded from our review because it was not a systematic review, provides a useful list of “value assessment frameworks” for use in HTA for clinical interventions.19
Decision criteria are the main criteria that the review authors identified across the frameworks which were included in their review.
Abbreviations: EtD, evidence-to-decision; GRADE, Grading of Recommendations, Development and Evaluation; HTA, health technology assessment; MCDA, multi-criteria decision analysis; n, number of studies; NR, not reported
2. EtD frameworks used by key organizations
Fourteen organizations that use EtD frameworks for recommendation formulation or decision-making were examined in detail (Table 2). One organization, the GRADE Working Group, is an informal network of individuals from a broad range of organizations, including academic institutions, national and international guideline development agencies, and healthcare provider organizations, among others.20 The GRADE Working Group does not, itself, publish guidelines, but rather develops processes and methods for use by other organizations which develop guidelines. The other thirteen organizations develop guidelines for specific audiences and with a clearly defined scope or set of topics: five are agencies of national governments,21–25 two are related to the World Health Organization (WHO),26, 27 one is a U.S. State agency (California Environmental Protection Agency (CalEPA)28) and the remainder are non-governmental organizations or academic groups.2, 19, 29–31
Evidence-to-decision frameworks: Characteristics of the organizations that developed the frameworks
Abbreviations: EtD, evidence-to-decision; GRADE, Grading of Recommendations, Development and Evaluation; MCDA, multi-criteria decision analysis; NR, not reported; WHO, World Health Organization
Footnotes:
(a) Organization that developed the framework
(b) Funder(s) for the development of the framework
(c) Declaration of interests of the developers of the framework
Two organizations focus primarily on clinical care: the Scottish Intercollegiate Guideline Network (SIGN)32 and the US Preventive Service Task Force (USPSTF).33 The Guide to Community Preventive Services (GCPS) focuses on interventions aimed at groups, communities or health systems34 and WHO and WHO-INTEGRATE primarily on public health interventions.26, 27 Three organizations are oriented to health technologies, with a prominent focus on economic evaluations and resource considerations.19, 30, 31 The UK National Institutes for Health and Care Excellence (NICE) examines a broad range of clinical, public health, and social interventions.35 The Advisory Committee on Immunization Practices (ACIP) focuses exclusively on vaccine recommendations for U.S. populations.36 Breast Cancer Prevention Partners (BCPP),29 CalEPA,28, 37 and the Navigation Guide2, 15 focus on the human health effects of hazardous substances in the environment.
The amount of information available on each of the 14 key organizations varied greatly. There are a large number of publications describing the GRADE system for guideline development, including the GRADE EtD framework and its use.20 On the other hand, there is only one document describing the approach of the BCPP.29
Development process and expertise
For the organizations that developed de novo EtD frameworks and described the process for developing them, all used an iterative approach based on an examination of other organizations, with input from experts in guideline methods and evidence synthesis (Table 2). The most comprehensive approach was taken by Rehfuess and colleagues27 in development of the WHO-INTEGRATE framework. Their approach included development of a theoretical framework, a review of WHO basic documents, a systematic review of EtD criteria, input from a range of stakeholders, and a thematic analysis to identify key domains.38
It was difficult to discern the expertise of contributors to framework development; most groups appeared to consist mainly of academic, generalist guideline methodologists with either clinical or public health experience. Social scientists led work on WHO-INTEGRATE27 and contributed to the GRADE public health framework.6 Eight of the 14 organizations reported who funded the development of the EtD framework and five reported the conflicts of interest of the framework developers; the remaining organizations did not provide this information.
Decision criteria
Eighteen frameworks were examined in detail: one from each of the 14 organizations, except for CalEPA and GRADE, where two28,37 and four 6, 9, 11, 16 unique frameworks were examined, respectively. There were significant commonalities across frameworks in the criteria for formulating recommendations (Table 3 and Figure 1). Unsurprisingly, all included consideration of benefits, except the CalEPA framework for pesticides28 which examines risks (of environmental exposures) and not benefits. All frameworks included an assessment of harms of the intervention under consideration. Fifteen of the 18 frameworks included some assessment of certainty or quality of the body of evidence across outcomes in the decision-making process (in contrast to an assessment of individual outcomes as part of an evidence review discussed below): BCPP, one of the CalEPA frameworks, and ISPOR (International Society for Pharmacoeconomics and Outcomes Research) did not.19, 28, 29 Some measure of costs, resource use or cost-effectiveness was found in all frameworks except BCPP, USPSTF and the GCPS.29, 33, 34 For the latter two frameworks, the U.S. government prohibits consideration of financial resources in the decision-making process. Other decision criteria were variably included: feasibility (13 frameworks) equity (12), values (11), and acceptability (11). Only two frameworks included human rights: WHO26 and WHO-INTEGRATE frameworks.38
Legend. Across the top of the figure, broad categories of evidence-to-decision criteria are presented. For each listed organization, the cell is shaded if their evidence-to-decision framework encompasses one or more criteria within a category.
Abbreviations:
AA, Alternatives Analysis
ACIP, Advisory Committee on Immunization Practices
BCPP, Breast Cancer Prevention Partners
CalEPA, California Environmental Protection Agency
EVIDEM, Evidence and Values Impact on DEcision Making
GRADE, Grading of Recommendations, Assessment, Development and Evaluation
GCPS, Guide Community Preventive Services
ICER, Institute for Clinical and Economic Review
ISPOR, International Society for Pharmacoeconomics and Outcomes Research
NICE, National Institutes for Health and Care Excellence
WHO, World Health Organization
Criteria included in evidence-to-decision frameworks
Abbreviations: NI, not included
Footnotes
(a) The criteria listed here are from both the “Considered judgement pro-forma 2014” form and the headings in the SIGN 50 guideline handbook as these differ.
“Priority of the problem” was included in the frameworks based on GRADE,26, 32, 36 as well as the BCPP,29 EVIDEM (Evidence and Values Impact on DEcision Making),30 ISPOR19 and Navigation Guide frameworks.15 However, in the background materials for several other frameworks, the burden of the disease (number of affected persons, severity of the disease, or economic and social costs) was mentioned as an important consideration when prioritizing an intervention for development of a guideline or policy. This latter consideration, however, differs from use of this criterion when deciding on the strength and direction of a recommendation.
The meaning of the criterion “values and preferences” has varied over time and across publications. This phrase encompasses two rather different constructs: the relative value that persons affected by the recommendations place on the outcomes of the intervention (values), and the preferences such persons have regarding the intervention options. The GRADE Working Group initially combined these two constructs5 with a definition encompassing both.39 The 2016 update of the GRADE EtD framework distinguished these constructs, with the inclusion of two separate criteria: variability of the value affected persons place on the main outcomes, and acceptability of the intervention11 to various stakeholders. The WHO EtD framework, published in 2014,26 preceded the GRADE modifications of 2016,11 includes “values and preferences” as well as “acceptability” and “feasibility”. The text explains that “values and preferences” “pertain to the relative importance people assign to the outcomes associated with the intervention or exposure; they have nothing to do with what people think about the intervention itself.”26 Other key organizations include a decision criterion related to values using verbiage closely aligned with GRADE15 and focused exclusively on the relative value of outcomes,32, 35 or they include the concept but with a somewhat different meaning.19, 28, 36
Evidence used to inform each criterion when organizations make decisions
All of the frameworks report that the EtD criteria should be informed by evidence obtained from a variety of sources, with an emphasis on systematic reviews of research evidence (Table 2). Twelve of the organizations recommend an assessment of the quality (validity or certainty) of the body of evidence for important outcomes as part of the evidence review, and most recommend the GRADE system with or without modifications. Only BCPP29 and one of the CalEPA guidelines,28 do not recommend some assessment of quality or certainty of the evidence as part of the systematic review process. The approaches used by the GCPS 34 and USPSTF33 differ somewhat from GRADE in their assessment of the “strength of evidence” and “certainty of net benefit”, respectively. The EtD frameworks with an economic focus describe varied approaches to assessing quality of the body of evidence.19, 30, 31
Nomenclature for recommendations
Nine of the included organizations provide specific guidance on the nomenclature for the types of recommendations formulated based on the EtD criteria (Table 2). The most common categorization of recommendations among those nine organizations was two levels of strength both for and against a recommendation, i.e., four categories.11, 26, 32 GRADE has extensive documents on this issue and uses the terms “strong, weak or conditional“.5 The USPSTF has five categories of recommendations,33 while ACIP36 and the GCPS34 have three. The economic-focused frameworks refer to “value” in various ways.30, 31, 40 NICE uses the wording of recommendations to reflect the strength of the evidence (e.g., offer, advise, consider) rather than standardized terms to represent the strength of the recommendation.35
Five organizations provide explicit guidance on the situations where recommendations cannot be formulated due to insufficient evidence or a close balance between benefits and harms (Table 2).11, 33–36 The USPSTF provides the most detailed guidance on this situation.41 Ten of the organizations suggest including knowledge or research gaps with the recommendations, with particular emphasis in ACIP, NICE, GCPS and the USPSTF.
Specific evidence-to-decision frameworks
i GRADE evidence-to-decision frameworks
GRADE includes EtD frameworks for four different purposes: i) clinical recommendations, individual perspective; ii) clinical recommendations, population perspective; iii) coverage decisions; and iv) health system and public health recommendations/decisions. (There is a fifth GRADE EtD framework for diagnostic, screening and other tests but this was not included in this review of interventions considered most relevant for application to environmental health.11) These frameworks are all very similar, all with 12 criteria covering the same concepts, with some variation in verbiage, tailored to the different audiences (Table 3). There is more emphasis on resource considerations, equity, acceptability and feasibility for health systems and public health decisions than for individual patient clinical recommendations.6, 9
ii Evidence-to-decision frameworks based directly on GRADE
ACIP’s EtD framework is derived directly from GRADE,36 while NICE25 and SIGN’s32 EtD criteria closely resemble those of GRADE. WHO uses the GRADE EtD framework which was current when the WHO Handbook for Guideline Development, 2nd edition was published in 2014.26 Guideline development groups supported by WHO can modify this EtD framework to meet the needs of individual guidelines, and are encouraged to use the current GRADE EtD framework published in 2016.11, 16
iii Other evidence-to-decision frameworks
The WHO-INTEGRATE EtD framework version 1.0,27, 38 first published in 2019 (Table 2), was developed in response to a perceived need to take a complexity perspective into account and to incorporate public health and WHO-specific values when developing WHO guidelines. The developers also sought to address several weaknesses that they noted in the GRADE EtD:27 the absence of a theoretical framework; failure to adequately consider the role of social and economic determinants of health; inadequate consideration of complex interventions, system changes, and the environment in which interventions are implemented; and an undue focus on benefits and harms and not on other considerations critical for decision-making in public health.
WHO-INTEGRATE includes six broad criteria (Table 3): balance of benefits and harms; human rights and sociocultural acceptability; health equity, equality and non-discrimination; societal implications; financial and economic considerations; and feasibility and health system considerations.27 A seventh criterion, quality of evidence, is a meta-criterion that applies to each of the six other criteria. WHO-INTEGRATE also includes a number of sub-criteria. For example, the main criterion “Health equity, equality and non-discrimination”27 includes several sub-criteria related to the intervention: impact on health equality and/or health equity, distribution of benefits and harms, affordability, and accessibility.
One of the key concepts of the WHO-INTEGRATE approach is the careful consideration of the relative importance of the various EtD criteria at the planning stage of guideline development. Logic models or conceptual frameworks are strongly recommended in order to understand the interrelationships between the intervention and the context in which it is delivered, with prioritization of EtD considerations for the specific decision or recommendation. Relevant evidence is then sought on this subset of criteria.
The USPSTF22 develops an analytic framework depicting the causal pathway by which the intervention may achieve its effects (benefits and harms) and the various questions which must be researched to address the overarching question of net benefit of the intervention. The USPSTF’s recommendations are based primarily on net benefits and the certainty thereof, although other factors may be considered (Table 3).
The GCPS modeled their original processes and methods on those of the USPSTF34 with some later modifications (personal communication Dr David Hopkins, 25 January 2021). Based on systematic reviews focused mainly on benefits and harms of the intervention, the beneficial effects of the intervention are assessed as strong, sufficient or insufficient strength of evidence. This assessment is then “upgraded” or “downgraded” based on factors including harms, equity considerations, magnitude of effect and applicability of the evidence to US populations.
Three of the frameworks from key organizations reflect an economic perspective for decision-making in health systems or in decisions on coverage at a system or national level: EVIDEM,30 ISPOR,42and ICER (Institute for Clinical and Economic Review).31 These frameworks all included consideration of comparative effectiveness. ISPOR presents data as quality adjusted life years (QALYs) and net costs, and includes other considerations such as “value of hope” and “insurance value” (Table 3).42 ICER31 focuses on comparative clinical effectiveness and incremental cost-effectiveness along with “contextual considerations”, such as “improving return to work and/or overall productivity”. EVIDEM30 also focuses on comparative effectiveness, along with cost consequences and other considerations (e.g., “environmental impact”).
iv Evidence-to-decision frameworks in environmental health
The three frameworks which focus on interventions in environmental health were generally less well developed and lacked specificity compared with the clinical and public health frameworks.2, 28, 29 BCPP developed a comprehensive breast cancer primary prevention plan for the US State of California, focusing on primary prevention and systemic interventions to reduce the incidence of breast cancer, based on the principles of social justice and equity.29 This organization prioritized a large number of potential interventions using criteria focused on alignment with the organization’s goals and guiding principles, whether the intervention addresses cross-cutting or systemic problems, past success of the intervention, and potential harms.
CalEPA’s Office of Environmental Health Hazard Assessment (OEHHA) leads California’s assessment of health risks posed by environmental contaminants43 and works with the State Department of Pesticide Regulation (DRP) to develop regulations related to pesticide worker health and safety. In response to a review by the US National Academy of Sciences (NAS),44 CalEPA published an update of their methods for risk assessment and risk management of pesticides28 which focuses on an assessment of a range of options and their risks, and examination of scientific, social, legal, and economic factors, as well as practicality and enforceability.
CalEPA’s Department of Toxic Substances Control Safer Products and Workplaces Program has published a formal approach to alternative analysis of potentially hazardous components of consumer products.37 This approach guides decisions on product removal, redesign or other change to reduce the potential impact of the chemical of concern. The analysis compares alternative products across factors including potential chemical exposures, public and environmental health impact, product function and requirements, and economic implications. This analysis considers all relevant life cycle consequences of the chemical or product of concern, from inputs into manufacturing, through end-of-life disposal. While not referenced in the summary “List of Factors for Consideration in Alternatives Analysis” (Appendix 3-1),37 other considerations are mentioned in the document text. For example, the entity performing the analysis determines which impacts are most significant based on its values. Likewise, the variable impacts across sensitive subpopulations are to be considered with respect to exposure to or use of the product. Feasibility is also discussed in the text, with respect to the assessment of product function, performance, and the legal requirements.37 Finally, acceptability is considered in regards to the function of alternative compounds.
The Program on Reproductive Health and the Environment at the University of California at San Francisco has developed a process and methods for integrating scientific findings from human and nonhuman studies to determine the overall strength of the evidence on the effects of environmental exposures on outcomes in humans.45 This approach, published as The Navigation Guide2, 15, 45 has adapted the GRADE approach to integrate multiple streams of evidence and a range of study designs, to draw conclusions on the quality and strength of the body of evidence of a substance’s toxicity. This assessment of toxicity is then combined with information on exposure (level, duration and timing) to provide an assessment of risk of adverse health outcomes. This risk assessment could then be combined with consideration of: 1) whether a less toxic agent is available as an alternative; and 2) “values and preferences”, to arrive at a recommendation.2, 15
Discussion
A number of EtD frameworks have been developed in a variety of clinical, and public and environmental health disciplines, and there is significant overlap in the criteria used to inform recommendations or decisions across these frameworks. Benefits and harms are almost universally included, and an assessment of certainty or quality of the body of evidence and some measure of resource use were included in most frameworks examined. Other decision criteria such as values, equity, feasibility, acceptability were variably included, while only two frameworks encompassed human rights. There was variation across frameworks in terminology, definitions and presentation of EtD criteria. The five systematic reviews of EtD frameworks describe decision criteria that were similar to those presented in the frameworks from key organizations.
The 18 frameworks identified for 14 key organizations focused mainly on clinical medicine or public health interventions. Of these, only one was based on a conceptual framework, and rarely was a systematic review of potential criteria performed during the framework’s development. The GRADE framework is the most well developed, with extensive information and guidance on its methods. A number of the key organizations examined have adopted GRADE, often with minor modifications.15, 26, 32, 35, 36 Three of the four frameworks related to environmental health lacked detail on the specific criteria for decision-making.19, 30, 31 Frameworks originating in the HTA realm unsurprisingly focus on economic considerations, while also including criteria similar to those encompassed by other frameworks.
General aspects of evidence-to-decision frameworks
The process and methods for developing EtD frameworks were often superficial and poorly reported. Only WHO-INTEGRATE38 and the GRADE coverage8 frameworks were developed following a systematic review of existing frameworks or potential decision criteria. Only WHO-INTEGRATE describes a conceptual framework27 underpinning the EtD criteria. The lack of a conceptual framework represents a significant weakness in most EtD frameworks: key considerations may be missed and long-standing criteria may be perpetuated without adequate scrutiny.
GRADE has dominated guideline methods in health care and public health over the last 15 years. This has led to productive collaborations, standardization of processes and methods, and the development of packages of detailed methods and guidance by diverse contributors from across the globe. On the other hand, there are instances where intellectual dominance by a single group – i.e., monopolies of knowledge – can suppress innovation and slow down development processes.46
EtD frameworks largely focus on, and are optimal for, single-component interventions with simple linear pathways from intervention to outcomes. In addition, most EtD frameworks do not consider the context in which the intervention is delivered. With the exception of WHO-INTEGRATE, EtD frameworks do not explicitly or even implicitly incorporate a complexity perspective including the inter-relationship between the intervention and the context or environment in which it is delivered. Only the GCPS, the USPSTF, and WHO-INTEGRATE recommend the use of visual depictions of the relationships among the intervention components, the causal pathway, and the context for the intervention (i.e., analytic frameworks, logic models, or similar depictions). Such approaches facilitate examination of multiple key questions linked across the causal pathway for intervention effects, as well as the consideration of contextual factors and externalities which may be important for decision-making.
Guidance on if and when to identify, synthesize and integrate evidence on criteria other than benefits and harms of the intervention into the decision-making process was provided only for WHO-INTEGRATE.27 Most guideline development processes focus almost exclusively on benefits and harms in the evidence review and in discussions, without explicit efforts to prioritize other criteria or seek evidence to inform them. Qualitative evidence, particularly on considerations such as acceptability or feasibility, is rarely gathered. Meetings at which recommendations are formulated often include only brief, ad hoc discussions of these other criteria: this is a major limitation of most current approaches to recommendation formulation.
Evidence-to-decision criteria across frameworks
It was difficult to compare specific EtD criteria across frameworks due to significant variation in approaches to lumping and splitting criteria across frameworks, as well as in the terminology and definitions used. While most frameworks presented broad categories of criteria (e.g., “equity”), only WHO-INTEGRATE27 provides detailed sub-criteria to facilitate understanding and decision-making. Some main criteria included several constructs: aggregating diverse elements within a single criterion can be difficult to implement in the decision-making context and when reporting the rationale for the recommendation or decision. For example, WHO-INTEGRATE’s “Human rights and sociocultural acceptability” includes both human rights and what other frameworks refer to as “preferences” or “acceptability”. CalEPA28 articulates in one question a group of criteria: “Evaluating those options according to a value system that includes scientific, social, legal and economic factors, as well as practicality and enforceability”, without providing additional guidance or sub-criteria for decision-makers.
There is also variability in how criteria are defined and whether operational definitions or guidance are provided. The criterion “values and preferences” is particularly problematic. Unfortunately, guideline developers continue to use the phrase “values and preferences”, often with unclear and variable meaning. The WHO EtD framework, published in 2014,26 includes an outdated definition of “values and preferences” which differs both from the 2008 GRADE guidance5 and from the 2016 modifications:11, 16 it needs to be updated.
The GRADE criterion of “priority of the problem” is also problematic. It was initially included in the GRADE framework to facilitate prioritization across interventions, such as at the national or sub-national levels. However, in most guideline development scenarios, the problem on which interventions and comparators are focused has already been determined to be of high priority. Thus, examination of the burden of disease and other priority considerations is irrelevant at the stage of recommendation formulation.
The criteria related to resource use vary considerably, and may include cost, affordability, infrastructure needs, personnel training, and measures of economic efficiency (e.g., cost-benefit and cost-effectiveness). This variability is due in part to the different perspectives of the sponsors of the EtD framework or of the target audiences for the organization’s products. For example, the USPSTF is prohibited by national legislation from considering cost or cost-effectiveness in their recommendations. On the other hand, for the UK’s NICE, cost-effectiveness is a critical part of their decisions. There is also variability in the extent and specificity of the guidance on how to incorporate resource use into decision-making. While well-developed for NICE,35 and reasonably so for GRADE,47 guidance is almost completely lacking for WHO26 and WHO-INTEGRATE.27
The criterion of “equity” is poorly described and little operational guidance is provided across the frameworks except for WHO-INTEGRATE which provides several sub-criteria for consideration under the main criterion of “health equity, equality and non-discrimination”.27
Group decision making
Group decision making when formulating recommendations in guidelines is rarely a simple, linear process. While EtD frameworks support the normative aspect of decision making in terms of how expert panels ‘‘should’’ or ‘‘ought to’’ develop recommendations by providing a structure for decision making,48–50 many factors affect how groups make decisions.48 In addition to the features or characteristics of the decision itself (e.g., criteria in the EtD framework such as resource use and equity), other factors may play a role: i) situational or contextual factors (e.g., time pressure, social context, gender bias, political pressures); ii) individual characteristics of the decision maker (e.g., role on the expert panels, cultural and professional background, race/ethnicity, methodological expertise); and iii) individual panel member’s emotions and experiences (e.g., personal experiences with a disease). How these various factors contribute to decision making and recommendation formulation in guidelines is unknown,48 but their potential effects must be kept in mind and made explicit to the extent possible.
While acknowledging the complex process of formulating recommendations in guidelines, EtD frameworks are a valuable tool and have led to vast improvements over “free-for-all” meetings where decision making criteria were selected in an ad hoc manner, and it was often unclear which considerations actually contributed to final decisions and their relative weight, what evidence was examined, and how and why the final recommendation or decision was arrived at.
Applying clinical and public health frameworks to environmental health
Applying approaches for clinical medicine interventions to other scientific fields, including environmental health, is challenging. There are important differences both in the assessment of the quality (certainty) of the body of evidence, and in translating evidence to recommendations or decisions. Woodruff and colleagues15 note that the GRADE system and other evidence-based medicine approaches have limitations in terms of applicability to questions and decision making in environmental health. The reasons include: i) the need to combine human, animal and (sometimes) mechanistic evidence; ii) the paucity of randomized controlled trials (RCTs) and other types of experimental studies in humans due to ethical considerations; and iii) differences in the decision-making context (e.g., weighing the benefits and harms is different for clinical interventions than for unintended exposures to substances in the environment).15 Nonetheless, the constructs of the GRADE and WHO-INTEGRATE frameworks likely apply to environmental health interventions.
Limitations of this analysis
The approach taken for this review and analysis has a number of limitations. The methods for identifying existing frameworks were limited: the systematic review included only reviews and did not include a search for all published EtD frameworks, and focused only on English language literature accessible through PubMed. Furthermore, many EtD frameworks are not published in the peer reviewed literature: rather, they are found on organizational web-sites and/or in organizational procedure manuals. For the review of key organizations, only a convenience sample was examined and EtD frameworks with important additional considerations or novel approaches may have been overlooked. However, it is unlikely that this approach missed key or significantly different frameworks in view of the dominance of GRADE and our consultation with experts on EtD frameworks.
Our analysis also has limitations related to the nature of the available data. Published information on the methods for developing frameworks was generally sparse, except for GRADE and WHO-INTEGRATE. The descriptions of the EtD criteria and how to operationalize them were also sparse for many frameworks. For some frameworks it was difficult to identify exactly which criteria were routinely included in the decision making process. While the publication might present a list or table of criteria, additional criteria might be mentioned in the adjacent text. However, the degree to which the guideline group or other decision maker addressed these additional criteria was often unclear.
Conclusions and next steps
EtD frameworks are an extremely useful tool for recommendation formulation and decision making in healthcare and other scientific fields. They enable decisions to be made based on research evidence and with explicit consideration of a range of constructs, and they facilitate clear articulation of the rationale for decisions.
The principles underpinning evidence-informed, transparent and impactful decision making are the same across scientific fields, including environmental health. The GRADE EtD framework, along with WHO-INTEGRATE with its focus on contextual issues and inter-relationships, provide a useful starting point for consideration for decision-making for interventions in environmental health. Significant modifications will be needed, however, given the nature of the evidence base and the complex context in which environmental health interventions are designed, implemented, regulated and evaluated. The process for developing an EtD framework for environmental health interventions requires a broad range of experts, including not only environmental health scientists, but evidence synthesis and guideline methodologists, public health generalists, human rights and ethics experts, social scientists, and economists, among others. An iterative development process will be needed: a draft framework should be pilot-tested, revised and evaluated. The framework’s utility and impact on decision making, and the quality and impact of the resultant recommendations need careful evaluation, with the results used to develop future iterations.
Data Availability
All data are available from the corresponding author.
Funding source
Funding for this work was provided by the JPB Foundation (Grant 681).
Annexes (online)
Annex 1. Search strategy and inclusion criteria
((“Policy Making”[Mesh]) OR (“Decision Making”[Mesh])) AND ( ((framework[Title/Abstract]) OR (frameworks[Title/Abstract])) OR ((template[Title/Abstract]) OR (templates[Title/Abstract]) OR (tool[Title/Abstract]) OR (tools[Title/Abstract])))
Filters: English language, systematic review, last 10 years
The search was executed 7 January 2021 in the PubMed search engine for Medline.
Annex 2. PRISMA flow diagram
Annex 3. Results of the search: full-text review
The search of PubMed performed on 7 Jan 2021. 399 citations were identified, 25 of which were reviewed in full text: these are listed below with the reason for exclusion.
Abbreviations: E, excluded citation; E2D, evidence-to-decision; I, included citation; LMICs, low- and middle-income countries; NA (Not applicable because the publication fulfilled inclusion criteria)
Annex 3. References
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
Footnotes
Declaration of competing financial interests: Susan L Norris is a member of the GRADE Working Group and has published on GRADE; she is a former employee of the World Health Organization and in that role contributed to the WHO Handbook for Guideline Development (2nd edition 2014) and to the WHO-INTEGRATE framework.
Max T. Aung: Nothing to declare
Nicholas Chartres: Nothing to declare
Tracey J Woodruff: Nothing to declare