ABSTRACT
Background The Predictive Approaches to Treatment Effect Heterogeneity (PATH) Statement provides guidance for using predictive modeling to identify differences (i.e., heterogeneity) in treatment effects (benefits and harms) among participants in randomized clinical trials (RCTs). It distinguished risk modeling, which uses a multivariable model to predict risk of trial outcome(s) and then examines treatment effects within strata of predicted risk, from effect modeling, which predicts trial outcomes using models that include treatment, individual participant characteristics and interactions of treatment with selected characteristics.
Purpose To describe studies of heterogeneous treatment effects (HTE) that use predictive modeling in RCT data and cite the PATH Statement,
Data Sources The Cited By functions in PubMed, Google Scholar, Web of Science and SCOPUS databases (Jan 7, 2020 - June 5, 2023).
Study Selection 42 reports presenting 45 predictive models.
Data Extraction Double review with adjudication to identify risk and effect modeling and examine consistency with Statement consensus statements. Credibility of HTE findings was assessed using criteria adapted from the Instrument to assess Credibility of Effect Modification Analyses (ICEMAN). Clinical importance of credible HTE findings was also assessed.
Data Synthesis The numbers of reports, especially risk modeling reports, increased year-on-year. Consistency with consensus statements was high, except for two: only 15 of 32 studies with positive overall findings included a risk model; and most effect models explored many candidate covariates with little prior evidence for effect modification. Risk modeling was more likely than effect modeling to identify both credible HTE (14/19 vs 5/26) and clinically important HTE (10/19 vs 4/26).
Limitations Risk of reviewer bias: reviewers assessing credibility and clinical importance were not blinded to adherence to PATH recommendations.
Conclusions The PATH Statement appears to be influencing research practice. Risk modeling often uncovered clinically important HTE; effect modeling was more often exploratory.
INTRODUCTION
Findings from randomized controlled trials (RCTs) have limitations for patients making personal treatment decisions. (1–6) RCTs are usually planned and sized to estimate overall or average treatment effects in trial populations. Even in very positive RCTs, some patients do not benefit from the study treatment, and some experience adverse effects of treatment. Patients have many characteristics that might influence their own likelihood or “risk” for study outcomes or for experiencing either benefit or adverse effects from the treatment.
Until recently, guidance for identifying possible differences, or heterogeneity, of treatment effects (HTE) among RCT participants (1,7–10) has focused on considering one characteristic at a time, testing hypotheses for “effect modification” or statistical interaction of the characteristic with treatment. Analyses test whether treatment effects differ between patient subgroups, such as men vs. women or persons with vs. without diabetes. Relative treatment effects (risk, odds, or hazard ratios) are usually compared. These subgroup comparisons are almost always underpowered for detecting interactions in RCTs populations, leading to false negative findings. At the same time, if many subgroups with low prior probabilities for effect modification are explored, chances for false positive findings are also high. Thus, guidelines have consistently recommended limiting the number of subgroups examined, ideally to those with prior evidence or strong biologic or clinical rationale for HTE and using caution in interpreting apparent interactions or applying findings to clinical practice. A fundamental limitation of “one-variable-at-a-time” subgroup analyses is that they do not provide a single treatment effect prediction for an individual because individuals simultaneously belong to multiple subgroups that can vary in whether or how they appear to benefit.
Both patient-centered outcomes research (11) and precision medicine (12) have heightened interest in identifying important HTE to improve individualized clinical decision-making. The authorizing legislation (13) for the Patient-Centered Outcomes Research Institute (PCORI) explicitly instructed PCORI to require awardees to look for individual differences in the effectiveness of health care treatments and services.
In 2018, an expert panel funded by PCORI described a new approach to HTE analyses. The Predictive Approaches to Treatment Heterogeneity (PATH) Statement (14,15), proposed predictive modeling for identifying clinically important HTE. The Predictive Approaches to Treatment Heterogeneity (PATH) Statement (14,15), proposed predictive modeling for identifying clinically important HTE. Predictive modeling accounts for the effects of multiple patient attributes (independent variables) on trial outcomes simultaneously and can produce individualized predictions of potential benefits and risks of study treatments. The Statement emphasized that important HTE can be seen on the absolute as well as relative scales (i.e., as variation among patients in treatment-related differences in outcomes as well as ratios) and that absolute treatment effects are more useful than relative effects for clinical decision-making.
The PATH Statement distinguished two approaches to predictive modeling. “Risk modeling” focuses on one potential effect modifier: the individual’s risk (or probability) of experiencing an outcome (usually the trial’s primary outcome). (16) Multiple baseline characteristics are incorporated into models predicting risk for the outcome and a risk score is generated for each participant. In a second step, treatment effects are examined across strata (e.g. by quartiles) of predicted risk. The Statement encouraged use of previously developed, validated prediction models when available and appropriate to the RCT population, but also suggested that if an appropriate model is not available, one can be developed internally, using observed study outcomes, covariates measured at baseline and all participants regardless of treatment assignment.
Risk modeling has both a mathematical and an empirical rationale in evaluating possible HTE. It builds on the clinical intuition that patients at greater risk for study outcome(s) have more to gain from a beneficial treatment. Mathematically, baseline risk is linked to the absolute treatment effect as follows:
For any value of relative treatment effect, or relative risk reduction (1 – RR), the absolute risk reduction (ARR) increases as baseline risk (the control event rate, CER) increases. This relationship has been called “risk magnification,” (17) but is more accurately a “benefit magnification” when the overall treatment effect is beneficial. Implicit in Equation 1 and the concept of benefit magnification is the assumption that the relative treatment effect (1-RR) is constant across baseline risk. Clinical guidelines and disease management strategies also make this assumption in recommending that persons at greater risk be treated first or more aggressively. In re-analyses of 14 RCTs with positive results, (18) 13 suggested benefit magnification, but in the 14th, (19), a significant interaction of baseline risk and treatment effect was found. Risk modeling allows for testing this assumption by modeling study outcomes as a function of the risk score, treatment assignment, and a statistical interaction between the two. Findings that relative treatment effects vary significantly across baseline risk could markedly change individual treatment recommendations.
In the second approach, “effect modeling”, a model predicting the trial’s outcome is developed within the RCT data and includes independent variables for treatment assignment, individual patient characteristics and interactions of treatment with selected characteristics. This allows direct estimation of the predicted treatment effect for individual participants. The Statement recognized the more exploratory or “data-driven” nature of most effect modeling and extended the general cautions of earlier guidelines, urging that only candidate treatment interaction terms with strong pre-existing evidence for HTE be included. It recommended use of statistical methods that penalize or “shrink” coefficient estimates to correct for over-fitting and urged caution in interpreting findings. The Statement also recognized the emergence and potential importance of newer data-driven machine-learning approaches to effect modeling for exploring more complex multi-level interactions among multiple variables but suggested that this field is not yet mature enough to justify specific recommendations or approaches.
This review evaluates the Statement’s impact following the standard methodology for scoping reviews (20). We review reports that have appeared since its publication, cited it and presented predictive models of potential HTE using RCT data. We assess consistency of analyses with the Statement’s Consensus Criteria (Supplement, Boxes A, B, D) and determine whether authors claimed that HTE was present, on either relative or absolute scales. We adapted criteria of the Instrument to assess Credibility of Effect Modification Analyses (ICEMAN) (21) to determine whether claims were credible and, if so, whether the HTE appeared to be clinically important, which the Statement defined as variation across patients in the treatment effect sufficient to span clinically defined decision thresholds, supporting differing treatment recommendations for patient subgroups.
METHODS
Identification of Reports for Inclusion
Using the Cited By functions in PubMed, Google Scholar, Web of Science and the SCOPUS database, we sought reports appearing after the Statement’s publication (January 7, 2020) through June 5, 2023 that presented analyses or re-analyses of data from RCTs using multivariable predictive modeling to identify HTE. Restricting the search to articles citing the Statement allows the assumption that authors were aware of its concepts and recommendations. The sources other than PubMed allowed inclusion of non-peer-reviewed reports posted on pre-print archives as well as dissertations posted on institutional websites.
A total of 211 citations were identified (Figure 1). Fifty-eight (22–79) involved analysis or re-analysis of data from one or more RCTs. In three instances, we combined two publications from the same authors and trial(s) (35–40). Thirteen reports (22–34) were excluded because they did not present a predictive model as defined by the Statement. Reasons for exclusion are presented in Supplement Table S1.
Flow Diagram for identification and screening of all reports citing the PATH Statement and for exclusion of reports not meeting study criteria for presenting a predictive model of individual treatment effects from RCT data. Abbreviations: RCT: randomized controlled trial; IPDMA: independent patient data meta-analysis.
Review of Predictive Model Reports
Variables collected during review and coding instructions are presented in Supplemental Tables S2 and S3, respectively. Features describing source RCT(s) were collected by the lead author (JS)., including 4 features the Statement suggested make risk modeling particularly likely to be of value for identifying clinically important HTE (Box A and Table S4 in Supplement). All aspects of analyses and findings for HTE were doubly reviewed by the lead author and one co-author (BF,DK,CM). An initial “learning” set of six reports was reviewed, discussed and resolved by all study co-authors. Thereafter, co-reviewers discussed and resolved initial disagreements.
Review determined whether a report used risk modeling, effect modeling or both. Effect models were further classified into those using regression methods (e.g., ordinary least squares, logistic, proportional hazards) and those that used more flexible, non-parametric data-driven machine-learning algorithms (e.g., 80-83).
Consistency with PATH Statement Criteria and Considerations
The Statement offered 10 guidance criteria for risk modeling (Supplement, Box B). The first recommended that risk modeling be conducted whenever an RCT had a positive overall result. Seven additional criteria were assessed, including using an external risk score (if available), including both treatment arms rather than only the control arm if developing the risk model internally; pre-specifying the analytic plan, including cut points for subgrouping of risk scores, before analyses; reporting risk model performance metrics (i.e., discrimination and calibration) when applied to the RCT population; presenting risk score distributions separately by treatment arm; reporting both absolute and relative effect sizes when reporting risk-stratified treatment effects; and reporting important adverse treatment effects by risk stratum when present. One remaining criterion could not be assessed at review and one was not a clearcut recommendation (see Supplement, Box B). For effect models, consistency with four Statement consensus criteria (Supplement, Box D) was recorded. These included limiting model covariates to those with strong prior evidence for effect modification; taking steps to reduce risks of model overfitting by applying penalization/regularization procedures to model coefficients (e.g., least absolute shrinkage and election operator (LASSO) (84), penalized ridge regression (85), elastic net regularization (86)) and/or using internal cross-validation; validating final model performance in a dataset external to the population in which it was developed (credit was given for using either an entirely distinct RCT dataset or a non-random subset of the original population selected before analyses on the basis of either geography (e.g., trial sites) or time of enrollment); and not relying solely on metrics intended for evaluating risk prediction when evaluating the performance of treatment effect models. In addition, we noted whether authors evaluated model performance in terms of predicting patient-specific treatment effects, including use of recently developed performance metrics for this purpose (e.g., 87-89).
Assessment of Credible and Clinically Important HTE
For reports that claimed to have identified HTE on either the absolute or relative scale, we assessed the credibility of HTE by adapting the ICEMAN criteria for RCTs (21,90). Although these criteria were developed for assessing credibility of findings from one-at-a-time subgroup analyses of interactions, three of the five apply readily to multivariable predictive modeling. These include 1) whether or not the number of interactions tested is small (three or fewer); 2) whether interactions tested are limited to covariates for which prior evidence of possible effect modification exists; and 3) whether arbitrary or data-driven cut-points are avoided in analyzing possible treatment interactions with continuous covariates.
Each ICEMAN criterion (90) rates compliance from 1 to 4 (definitely not, probably not, probably and definitely compliant). If all criteria are scored as probably or definitely compliant, credibility of HTE is rated as “high”. If at least 2 criteria are scored as definitely not compliant or if all 3 are scored as probably not compliant or worse, credibility is rated as “very low” or “low”, respectively. Remaining reports are rated as either “low” or “moderate”, using reviewer discretion with guidance from the ICEMAN manual considering the quality of the methods employed and whether statistical tests, when present, supported a hypothesis of HTE. High or moderate ratings were classified as credible HTE for this review.
Two ICEMAN criteria were not readily applicable to predictive modeling. The criterion that a statistical test for interaction be performed and highly significant, was not always applicable. In risk modeling, important HTE could be identified on the absolute scale even if testing for interaction on the relative scale was null or not performed. In effect modeling, multiple possible interactions were usually tested and results for specific interactions were often not reported.
When present, results of statistical tests for interaction or overall HTE were considered and may have weighed in differentiating between low and moderate credibility. The fifth criterion, that authors pre-specify direction of the interaction, was not considered feasible in predictive modeling, given the potentially complex interactions of multiple covariates with each other and with treatment.
Reviewers assessed reports of credible HTE for clinical importance by determining whether observed differences in size and direction of absolute treatment effect between subgroups supported different treatment recommendations. An additional consideration was whether findings for all outcomes studied, including adverse effects of treatment, were consistent in identifying preferred treatments, or whether they conflicted.
Results
General Description
The 42 reports (35–79) included 35 peer-reviewed publications, 4 postings on pre-print archives, and 3 dissertations. Five appeared in 2020, 11 in 2021, 13 in 2022 and 13 in the first five months of 2023. Among the 42, 25 were re-analyses of single RCTs, 14 were individual patient data meta-analyses (IPDMAs) of two or more RCTs, and three were initial reports from single RCTs that included HTE analyses. Forty-one reports examined HTE for a clinical treatment and one (54) evaluated behavioral interventions to boost educational performance among university students. A total of 19 risk models and 26 effect models were reported, with three reports presenting both risk and effect models (49,73,75).
Reviewer Agreement
After excluding six reports (presenting six effect models and one risk model) used for training reviewers, initial between-reviewer disagreement rates for doubly-reviewed items in 36 reports ranged from 0 to 47%, with an overall average of 17.4% (details, Supplement Table S2). Disagreements were resolved with discussion. Items with higher levels of initial disagreement included both the credibility and clinical importance of claimed HTE, especially for risk models.
Risk Models
Risk modeling appeared with increasing frequency over time, with six appearing during 2020-21 and an additional 13 found in 2022 and the first five months of 2023. Consistency with Statement criteria (Table 1) was above 65% for all but three criteria. Slightly fewer than half of reports with positive findings included a risk model. External prediction models were employed in only eight of 19 analyses, possibly because an appropriate validated model was not available; and risk score distributions were presented separately by trial arm in only 12 reports.
Study authors claimed findings of HTE in 15 risk modeling reports (Figure 2). For six, (49,60,64,71,75,77) heterogeneity was found on the absolute but not the relative scale (i.e., benefit magnification) and for nine (36,38,43,48,50,54,58,68,73), heterogeneity was also found on the relative scale (i.e., relative treatment effects also varied across levels of baseline risk). The four remaining reports (56,57,61,76) did not find clear overall treatment effects and none claimed HTE on either scale. Risk models generally scored highly on the three ICEMAN criteria for credibility of HTE. They always involved a single effect modifier (the baseline risk score); those finding benefit magnification had strong prior theoretical and empirical reasons (18) to expect such HTE; and all models used pre-specified rather than data-driven cut-points to define risk score subgroups.
Adjudicated results of review of all eligible reports for type of predictive modeling (risk or effect), for claims by authors of heterogeneity of treatment effects (HTE), for credibility of HTE (using adapted ICEMAN criteria), and for clinical importance of HTE found to be credible.
We scored all risk models finding benefit magnification and eight of nine that found HTE on a relative scale as credible HTE. Four of these eight found that relative as well as absolute treatment effects were greater in individuals at higher risk for experiencing trial outcomes. (38,43,50,68) In the remaining four (36,48,58,73), those at greatest risk showed no evidence of benefit. In two (36,58), both relative and absolute effects of treatment were greatest for individuals in the middle of the risk distribution, and in two (48,73), only those at lower risk experienced a treatment benefit. In the single report that was not found to be credible, multiple treatment-by-risk interactions were tested across a variety of outcomes and findings were inconsistent. (54)
Effect Models
The 26 effect model analyses used diverse types of models and analytic strategies. Nine analyses used regression methods; the remainder employed various data-driven machine-learning approaches. Machine-learning approaches became more frequent over time (5/11 reports in 2020-21 vs. 12/15 reports in 2022-23).
Few effect model reports (41,44,59) restricted analyses to potential effect modifiers with strong prior evidence for effect modification (Table 3). The majority explored many candidate effect modifiers with little prior evidence. Most used recommended steps to reduce risks of over-fitting, including coefficient shrinkage methods and internal cross-validation. Six (40,41,44,59,69,79) applied effect model findings to external datasets for validation. Only four studies (52,75,78,79) used performance metrics designed for risk prediction without also reporting performance for predicting treatment effects. In all, eight effect modeling reports (41,42,44,49,59,62,69,74) specifically assessed model performance for predicting individual treatment effects.
Authors claimed HTE in 20 effect modeling reports (Figure 2). Most failed to meet the adapted ICEMAN credibility criteria because they explored many variables with little prior evidence for effect modification. Many also reported data-driven rather than pre-specified cut points for continuous predictors. However, five of the 20 were judged to present credible HTE. These included three (41,44,59) that restricted analyses to small numbers of pre-specified effect modifiers with strong prior evidence. These, along with two other reports, (40,97) also validated model predictions of individual treatment effect in external RCT datasets. Because these validation analyses tested only effect modifiers with prior evidence and pre-specified cut points (i.e., the evidence and cut points from derivation analyses), they scored highly for credibility of HTE. Four of these used regression models; (40,41,44,59) the fifth (69) used a causal forest machine-learning algorithm. (82)
Assessment for Clinically Important HTE
Reviewers judged findings from 14 reports with credible HTE to also be clinically important (Figure 2, Table 3), including 10 of 14 risk modeling analyses and four of five effect modeling reports. Table 3 gives the decision thresholds for important subgroup differences in treatment recommendations. Reasons for concluding that credible HTE was not clinically important included lack of clarity in presentation of findings, (75) failure to identify a threshold for differing treatment choices, (64) conflicting findings across outcomes, (77) failure to add clinical value to previous risk-based selection strategies, (68) and concurrence with authors on the need for additional investigation, possibly testing additional potential effect modifiers. (44)
Discussion
Popular approaches to evidence-based medicine have encouraged reliance on average treatment effects from RCTs to support decision-making by individual patients, (91) despite appreciation of the limitations of this approach. Herein, we reviewed early efforts in applying predictive modeling within RCTs to deliver more patient-centered evidence for decision making. During the first three years, five months following publication of the PATH Statement, we identified 42 reports that cited the Statement and presented predictive modeling across a range of clinical conditions and types of interventions. Fully one third of these reports found HTE that met adapted ICEMAN criteria for credibility and the PATH definition for clinical importance, providing strong evidence that recommendations should vary among patients facing the same treatment choices.
The Statement recommended that risk modeling be conducted when RCTs report positive overall results. Risk modeling was much more likely than effect modeling to produce findings that met criteria for credible HTE because risk modeling tests only a single effect modifier, one with strong prior evidence and a theoretical rationale for effect modification. Nevertheless, fewer than half of reports from positive RCTs presented risk modeling. Many effect modeling reports had features the Statement indicated would make risk-modeling a promising place to begin (Supplement, Box A, Table S4), suggesting that a simpler approach could have been more informative.
Contrary to assumptions that relative treatments effects are constant across levels of baseline risk, the risk modeling studies reviewed here more often found that relative as well as treatment effects varied importantly. In one report, (73) persons at opposite ends of the predicted risk range experienced opposite effects of treatment. In others, relative effects were greater in or completely confined to persons at higher (38,43,50,68) or lower ends (48) of predicted risk, and in two, (36,58) maximal benefit was found for those in the mid-range. This U-shaped, or “sweet spot,” pattern (36) has also been observed elsewhere. (92)
Explanations for such variation in relative treatment effects across baseline risk are not always obvious. Risk scores may incorporate traits that are both strong predictors of study outcomes and also potent relative treatment effect modifiers, either directly or as proxies for unmeasured attributes. In a study of therapeutic-dose heparin vs. usual care pharmacologic thromboprophylaxis for patients hospitalized with COVID-19, (73) a better initial respiratory status was the most potent predictor of good clinical outcomes. When included in the risk model, respiratory status dominated the risk score. It had also proved to be a strong modifier of treatment effect in earlier subgroup analyses. Persons with low risk scores (better baseline respiratory status) were then found to be the only subgroup that benefited from heparin treatment. In three RCTs (36,48,58) where incidence of study outcomes was particularly high (range 27-61%), no benefit was observed for those in the highest stratum of predicted risk. For these extremely high-risk individuals, models may have captured attributes whose presence reflected irreversible disease or competing causes of the outcome that would make treatment futile. These observations of risk – treatment interactions and others noted elsewhere (93) demonstrate that assumptions of simple benefit magnification are not well-founded and should be tested routinely.
The number of effect modeling analyses and the increasing use of exploratory machine-learning methods over time suggest continuing enthusiasm for individualizing treatment recommendations beyond risk stratification. Several authors motivated their approaches by pointing to limitations of risk modeling. (47,65,77) Although risk scores can create patient subgroups well-matched on risk, subgroup members may be heterogeneous for the specific characteristics that contributed to their risk scores and therefore potentially heterogeneous in terms of their treatment response. (65)
Five effect models did find credible HTE, either because they adhered to Statement recommendations to include only effect modifiers with strong prior evidence or because they took the added step of validating their HTE findings in external populations. Many additional reports provided evidence suggesting multivariable HTE that now deserves such external validation.
Findings from several effect modeling reports revealed the uncertainty that can remain after initial findings suggest HTE and reinforce the necessity of validating such findings in other populations before they are accepted as credible. Two reports (47,74) explored data from the SPRINT and Action to Control Cardiovascular Risk in Diabetes (ACCORD) trials using causal forest algorithms. One (74) found evidence of HTE, the other did not. In two reports (52,70) from a trial of dabigatran vs. warfarin for stroke prevention in atrial fibrillation, one model (52) suggested interactions of three covariates with treatment choice and significant HTE; the second report, (70) using four machine-learning algorithms applied to the same RCT data, found no evidence for HTE. Sinha et al (51) applied four widely used machine-learning algorithms to RCTs of treatments for acute respiratory distress syndrome finding inconsistent evidence for HTE, not only between algorithms but within algorithms when random initiation seeds were altered. Both authors (51,70) acknowledged the challenges from false signals of effect modification in exploratory analyses.
As the Statement suggests, additional steps will often be needed before implementing HTE findings from predictive models into clinical practice. Prediction models used in risk modeling, especially those developed internally, may not yet generalize well to clinical populations with differing risk distributions. The performance of all models for predicting treatment effects may need further validation in differing populations. Ultimately, it will be critical to demonstrate that clinical outcomes improve when treatment recommendations are personalized using predictive modeling.
There are learnings from this review for funding, conducting, and publishing clinical research. The value of external validation, especially for effect models, points to the fundamental importance of sharing data from completed RCTs. In the absence of additional appropriate RCTs, large well-characterized observational cohorts with treatment and covariate data could also be valuable both for validation (94) as well as for developing and validating new, representative risk prediction models. In planning and funding new RCTs, the potential existence of appropriate external risk models should be considered and data collection should include baseline data necessary for classifying individual risk of study outcomes.
In recent years, guidelines for reporting clinical trials findings have recognized the importance of presenting absolute as well as relative measures of the overall treatment benefit or harm because of the greater relevance of absolute measures to clinical decision-making. (95–97) We believe the present review supports consideration of an additional editorial requirement that initial reports of RCTs or IPDMA’s routinely present treatment effects in relation to baseline risk when overall results are positive.
Limitations
There is inherent subjectivity in assessing credibility and importance of HTE. The close association of two authors (DK, JS) with production of the PATH statement should be kept in mind.
Conclusions
The PATH statement appears to be influencing research practice. Effect modeling holds promise for predicting individualized treatment effects but the need for external validation is a constraint. Risk modeling provides a more straightforward initial approach when overall trial findings are positive and often identifies clinically important HTE.
Data Availability Statement
All data produced in the present study are available upon reasonable request to the authors.
Funding
Drs. Selby and Maas and Mr. Fireman report no funding related to work performed on this publication. Dr. Kent was funded by a National Institutes of Health (NIH)/National Center for Advancing Translational Sciences (NCATS) grant (UM1TR004398-01). Dr. Selby previously served as the Executive Director of the Patient-Centered Outcomes Research Institute (PCORI). The views and findings presented in this publication are solely the responsibility of the authors and are not presented on behalf of or as the views of PCORI.
Acknowledgments
The authors gratefully acknowledge Harold Sox, MD, Department of Medicine and The Dartmouth Institute (emeritus), Geisel School of Medicine at Dartmouth, Hanover NH, for careful review and helpful suggestions on earlier drafts of the manuscript; Jinny G. Park, MPH, Tufts Predictive Analytics and Comparative Effectiveness Center, Tufts University School of Medicine, Boston MA, for conducting all literature database searches; and Ivan Rivera, MIS, Division of Research, Kaiser Permanente Northern CA, for retrieving reprints and supplemental materials of study citations.
References
- 1.↵
- 2.
- 3.
- 4.
- 5.
- 6.↵
- 7.↵
- 8.
- 9.
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.↵
- 35.↵
- 36.↵
- 37.
- 38.↵
- 39.
- 40.↵
- 41.↵
- 42.↵
- 43.↵
- 44.↵
- 45.
- 46.
- 47.↵
- 48.↵
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.
- 54.↵
- 55.
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.
- 64.↵
- 65.↵
- 66.
- 67.
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.↵
- 78.↵
- 79.↵
- 80.
- 81.
- 82.↵
- 83.
- 84.↵
- 85.↵
- 86.↵
- 87.
- 88.
- 89.
- 90.↵
- 91.↵
- 92.↵
- 93.↵
- 94.↵
- 95.↵
- 96.
- 97.↵