The impact of blinding on trial results: A systematic review and meta-analysis

Tyler Pitre, Sarah Kirsh, Tanvir Jassal, Mason Anderson, Adelia Padoan, Alexander Xiang, Jasmine Mah, Dena Zeraatkar
doi: https://doi.org/10.1101/2023.03.05.23286821

Abstract

Background Blinding—the concealment of the arm to which participants have been randomized—is an important consideration for assessing risk of bias of randomized trials. A growing body of evidence has, however, yielded inconsistent results on whether trials without blinding produce biased findings.

Objective To conduct a systematic review and meta-analysis of the evidence addressing whether trials with and without blinding produce different results.

Methods We searched MEDLINE, EMBASE, Cochrane Reviews, JBI EBP, and Web of Science, from inception to May 2022, for studies comparing the results of trials with and without blinding. Pairs of reviewers, working independently and in duplicate, reviewed search results for eligible studies and extracted data.

We pooled the results of studies comparing trials with and without blinding of patients, healthcare providers/investigators, and outcome assessors/adjudicators using frequentist random-effects meta-analyses. We coded study results such that a ratio of odds ratio (ROR) < 1 and difference in standardized mean difference (dSMD) < 0 indicate that trials without blinding overestimate treatment effects.

Results We identified 47 eligible studies. For dichotomous outcomes, we found low certainty evidence that trials without blinding of patients and healthcare providers, outcome assessors/adjudicators, and patients may slightly overestimate treatment effects. For continuous outcomes, we found low certainty evidence that trials without blinding of outcome assessors/adjudicators and patients may slightly overestimate treatment effects.

Conclusion Our systematic review and meta-analysis suggests that blinding may influence trial results in select situations—albeit the findings are of low certainty and the magnitude of effect is modest. In the absence of high certainty evidence suggesting that trials with and without blinding produce similar results, investigators should be cautious about interpreting the results of trials without blinding.

Background

Randomized trials represent the optimal design for assessing the effectiveness of interventions. Randomization ensures—more or less—that important known and unknown prognostic factors are balanced between arms such that any observed differences between trial arms can be attributed to the intervention under study. For decades, blinding—the concealment of the arm to which participants have been randomized—has been an important consideration for assessing risk of bias of randomized trials (1, 2).

Blinding of participants and healthcare providers reduces opportunity for differences in care (co-interventions)—referred to as performance bias—whereas blinding of outcome assessors/adjudicators reduces differences in the measurement and adjudication of outcomes between trial arms—called ascertainment bias (3, 4). Preconceived notions about the efficacy and safety of interventions, even subconsciously, may theoretically impact decisions about co-interventions and outcome expectancy (5, 6).

Though there are clear advantages to blinding, blinding requires significant resources, increases the operational complexity of trials, and may not always be feasible. Funding and research organizations have invested in several initiatives to streamline and simplify clinical trials, and foregoing blinding is a logical step in this process (7-9). It is unclear, however, the extent to which open-label trials may produce biased findings.

Meta-epidemiologic studies (studies that analyze the results of previous studies to address how methodological characteristics of the studies influence their results) have yielded inconsistent findings (10-15). In this systematic review and meta-analysis, we summarize evidence addressing the effects of blinding on trial results.

Methods

We registered a protocol on Open Science Framework (https://osf.io/qenky) on June 9, 2022. We report our systematic review using the PRISMA reporting guidelines and guidelines for reporting meta-epidemiologic methods studies (16, 17).

Search strategy

In consultation with an experienced research librarian, we searched MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews, JBI EBP, and Web of Science, from inception to May 2022, for studies comparing the results of trials with and without blinding. We supplemented our search by reviewing the reference lists of eligible studies. We did not restrict the search by date or language of publication Supplement 1 presents our search strategy.

Screening and study eligibility

Following training and calibration exercises to ensure sufficient agreement, pairs of reviewers, working independently and in duplicate, reviewed titles and abstracts of search records and subsequently the full-texts of records deemed potentially eligible at the title and abstract screening stage. Reviewers resolved discrepancies by discussion, or, when necessary, by adjudication by a third party.

We included meta-epidemiologic studies that compared the results of trials in which either patients, healthcare providers, outcome assessors/adjudicators, analysts, or manuscript writers were blinded to trials in which either of these parties were not blinded. We included studies that investigated the effects of blinding for trials addressing the effectiveness or safety of a treatment or management strategy and excluded studies that investigated the effects of blinding in diagnostic studies. We did not restrict eligibility based on the population, intervention, or outcomes or by language or date of publication.

We excluded single systematic reviews and meta-analyses that presented subgroup or sensitivity analyses addressing blinding because single reviews are unlikely to provide precise estimates and because such estimates have limited generalizability. We excluded conference abstracts and studies addressing the effects of blinding in animal studies or in vitro studies.

Data extraction and risk of bias assessments

Pairs of reviewers, working independently and in duplicate, collected data and assessed the risk of bias of eligible studies.

We collected data on study characteristics (clinical condition, number of reviews/meta-analyses and trials included, type of outcome investigated), blinding characteristics (blinding status of participants, healthcare providers/investigators, outcome assessors/adjudicators, or other parties), and the results for studies with and without blinding from the least and most adjusted estimate. For studies in which multiple adjusted estimates were presented, based on previous evidence that the effects of blinding may be confounded by allocation concealment, we preferentially extracted the estimate that was adjusted for allocation concealment (10, 18).

When reported, we extracted results separately for objective and subjective outcomes. We considered outcomes objective if they were established laboratory measures and other outcomes not subject to interpretation (e.g., all-cause mortality, hospitalization) and subjective if outcomes were patient-reported or subject to interpretation from healthcare providers or investigators (e.g., adverse events).

To our knowledge, there are no risk of bias tools for meta-epidemiologic studies. We assessed the risk of bias of eligible studies using ad-hoc criteria: selection bias (association between trial characteristics and outcomes in trials selected for analysis), confounding (distortion of the effect of blinding on study results due to the correlation of the study characteristic of interest with other study characteristics that influence the results), bias in classification of trial characteristics (misclassification of blinding status), bias in measurement of outcomes (errors in extraction or reporting of results that may be related to blinding), and selective reporting bias (selective reporting of study results based on the magnitude, statistical significance, or direction of findings). To make judgements about confounding bias, we considered whether the study adjusted for allocation concealment, which has been shown to influence trial results (10, 18), and to make judgements about bias in classification of trial characteristics and measurement of outcomes, we considered the criteria used by the study to ascertain blinding status and whether trial characteristics and outcome data were collected independently and in duplicate to reduce the potential for errors (15, 19).

We rated risk of bias criteria as either low, probably low, probably high, or high risk of bias and considered results at low risk of bias overall if all domains were rated as either low or probably low risk of bias.

Reviewers resolved discrepancies by discussion or, when necessary, by third-party adjudication.

Data synthesis and analysis

We report categorical trial characteristics as proportions and percentages and continuous characteristics as medians and interquartile ranges (IQRs). We synthesize the results of studies that compared trials with and without blinding of patients and healthcare providers/investigators (corresponding to ‘double-blind’) (20), outcome assessors/adjudicators, patients, healthcare providers/investigators, statisticians, and manuscript writers.

We performed a frequentist random-effects meta-analysis with the restricted maximum likelihood (REML) heterogeneity estimator to pool the results of studies. For studies that addressed dichotomous outcomes, we present results as ratios of odds ratios (ROR) and for studies that addressed continuous outcomes, we present differences in standardized mean differences (dSMD) with the associated confidence intervals. We recoded the data such that ROR < 1 and dSMD < 0 correspond to an exaggerated effect in trials without blinding compared to trials with blinding.

Where possible, we meta-analyzed the least and most-adjusted results from primary studies. For studies that converted dichotomous outcome measures to continuous outcome measures, or vice versa, we combined them in analyses with the measure to which they were converted. When studies reported multiple outcomes, we synthesized results for the outcome reported by the greatest number of trials.

We summarize heterogeneity using the I2 statistic. We considered heterogeneity of 0%–40% as potentially unimportant, 30%–60% as moderate, 50%–90% as substantial and 75%–100% as critical (21). For analyses with 10 or more studies, we assessed for publication bias using funnel plots and the Egger’s test (22).

We performed all analyses in R (version 4.03, R Foundation for Statistical Computing), using the meta and metafor packages. Open Science Framework (https://osf.io/qb23h/) presents the data and code to generate the results in this study.

Where meta-analysis is not possible, we report the range and distribution of results from primary studies (23).

Assessment of the certainty of evidence and interpretation of study results

We use a modified GRADE approach to assess the certainty of evidence (24). While GRADE has not been developed to assess the certainty of meta-epidemiologic evidence, we anticipate that GRADE criteria are still relevant to methodologic questions and meta-epidemiologic studies.

The GRADE approach rates the evidence as either high, moderate, low, or very low certainty based on considerations of risk of bias (study limitations), inconsistency (heterogeneity in study results), indirectness (differences between the questions addressed in studies and the question of interest), publication bias (the tendency for studies with statistically significant results or positive results to be published, published faster, or published in journals with higher visibility), and imprecision (random error).

High certainty evidence indicates situations in which we are confident that the estimated effect represents the true effect, and low or very low certainty evidence indicates situations in which the estimated effect may be substantially different from the true effect.

We considered an ROR of 1.10 or 0.90 and a dSMD of 0.1 to indicate minimally important effects of blinding on trial results, which was informed by our experiences in differences in effect estimates that may influence decision-making (25, 26). Such an effect corresponds to a 10% difference in the results of blinded and unblinded trials for dichotomous outcomes and a quarter standard deviation difference for continuous outcomes.

We report our results using GRADE guidance (27), which involves describing the effect of an intervention based on the certainty of evidence (i.e., high certainty evidence the exposure is effective, moderate certainty evidence the exposure is probably effective, low certainty evidence the exposure may be effective and very low certainty evidence the effect of the exposure is unclear).

Results

Search results

Our search yielded a total of 6,623 unique records, of which 47 were eligible to be included (18, 28-73). Figure 1 presents the results of the search and screening.

Figure

Study characteristics

Table 1 and Supplement 2 present study characteristics. Typical studies sampled meta-analyses and compared the results of trials with and without blinding using either frequentist meta-regression or Bayesian hierarchical models (74, 75). Studies that used frequentist meta-regressions compared the results of trials with and without blinding within each meta-analysis using meta-regression and then pooled the results of coefficients corresponding to blinding across meta-analyses.

View this table:
Table 1: Table of study characteristics

A minority of studies reported on studies that made within trial comparisons (37, 38, 50-52). Such studies included trials in which patients were randomized to blinded or open-label substudies or studies that performed both blinded and open-label assessment of outcomes.

Studies included a median of 26 meta-analyses and 150 trials.

Risk of bias assessments

Figure 2 presents the risk of bias of studies that were included in the analysis investigating the effects of blinding of patients and healthcare providers/investigators and Supplement 3 presents risk of bias judgements for all studies.

Figure

All studies were rated at high risk of bias, primarily due to confounding and selective reporting bias. Studies seldom adjusted for allocation concealment and registered or published their protocols.

Blinding of patients and healthcare providers/investigators

Twenty-four studies addressed the effects of blinding of patients and healthcare providers/investigators on trial results (18, 28, 29, 32, 36, 39, 40, 43-46, 49, 50, 54-56, 58-60, 64-66, 69). Table 2 presents the results of our analyses.

View this table:
Table 2: Results

For dichotomous outcomes, we meta-analyzed seven studies, including 29546 meta-analyses and 3,732 trials, and found evidence that trials without blinding of patients and healthcare providers may overestimate dichotomous outcomes (low certainty due to risk of bias and imprecision) (Supplement 4). When we restricted our analysis to studies that reported adjusted effect estimates, we found a smaller difference between the results of trials with and without blinding.

For continuous outcomes, we meta-analyzed four studies, including 45 meta-analyses and 696 trials, and found that trials without blinding may not produce different results compared to trials with blinding (low certainty due to risk of bias and imprecision).

Blinding of outcome assessors

Twenty-three studies addressed the effects of blinding of outcome assessors/adjudicators (30, 33-35, 37-39, 41, 42, 44, 51-53, 55, 57, 58, 61-63, 71-73).

For dichotomous outcomes, we meta-analyzed 12 studies, including 119 meta-analyses and 1,579 trials, and for continuous outcomes, we meta-analyzed 7 studies, including 78 meta-analyses and 1,426 trials, and found evidence that trials without blinding of outcome assessors may overestimate results (low certainty due to risk of bias and imprecision). We found similar results when we restricted our analyses to subjective outcomes. For dichotomous outcomes, the funnel plot showed some evidence of asymmetry but the Egger test for funnel plot asymmetry was not statistically significant (p=0.38).

Blinding of patients

Twelve studies addressed the effects of blinding of patients (31, 33-35, 41, 44, 47, 48, 55, 62, 71, 73).

For dichotomous outcomes, we meta-analyzed four studies, including 78 meta-analyses and 1,106 trials and for continuous outcomes, we meta-analyzed seven studies, including 50 meta-analyses and 1,114 trials. We found evidence that trials without blinding of patients may overestimate treatment effects (low certainty due to risk of bias and imprecision). The results were similar when we restricted the analysis to subjective outcomes.

Blinding of healthcare providers/investigators

Nine studies addressed the effects of blinding of healthcare providers (33-35, 41, 44, 55, 63, 71, 73).

We meta-analyzed four studies reporting on dichotomous outcomes, including 54 meta-analyses and 902 trials, and four studies reporting on continuous outcomes, including 18 meta-analyses and 859 trials. For both dichotomous and continuous outcomes, we found that trials without blinding of healthcare providers may not overestimate treatment effects (low certainty due to risk of bias and imprecision).

Statisticians and manuscript writers

Two studies addressed the effects of blinding of statisticians, one of which did not find sufficient data, and none of the studies addressed the effects of blinding of manuscript writers (34, 55).

The one study with data addressing the effects of blinding of statisticians found that blinding of statisticians may have no effect on trial results but that the results are very uncertain (very low certainty due to risk of bias, imprecision, and indirectness).

Discussion

Main findings

Our systematic review presents evidence that trials without blinding may overestimate treatment effects—albeit the findings are of low certainty and the magnitude of effect is modest.

We found inconsistencies in the effects of blinding. While we found evidence, for example, that trials without blinding of patients and healthcare providers may overestimate dichotomous outcomes, we did not find evidence for such an effect for continuous outcomes. Similarly, we also found inconsistencies across studies. The MetaBLIND study, for example, did not find evidence that trials with and without blinding produce different results—contrary to several other studies of similar magnitude included in our review (40, 41, 59, 69). Further, studies that compared the effects of blinding within rather than across trials showed blinding to have larger effects (38, 50-52).

Theoretical considerations about blinding may explain some of the observed inconsistencies. For example, we anticipate several factors that may vary based on the question being investigated and that may influence the effect of blinding: investigators’ and patients’ preconceived notions about the effectiveness of the intervention; whether there are co-interventions that may influence the outcome of interest; the magnitude of effect of the co-interventions on the outcome of interest; the potential for imbalances in the administration of co-interventions; and the degree of interpretation involved in assessment of outcomes (18, 76).

An open-label trial addressing a condition for which there are no effective co-interventions or a trial in which healthcare providers and patients either do not have access to co-interventions or in which the administration of co-interventions is strictly regimented across arms is unlikely to be at risk of performance bias. Similarly, an open-label trial addressing outcomes that are objective and not subject to interpretation is unlikely to be at risk of ascertainment bias.

In the absence of high certainty evidence suggesting that trials with and without blinding produce similar results, investigators should still be cautious about open-label trials.

Strengths and limitations

The strengths of our systematic review include a priori methods, a systematic search for eligible studies, duplicate screening and data extraction, and assessment of the role of blinding across a range of clinical areas. Our review improves on previous reviews by performing an assessment of risk of bias and the certainty of evidence and including several recently published studies not included in previous reviews (10, 15, 18, 19, 33, 40, 44).

Our results are limited by the risk of bias of primary studies. Nearly all studies were rated at high risk of bias, primarily due to potential for confounding and selective reporting bias. There is also potential for misclassification of blinding status. A substantial proportion of trials, for example, did not explicitly report the blinding status for all relevant parties (77). When trials did not explicitly report blinding status, studies either used validated guidance to ascertain blinding status or grouped unblinded studies with studies with unclear blinding (78). Even when blinding status was reported, the reporting may not have been accurate (79). Several studies used the risk of bias judgements from meta-analyses as a surrogate for blinding status, which may also contain errors or may reflect reviewers’ judgements regarding the potential impact of blinding or lack thereof on results rather than whether the trial was blinded. This potential for misclassification means that the role of blinding may be more important than our estimates suggest.

Our results are further limited by the potential inconsistency in the direction of the effect of blinding. While we anticipate that trials without blinding overestimate the beneficial effects of treatments, it is possible that the direction of bias caused by lack of blinding may not be consistent. That is, in certain circumstances open-label trials may overestimate effects and in other circumstances they may underestimate effects.

All of our analyses produced results that were imprecise. Further, our estimated effects for blinding were generally small and evidence users may consider such effects too small to influence decision-making, though most medical treatments show modest effects and hence even small biases may impact decision-making (80, 81).

Trials addressing questions that are sensitive to performance bias and ascertainment bias may be more likely to be blinded, thus reducing observed differences between trials with and without blinding.

There may be overlap between trials and meta-analyses included in meta-epidemiologic studies. An inspection of the search strategies and clinical areas addressed by the included studies, however, suggests that this overlap is likely minimal. To reduce the potential for double-counting, when meta-epidemiologic studies reported more than one estimate that would be eligible to be included in each analysis, we only included the estimate from the analysis that included the greatest number of trials.

While we tried to pool studies addressing the effects of blinding different individuals together (e.g., patients, healthcare providers), the ways in which types of blinding were defined and operationalized across studies may have been slightly different.

It is possible that blinding may be compromised during a trial (e.g., due to obvious adverse events, poorly matched placebos, or large therapeutic effects), which may have attenuated the difference between trials with and without blinding.

Implications

The results of our systematic review have implications for evidence users who are appraising and applying evidence from open-label trials, for clinical trialists considering the potential benefits and burdens of implementing blinding in their clinical trials, and for global efforts to streamline and simplify clinical trial methods (7-9).

Results from the MetaBLIND study encouraged a critical evaluation of the importance of blinding in clinical trials (41). The study, for example, led to a revision of the Cochrane-endorsed risk of bias tool (RoB 2.0) to include a more nuanced approach for assessing bias due to lack of blinding (76). Results from MetaBLIND, however, were imprecise and were consistent with lack of blinding resulting in both an over- and underestimation of treatment effects. Further, there is no evidence whether the new approach to risk of bias assessments, proposed by the RoB 2.0 tool, that places less emphasis on blinding is better correlated with trial results.

The imprecision in results across meta-epidemiologic studies may be driven by heterogeneity in the effects of blinding across trials. That is, in some situations trials without blinding may overestimate treatment effects. Our findings suggest that evidence users should be cognizant of the ways in which lack of blinding may bias trial results, that clinical trialists should be judicious in their decisions to forego blinding, and that evidence users should be cognizant of the ways in which lack of blinding may bias trial results.

Evidence users should be mindful that the inability to identify evidence of when and how blinding affects trial results does not mean that open-label designs are unbiased. In fact, our review shows evidence that trials without blinding may overestimate treatment effects.

The subject of blinding invites comparisons to judgements about the quality of evidence from non-randomized studies. Despite evidence that randomized trials and non-randomized studies produce consistent results in many situations (82-84), we typically consider non-randomized studies to produce lower certainty evidence compared to randomized trials because evidence users are unable to confidently distinguish between situations in which non-randomized studies produce trustworthy or untrustworthy results. Uncertainty about the effects of blinding may analogously demand always or almost always rating open-label studies at high risk of bias.

We show that we need higher certainty evidence to determine when and how blinding affects trial results. Initiatives such as Studies Within a Trial (SWATs)—research studies embedded within a host trial to assess alternative ways of delivering or organizing a particular trial process—can help address this need (85). While incorporating SWATs within trials to study how blinding affects results may increase the costs of trials, such initiatives may prove cost-effective in the long term by simplifying and streamlining future trials without compromising the trustworthiness of results.

Conclusion

Our systematic review and meta-analysis presents evidence that trials without blinding may overestimate treatment effects—albeit the findings are of low certainty and the magnitude of effect is modest. In the absence of high certainty evidence suggesting that trials with and without blinding produce similar results, investigators should be cautious about interpreting the results of trials without blinding.

Data Availability

All data produced in the present study are available upon reasonable request to the authors

Acknowledgements

None.

Footnotes

References

  1. 1.↵
    Schulz KF, Grimes DA. Blinding in randomised trials: hiding who got what. Lancet. 2002;359(9307):696–700.
    OpenUrlCrossRefPubMedWeb of Science
  2. 2.↵
    Day SJ, Altman DG. Statistics notes: blinding in clinical trials and other studies. Bmj. 2000;321(7259):504.
    OpenUrlFREE Full Text
  3. 3.↵
    Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
    OpenUrlFREE Full Text
  4. 4.↵
    Karanicolas PJ, Farrokhyar F, Bhandari M. Practical tips for surgical research: blinding: who, what, when, why, how? Can J Surg. 2010;53(5):345–8.
    OpenUrlPubMed
  5. 5.↵
    Nickerson RS. Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. Review of General Psychology. 1998;2(2):175–220.
    OpenUrlCrossRef
  6. 6.↵
    Rosenthal R. On the social psychology of the psychological experiment: the experimenter’s hypothesis as unintended determinant of experimental results. Am Sci. 1963;51:268–83.
    OpenUrlPubMedWeb of Science
  7. 7.↵
    Protas. 2022 [Available from: https://protas.co.uk/.
  8. 8.
    Corneli A, Hallinan Z, Hamre G, Perry B, Goldsack JC, Calvert SB, et al. The Clinical Trials Transformation Initiative: Methodology supporting the mission. Clin Trials. 2018;15(1_suppl):13–8.
    OpenUrl
  9. 9.↵
    Kelly D, Spreafico A, Siu LL. Increasing operational and scientific efficiency in clinical trials. Br J Cancer. 2020;123(8):1207–8.
    OpenUrl
  10. 10.↵
    Moustgaard H, Clayton GL, Jones HE, Boutron I, Jørgensen L, Laursen DRT, et al. Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study. Bmj. 2020;368:l6802.
    OpenUrlAbstract/FREE Full Text
  11. 11.
    Anthon CT, Granholm A, Perner A, Laake JH, Møller MH. No firm evidence that lack of blinding affects estimates of mortality in randomized clinical trials of intensive care interventions: a systematic review and meta-analysis. J Clin Epidemiol. 2018;100:71–81.
    OpenUrlPubMed
  12. 12.
    Baiardo Redaelli M, Belletti A, Monti G, Lembo R, Ortalda A, Landoni G, et al. The impact of non-blinding in critical care medicine trials. J Crit Care. 2018;48:414–7.
    OpenUrl
  13. 13.
    Martin GL, Trioux T, Gaudry S, Tubach F, Hajage D, Dechartres A. Association Between Lack of Blinding and Mortality Results in Critical Care Randomized Controlled Trials: A Meta-Epidemiological Study. Crit Care Med. 2021;49(10):1800–11.
    OpenUrl
  14. 14.
    Savovic J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, et al. Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study. Am J Epidemiol. 2018;187(5):1113–22.
    OpenUrlCrossRefPubMed
  15. 15.↵
    Page MJ, Higgins JP, Clayton G, Sterne JA, Hróbjartsson A, Savović J. Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies. PLoS One. 2016;11(7):e0159267.
    OpenUrlCrossRefPubMed
  16. 16.↵
    Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Rev Esp Cardiol (Engl Ed). 2021;74(9):790–9.
    OpenUrlCrossRefPubMed
  17. 17.↵
    Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42.
    OpenUrlCrossRefPubMed
  18. 18.↵
    Zeraatkar D, Pitre T, Diaz-Martinez JP, Chu D, Rochwerg B, Lamontagne F, et al. Effects of allocation concealment and blinding in trials addressing treatments for COVID-19: A methods study. medRxiv. 2022:2022.08.03.22278348.
  19. 19.↵
    Dechartres A, Trinquart L, Faber T, Ravaud P. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol. 2016;77:24–37.
    OpenUrlCrossRefPubMed
  20. 20.↵
    Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. Jama. 2001;285(15):2000–3.
    OpenUrlCrossRefPubMedWeb of Science
  21. 21.↵
    Cumpston M, Li T, Page MJ, Chandler J, Welch VA, Higgins JP, et al. Updated guidance for trusted systematic reviews: a new edition of the Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Database Syst Rev. 2019;10:Ed000142.
    OpenUrlCrossRefPubMed
  22. 22.↵
    Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. Bmj. 1997;315(7109):629–34.
    OpenUrlAbstract/FREE Full Text
  23. 23.↵
    Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. Bmj. 2020;368:l6890.
    OpenUrlFREE Full Text
  24. 24.↵
    Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. Bmj. 2008;336(7650):924–6.
    OpenUrlFREE Full Text
  25. 25.↵
    Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, et al. The GRADE Working Group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017;87:4–13.
    OpenUrlCrossRefPubMed
  26. 26.↵
    Zeng L, Brignardello-Petersen R, Hultcrantz M, Mustafa RA, Murad MH, Iorio A, et al. GRADE Guidance 34: update on rating imprecision using a minimally contextualized approach. J Clin Epidemiol. 2022.
  27. 27.↵
    Santesso N, Glenton C, Dahm P, Garner P, Akl EA, Alper B, et al. GRADE guidelines 26: informative statements to communicate the findings of systematic reviews of interventions. J Clin Epidemiol. 2020;119:126–35.
    OpenUrlCrossRefPubMed
  28. 28.↵
    Ioannidis JPA, Cappelleri JC, Sacks HS, Lau J. The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Controlled Clinical Trials. 1997;18(5):431–44.
    OpenUrlCrossRefPubMedWeb of Science
  29. 29.↵
    Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Annals of Internal Medicine. 2001;135(11):982–9.
    OpenUrlCrossRefPubMedWeb of Science
  30. 30.↵
    Liu CJ, LaValley M, Latham NK. Do unblinded assessors bias muscle strength outcomes in randomized controlled trials of progressive resistance strength training in older adults? American journal of physical medicine & rehabilitation / Association of Academic Physiatrists. 2011;90(3):190–6.
    OpenUrl
  31. 31.↵
    Nuesch E, Reichenbach S, Trelle S, Rutjes AWS, Liewald K, Sterchi R, et al. The importance of allocation concealment and patient blinding in osteoarthritis trials: A meta-epidemiologic study. Arthritis Care and Research. 2009;61(12):1633–41.
    OpenUrl
  32. 32.↵
    Schulz KF, Chalmers L, Hayes RJ, Altman DG. Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association. 1995;273(5):408–12.
    OpenUrlCrossRefPubMedWeb of Science
  33. 33.↵
    Amer MA, Herbison GP, Grainger SH, Khoo CH, Smith MD, McCall JL. A meta-epidemiological study of bias in randomized clinical trials of open and laparoscopic surgery. The British journal of surgery. 2021;108(5):477–83.
    OpenUrl
  34. 34.↵
    Armijo-Olivo S, Fuentes J, da Costa BR, Saltaji H, Ha C, Cummings GG. Blinding in Physical Therapy Trials and Its Association with Treatment Effects: A Meta-epidemiological Study. American journal of physical medicine & rehabilitation. 2017;96(1):34–44.
    OpenUrlCrossRefPubMed
  35. 35.↵
    Bolvig J, Juhl CB, Boutron I, Tugwell P, Ghogomu EAT, Pardo JP, et al. Some Cochrane risk-of-bias items are not important in osteoarthritis trials: a meta-epidemiological study based on Cochrane reviews. Journal of clinical epidemiology. 2018;95:128–36.
    OpenUrlPubMed
  36. 36.↵
    Cuijpers P, Karyotaki E, Andersson G, Li J, Mergl R, Hegerl U. The effects of blinding on the outcomes of psychotherapy and pharmacotherapy for adult depression: A meta-analysis. European psychiatry : the journal of the Association of European Psychiatrists. 2015;30(6):685–93.
    OpenUrl
  37. 37.↵
    Dello Russo C, Cappoli N, Pilunni D, Navarra P. Local Investigators Significantly Overestimate Overall Response Rates Compared to Blinded Independent Central Reviews in Phase 2 Oncology Trials. Journal of Clinical Pharmacology. 2021;61(6):810–9.
    OpenUrl
  38. 38.↵
    Hrobjartsson A, Emanuelsson F, Thomsen ASS, Hilden J, Brorson S. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. International Journal of Epidemiology. 2014;43(4):1272–83.
    OpenUrl
  39. 39.↵
    Koletsi D, Spineli LM, Lempesi E, Pandis N. Risk of bias and magnitude of effect in orthodontic randomized controlled trials: a meta-epidemiological review. European journal of orthodontics. 2016;38(3):308–12.
    OpenUrlCrossRefPubMed
  40. 40.↵
    Martin GL, Trioux T, Gaudry S, Tubach F, Hajage D, Dechartres A. Association Between Lack of Blinding and Mortality Results in Critical Care Randomized Controlled Trials: A Meta-Epidemiological Study. Critical care medicine. 2021;49(10):1800–11.
    OpenUrl
  41. 41.↵
    Moustgaard H, Clayton GL, Jones HE, Boutron I, Jorgensen L, Laursen DLT, et al. Impact of blinding on estimated treatment effects in randomised clinical trials: Meta-epidemiological study. The BMJ. 2020;368:l6802.
    OpenUrl
  42. 42.↵
    Poolman RW, Struijs PAA, Krips R, Sierevelt IN, Marti RK, Farrokhyar F, et al. Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter? The Journal of bone and joint surgery American volume. 2007;89(3):550–8.
    OpenUrlCrossRef
  43. 43.↵
    Probst P, Zaschke S, Heger P, Harnoss JC, Huttner FJ, Mihaljevic AL, et al. Evidence-based recommendations for blinding in surgical trials. Langenbeck’s archives of surgery. 2019;404(3):273–84.
    OpenUrlCrossRefPubMed
  44. 44.↵
    Saltaji H, Armijo-Olivo S, Cummings GG, Amin M, da Costa BR, Flores-Mir C. Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. BMC medical research methodology. 2018;18(1):42.
    OpenUrl
  45. 45.
    Anthon CT, Granholm A, Pemer A, Laake JH, Moller MH. No firm evidence that lack of blinding affects estimates of mortality in randomized clinical trials of intensive care interventions: a systematic review and meta-analysis. JOURNAL OF CLINICAL EPIDEMIOLOGY. 2018;100:71–81.
    OpenUrlPubMed
  46. 46.↵
    Baiardo Redaelli M, Belletti A, Monti G, Lembo R, Ortalda A, Landoni G, et al. The impact of nonblinding in critical care medicine trials. Journal of Critical Care. 2018;48:414–7.
    OpenUrl
  47. 47.↵
    Braithwaite FA, Walters JL, Li LSK, Lorimer Moseley G, Williams MT, McEvoy MP. Effectiveness and adequacy of blinding in the moderation of pain outcomes: Systematic review and meta-analyses of dry needling trials. PeerJ. 2018;2018(7):e5318.
    OpenUrl
  48. 48.↵
    Feys F, Bekkering GE, Singh K, Devroey D. Do randomized clinical trials with inadequate blinding report enhanced placebo effects for intervention groups and nocebo effects for placebo groups? Systematic Reviews. 2014;3(1):14.
    OpenUrl
  49. 49.↵
    Howard JP, Shun-Shin MJ, Hartley A, Bhatt DL, Krum H, Francis DP. Quantifying the 3 Biases That Lead to Unintentional Overestimation of the Blood Pressure-Lowering Effect of Renal Denervation. Circulation: Cardiovascular Quality and Outcomes. 2016;9(1):14–22.
    OpenUrlAbstract/FREE Full Text
  50. 50.↵
    Hrobjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Rasmussen JV, Hilden J, et al. Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. International journal of epidemiology. 2014;43(3):937–48.
    OpenUrlCrossRefPubMed
  51. 51.↵
    Hrobjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and nonblinded outcome assessors. BMJ (Clinical research ed). 2012;344:e1119.
    OpenUrlAbstract/FREE Full Text
  52. 52.↵
    Hrobjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. CMAJ : Canadian Medical Association journal = journal de l’Association medicale canadienne. 2013;185(4):E201–11.
    OpenUrl
  53. 53.↵
    Ndounga Diakou LA, Trinquart L, Hrobjartsson A, Barnes C, Yavchitz A, Ravaud P, et al. Comparison of central adjudication of outcomes and onsite outcome assessment on treatment effect estimates. Cochrane Database of Systematic Reviews. 2016;2016(3):MR000043.
    OpenUrl
  54. 54.↵
    Trone JC, Chapelle C, Ollier E, Bertoletti L, Cucherat M, Mismetti P, et al. Statistical controversies in clinical research: limitations of open-label studies assessing antiangiogenic therapies with regard to evaluation of vascular adverse drug events-a meta-analysis. Annals of Oncology. 2018;29(4).
  55. 55.↵
    Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Jama. 2002;287(22):2973–82.
    OpenUrlCrossRefPubMedWeb of Science
  56. 56.↵
    Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess. 1999;3(12):i-iv, 1-98.
    OpenUrlPubMed
  57. 57.↵
    Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. Jama. 1999;282(11):1054–60.
    OpenUrlCrossRefPubMedWeb of Science
  58. 58.↵
    Lega JC, Mismetti P, Fassier T, Bertoletti L, Cucherat M, Vital-Durand D, et al. Double blind vs. open design on treatment effect of new oral anticoagulants in atrial fibrillation: A meta-analysis. Fundamental and Clinical Pharmacology. 2012;26(SUPPL.1):17.
    OpenUrl
  59. 59.↵
    Savović J, Jones HE, Altman DG, Harris RJ, Jüni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157(6):429–38.
    OpenUrlCrossRefPubMedWeb of Science
  60. 60.↵
    Unverzagt S, Prondzinsky R, Peinemann F. Single-center trials tend to provide larger treatment effects than multicenter trials: a systematic review. J Clin Epidemiol. 2013;66(11):1271–80.
    OpenUrlPubMed
  61. 61.↵
    Hartling L, Hamm MP, Fernandes RM, Dryden DM, Vandermeer B. Quantifying bias in randomized controlled trials in child health: a meta-epidemiological study. PLoS One. 2014;9(2):e88008.
    OpenUrl
  62. 62.↵
    Chaimani A, Vasiliadis HS, Pandis N, Schmid CH, Welton NJ, Salanti G. Effects of study precision and risk of bias in networks of interventions: a network meta-epidemiological study. Int J Epidemiol. 2013;42(4):1120–31.
    OpenUrlCrossRefPubMed
  63. 63.↵
    Bialy L, Vandermeer B, Lacaze-Masmonteil T, Dryden DM, Hartling L. A meta-epidemiological study to examine the association between bias and treatment effects in neonatal trials. Evid Based Child Health. 2014;9(4):1052–9.
    OpenUrl
  64. 64.↵
    Pildal J, Hróbjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36(4):847–57.
    OpenUrlCrossRefPubMedWeb of Science
  65. 65.
    Contopoulos-Ioannidis DG, Gilbody SM, Trikalinos TA, Churchill R, Wahlbeck K, Ioannidis JPA. Comparison of Large Versus Smaller Randomized Trials for Mental Health-Related Interventions. American Journal of Psychiatry. 2005;162(3):578–84.
    OpenUrlCrossRefPubMed
  66. 66.↵
    Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7(1):1–76.
    OpenUrl
  67. 67.
    Siersma V, Als-Nielsen B, Chen W, Hilden J, Gluud LL, Gluud C. Multivariable modelling for meta-epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Stat Med. 2007;26(14):2745–58.
    OpenUrlCrossRefPubMedWeb of Science
  68. 68.
    Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. Bmj. 2008;336(7644):601–5.
    OpenUrlAbstract/FREE Full Text
  69. 69.↵
    Savović J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, et al. Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study. American Journal of Epidemiology. 2017;187(5):1113–22.
    OpenUrl
  70. 70.
    Dechartres A, Altman DG, Trinquart L, Boutron I, Ravaud P. Association Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses. JAMA. 2014;312(6):623–30.
    OpenUrlCrossRefPubMedWeb of Science
  71. 71.↵
    van Tulder MW, Suttorp M, Morton S, Bouter LM, Shekelle P. Empirical evidence of an association between internal validity and effect size in randomized controlled trials of low-back pain. Spine (Phila Pa 1976). 2009;34(16):1685–92.
    OpenUrl
  72. 72.
    Fenwick J, Needleman IG, Moles DR. The effect of bias on the magnitude of clinical outcomes in periodontology: a pilot study. J Clin Periodontol. 2008;35(9):775–82.
    OpenUrlCrossRefPubMed
  73. 73.↵
    Hempel S, Miles J, Suttorp MJ, Wang Z, Johnsen B, Morton S, et al. AHRQ Methods for Effective Health Care. Detection of Associations Between Trial Quality and Effect Sizes. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012.
  74. 74.↵
    Welton NJ, Ades AE, Carlin JB, Altman DG, Sterne JAC. Models for potentially biased evidence in meta-analysis using empirically based priors. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2009;172(1):119–36.
    OpenUrlCrossRefWeb of Science
  75. 75.↵
    Sterne JA, Jüni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in ’meta-epidemiological’ research. Stat Med. 2002;21(11):1513–24.
    OpenUrlCrossRefPubMedWeb of Science
  76. 76.↵
    Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.
    OpenUrlFREE Full Text
  77. 77.↵
    Montori VM, Bhandari M, Devereaux PJ, Manns BJ, Ghali WA, Guyatt GH. In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol. 2002;55(8):787–90.
    OpenUrlCrossRefPubMedWeb of Science
  78. 78.↵
    Akl EA, Sun X, Busse JW, Johnston BC, Briel M, Mulla S, et al. Specific instructions for estimating unclearly reported blinding status in randomized trials were reliable and valid. J Clin Epidemiol. 2012;65(3):262–7.
    OpenUrlCrossRefPubMed
  79. 79.↵
    Haahr MT, Hróbjartsson A. Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials. 2006;3(4):360–5.
    OpenUrlCrossRefPubMedWeb of Science
  80. 80.↵
    Kent DM, Trikalinos TA. Therapeutic innovations, diminishing returns, and control rate preservation. Jama. 2009;302(20):2254–6.
    OpenUrlCrossRefPubMed
  81. 81.↵
    Pereira TV, Horwitz RI, Ioannidis JP. Empirical evaluation of very large treatment effects of medical interventions. Jama. 2012;308(16):1676–84.
    OpenUrlCrossRefPubMedWeb of Science
  82. 82.↵
    Janiaud P, Agarwal A, Tzoulaki I, Theodoratou E, Tsilidis KK, Evangelou E, et al. Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics. BMC Med. 2021;19(1):157.
    OpenUrlCrossRefPubMed
  83. 83.
    Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2014;2014(4):Mr000034.
    OpenUrl
  84. 84.↵
    Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–86.
    OpenUrlCrossRefPubMedWeb of Science
  85. 85.↵
    Treweek S, Bevan S, Bower P, Campbell M, Christie J, Clarke M, et al. Trial Forge Guidance 1: what is a Study Within A Trial (SWAT)? Trials. 2018;19(1):139.
    OpenUrlCrossRefPubMed
Back to top
Posted March 06, 2023.
Download PDF

Supplementary Material

Data/Code
Email
The impact of blinding on trial results: A systematic review and meta-analysis
Tyler Pitre, Sarah Kirsh, Tanvir Jassal, Mason Anderson, Adelia Padoan, Alexander Xiang, Jasmine Mah, Dena Zeraatkar
medRxiv 2023.03.05.23286821; doi: https://doi.org/10.1101/2023.03.05.23286821
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools

Subject Area