Abstract
Introduction Recently, a Cochrane review by Jefferson et al. on physical interventions to slow the spread of respiratory viruses concluded that, “Wearing masks in the community probably makes little or no difference to the outcome of laboratory-confirmed influenza/SARS-CoV-2 compared to not wearing masks”, though this finding had a wide confidence interval. Cochrane issued a rare clarifying statement, fueling controversy. We sought to contextualize the findings of the review by Jefferson et al.
Methods We searched for consecutive reviews by Cochrane published on or before March 9th, 2023. We included studies where a central finding showed an intervention offered no statistically significant benefit, and ascertained the language used by reviewers to describe that result. We compare this to the report by Jefferson et al., and deemed it consistent or inconsistent with the language of their report.
Results We found between November 21st, 2022, and March 9th, 2023, there were 20 Cochrane reviews that met the inclusion criteria. We found that 95% (n = 19) of the reviews used language that was consistent with Jefferson’s findings, while 5% (n = 1) used language inconsistent with Jefferson’s conclusion, describing the effect of the intervention on the outcome as “unclear”.
Discussion Most reviews performed by Cochrane conclude that interventions which fail to show statistically significant benefits make “no difference” have “no effect” or do not “increase or decrease” the outcome, and this occurs despite wide confidence intervals. The conclusions by Jefferson et al. are consistent with Cochrane reporting guidelines and clarification from the organization was unjustified.
Introduction
An updated Cochrane review on physical interventions to slow the spread of respiratory viruses1 has sparked debate among researchers and in the media over the interpretation of the results, leading Cochrane’s editor-in-chief to issue a statement attempting to clarify comments made by the lead author2.
Among other topics, the review examined the effect of medical or surgical masks on the spread of respiratory viruses in the community and found a relative risk of 1.01 95% CI (0.72 - 1.42) after pooling 6 trials. The authors of the Cochrane review concluded, “Wearing masks in the community probably makes little or no difference to the outcome of laboratory-confirmed influenza/SARS-CoV-2 compared to not wearing masks’’, and the first author and senior reviewer Tom Jefferson was quoted by a news outlet saying, “There is just no evidence that they make any difference. Full stop.”3
In response, Cochrane editor-in-chief Karla Soares-Weiser, issued an unprecedented clarification, stating, “Many commentators have claimed that a recently-updated Cochrane Review shows that ‘masks don’t work’, which is an inaccurate and misleading interpretation. It would be accurate to say that the review examined whether interventions to promote mask wearing help to slow the spread of respiratory viruses, and that the results were inconclusive.” The editor went on to specifically criticize Jefferson, “Soares-Weiser also said, though, that one of the lead authors of the review even more seriously misinterpreted its finding on masks by saying in an interview that it proved ‘there is just no evidence that they make any difference.’ In fact, Soares-Weiser said, ‘that statement is not an accurate representation of what the review found.’”4
Dueling interpretations of the central findings beg the question of consistency. In the case of masking to slow the spread of respiratory viruses, masking proponents point to a wide confidence interval, which includes values compatible with moderate benefit (e.g., 0.8) as evidence that the analysis cannot show ‘masks don’t work’ but rather ‘masks might work, with longer or better studies.’ In contrast, critics point to the fact that, as a general rule in medicine, interventions that fail to show benefit despite repeated testing are considered negative interventions. As such we sought to empirically assess the interpretation of Cochrane reviews with broad confidence intervals spanning one.
Methods
Data collection
We searched the Cochrane Library for reviews where a central finding had a confidence interval that crossed one. We included studies that were: 1. Intervention type reviews 2. Had a main outcome measured as a risk ratio, odds ratio, or hazard ratio 3. Had a confidence interval spanning one with moderate or high certainty evidence. 4. Published on or before March 9th, 2023. One reviewer (SM) selected twenty consecutive studies that met the inclusion criteria. We extracted quotes regarding Cochrane’s interpretation listed in the Table. The Figure shows the corresponding point estimates and confidence intervals.
Results and conclusions from twenty consecutive Cochrane reviews reporting null findings published on or before March 9th, 2023
Forest plot showing point estimates and 95% confidence intervals from 20 consecutive Cochrane reviews with null findings published on or before March 9th, 2023. Point estimates are represented as risk ratios (RR), hazard ratios (HR), or odds ratios (OR). The dashed line indicates the line of no effect.
Statistical analysis
Descriptive statistics were gathered. The primary outcome of interest was the specific reviewer language used in the interpretation of the findings. We concluded interpretations were consistent with Jefferson’s interpretations if they described the intervention as having no effect on the outcome, using the phrases “no difference” or “no effect” or “does not increase or decrease” when describing the effect of the intervention on the outcome. Interpretations describing the effect of the intervention on the outcome as being “unclear” were categorized as inconsistent with Jefferson’s language.
Results
We found between November 21st, 2022, and March 9th, 2023, there were 20 Cochrane reviews that met the inclusion criteria. We found that 95% (n = 19) of the reviews used language that was consistent with Jefferson’s findings, while 5% (n = 1) described their results as “unclear”.
The argument that Jefferson’s findings could be compatible with some benefit would apply to all the analyses in the Figure. In some cases the confidence interval is primarily on the side of benefit, e.g. Kim et al. on the effect of dose-escalated radiotherapy on time to death from prostate cancer (HR 0.83, 95% CI 0.66 - 1.04).5 In comparison, Jasani’s finding on the effect of paracetamol compared with ibuprofen on failure of ductal closure (RR 1.02, 95% CI 0.88 - 1.18)6 and Jefferson’s study (RR 1.01, 95% CI 0.72 - 1.42)1 indicate no differences in outcome between the arms and have fairly balanced confidence intervals above and below one, indicating that they aren’t any more compatible with relative benefit than they are harm.
It’s worth noting that the precision around the estimates, represented by the widths of the confidence intervals, do vary between the studies. Agabio’s finding on the effect of baclofen on the number of participants with at least one adverse event in a study on alcohol use disorder7 has a relatively narrow confidence interval (RR 1.05, 95% CI 0.99 – 1.11) in comparison to Jefferson’s findings. It’s reasonable to be more skeptical of findings with wide confidence intervals and consider whether it is worth running additional high-quality studies to improve precision. However, null findings like these are still often enough to deprioritize an intervention or abandon it entirely, especially when those findings are derived from randomized trials.
Discussion
We found that Jefferson’s conclusion on the effect of medical/surgical masks on laboratory-confirmed influenza or SARS-CoV-2 was similar to other reviews with null findings. For example, in their review assessing Remdesivir for the treatment of COVID-198, Grundeis et al. conclude “With moderate-certainty evidence, remdesivir probably makes little or no difference to all-cause mortality at up to day 28”. The language used to describe the results is nearly identical to the language used by Jefferson et al. apart from the study-specific variables. In fact, it’s similar to most of the conclusions listed in the Table, with some variation depending on the certainty of the evidence. This is no coincidence; the Cochrane Handbook9 provides guidance on how to report and interpret findings and this is the language they recommend for reports where the size of the effect estimate is “trivial, small, unimportant effect, or no effect”, with the inclusion of the word “probably” for moderate certainty of evidence.
On these grounds, the conclusions made by Jefferson and colleagues were not only appropriate, but in line with the standardized approach created by Cochrane. Further, Jefferson’s comment in the media about there being “no evidence that they make any difference”3 is consistent with their conclusion in the Cochrane review in which they stated, “Wearing masks in the community probably makes little or no difference to the outcome of laboratory-confirmed influenza/SARS-CoV-2 compared to not wearing masks.”1
The pooling of the best available evidence in meta-analyses informs our understanding of how well interventions work. Null findings often do contribute to decision-making, despite the fact that they don’t provide conclusive proof that an intervention doesn’t work under any circumstances. We found no obvious difference between Jefferson’s review and other recent reviews that would justify the differential interpretation and treatment of this study or the unprecedented comments made over its findings. Clarifying comments of the editor-in-chief of Cochrane appear unjustified.
Data Availability
All data produced in the present work are contained in the manuscript
Footnotes
Funding: None. VP is funded by Arnold Ventures to study Low Value Care. This study received no specific funding.
Disclosure: Vinay Prasad’s Disclosures. (Research funding) Arnold Ventures (Royalties) Johns Hopkins Press, Medscape, and MedPage (Honoraria) Grand Rounds/lectures from universities, medical centers, non-profits, and professional societies. (Consulting) UnitedHealthcare and OptumRX. (Other) Plenary Session podcast has Patreon backers, YouTube, and Substack. All other authors have no financial nor non-financial conflicts of interest to report.
