An analysis of studies pertaining to masks in Morbidity and Mortality Weekly Report: Characteristics and quality of all studies from 1978 to 2023

Introduction

Prior to the COVID-19 pandemic, pooled randomized data¹ on surgical and N95 respirator masks in the community and healthcare setting failed to demonstrate evidence of efficacy against influenza or influenza-like illness. In March of 2020, the Centers for Disease Control and Prevention (CDC) did not generally recommend mask wearing for healthy people², consistent with the advice from the US Surgeon General.³ Over several weeks in March and early April 2020, a coordinated social media campaign to recommend masks began.⁴ Then on April 3rd, 2020, the CDC recommended people ages 2 years and older wear a cloth face covering in public.⁵ On July 15th, 2020, the CDC Director recommended all Americans start wearing masks as a way to “get the epidemic under control” ^{5, 6} citing a Morbidity and Mortality Weekly (MMWR) study involving two hairstylists in Missouri. ⁷ That coming Fall of 2020, universal masking in schools and daycares was recommended by the CDC⁸ and widespread mandates were enacted at the state, district and county levels for children as young as two. Masking on public transportation was required by federal mandate starting January of 2021.⁹

MMWR is a weekly scientific journal without external peer review overseen by the CDC to publish data on nationally notifiable infectious diseases, which can then be used for program planning, evaluation, and policy development.¹⁰ It is considered their primary avenue for disseminating scientific information and is often referred to as “the voice of CDC.” ¹⁰ The review and publication process at this journal, the levels of evidence generated, and the extent to which the studies published in this journal represent and advance current international scientific understanding remain largely opaque to the general public.

The aim of the present study was to evaluate all studies published in MMWR pertaining to masks, looking specifically at what conclusions were drawn about mask effectiveness and whether or not the conclusions were appropriate given the data presented. If causality was inferred, we determined whether or not this was appropriate, given the study’s methodology. Secondary aims included describing multiple study characteristics, including study type, number of authors, if some or all of the authors were from the CDC, and whether or not studies cited randomized or conflicting data.

Methods

Results

Our search, spanning the years 1978 to 2023, identified 83 MMWR published studies on PubMed, all of which were published after 2019. We excluded 5 guidance documents and a search result that was a stand-alone figure. Our search identified 83 MMWR published studies on PubMed. We excluded 5 guidance documents and a search result that was a stand-alone figure. Of the included 77 studies, 23 (29.9%) studies were graded as assessing mask effectiveness with the remaining 54 (70.1%) not having the methodology to do so. 72 (93.5%) pertained to SARS-CoV-2, one pertained to SARS-CoV-2 and influenza co-infection, three studies pertained mainly to influenza and one pertained to rhino and enteroviruses.

No studies met our inclusion criteria from 1978 through 2019. Thirty studies were published in 2020; 33 were published in 2021; and 14 were published in 2022. The median number of participants was 558 (IQR: 171, 2964). The median number of authors was 13 (IQR: 9, 26), total listed authors including duplicates was 1544. Seventy studies (90.9%) had one or more authors affiliated with the CDC.

The kappa statistic for intra-author agreement in the determination of whether studies made a conclusion about masks was 0.69 (p<0.0001), and the kappa statistic for the use of causal language was 0.66 (p<0.0001). These numbers suggest that the agreement was substantial for both.

Study characteristics, stratified by whether or not masks were tested for effectiveness, are shown in Table 1 and described in detail in the supplementary material. 75/77 (97.4%) studies were from the United States alone, one was from Chile and one was from multiple countries. All geographic regions were represented with 32/77 (41.6%) using multi-state data. The most common study design was observational without a control or comparator group 22/77 (28.6%). All age groups were represented. The most common setting was community (35/77; 45.5%) followed by kindergarten through high school (13/77; 16.9%). The characteristics of the 77 studies, by whether or not they had appropriate methodology to test masks, are described in the Supplementary Material.

In Figure 1, we show a total of 23/77 (29.9%) identified studies that assessed the effectiveness of masks, however, 58/77 (75.3%) stated masks were effective. Of these 58 studies, 41/58 (70.7%) used causal language and 40/58 (69.0%) used causal language inappropriately. One mannequin study allowed causal inference. 11/77 (14.3%) found a statistically significant inverse relationship between masking and cases. No studies (0/77; 0%) were randomized. 4/77 (5.2%) had a numerically higher number of cases in the mask group than the comparator group but all 4/4 (100%) concluded masks were effective. Of all publications included, 0/77 (0.0%) cited a randomized study or review of only randomized studies. Of all 58 studies stating masks were effective, only 1/58 (1.7%), which mainly focused on influenza¹¹ mentioned conflicting data on mask effectiveness.

• Download figure
• Open in new tab

Figure 1.

Select characteristics of the 77 Morbidity and Mortality Weekly Report (MMWR) publications pertaining to masks

Table 2. Shows examples of language used in multiple MMWR studies which did not have appropriate methodology to draw conclusions about the effectiveness of masking.

Table 3 shows the data characteristics of the included MMWR studies, overall and stratified by the testing of masks or not.

As shown in Table 4, of the studies that evaluated masks, 22/22 (100%) concluded masks were effective; 18/22 (81.8%) reported results favoring masks; 13/22 (59.1%) tested for statistical differences; and 12/22 (54.5%) of which were statistically significant.

Details about the included studies and grading of subjective endpoints are publicly available at the following GitHub repository https://github.com/tracybethhoeg/mmwrmasks under an MIT license.

Discussion

We found that, among the 77 studies identified pertaining to masks published in MMWR, 30% tested the effectiveness of masks, with 14% having statistically significant results, yet over 75% of all 77 studies concluded masks were effective. Of the 5% that reported higher numbers of cases in the masked group than the comparator group, all concluded masks were effective. MMWR studies consistently drawing conclusions about mask effectiveness without supporting evidence is particularly problematic and difficult to justify considering the totality of randomized evidence about the use of surgical or N95 masks to prevent the spread of respiratory viruses has been negative.^{1, 12}

Over 50% of the identified studies used causal language in their conclusions about mask effectiveness. Only one of these studies, which was a mannequin study, had methodology which permitted causal inference. In other words, the remaining 40 studies used language that indicated with certainty that masks lower transmission rates in spite of the fact their results, at most, found a correlation. 25 of these 40 studies, however, did not even test mask effectiveness. We have provided examples of study conclusions, which stated masks resulted in case reductions, in spite of the fact none of the studies had the appropriate methodology to assess mask effectiveness (Table 2). There were a total of 25 studies that did not evaluate masks but made causal claims about their effectiveness. It is important to note that the one identified study, which permitted causal inference, was a study of particle filtration on mannequins¹³, with unknown relevance for human health.

The inappropriate use of causal language used in MMWR studies was also adopted directly by the CDC director when she cited an observational phone survey, which also happened to be included in the present analysis¹⁴, stating to the public “Masks can help reduce your chance of #COVID19 infection by more than 80%.”¹⁵ This referenced study found an association between respondents’ recollection about mask wearing and self-reported COVID-19 tests, which was non-significant for cloth masks.

A number of studies that were particularly influential in shaping policy recommendations around masking in the public and schools were not even among the studies that attempted to properly evaluate masks, as they had no control group or comparative time period. These studies included the investigation of two Missouri hairdressers⁷, the Georgia overnight camp outbreak investigation¹⁶, and the Marin school outbreak investigation.¹⁷ None of these had methodology that permitted an evaluation of mask effectiveness, but they nonetheless drew conclusions about mask effectiveness (Table 2), which were then rapidly communicated to the public via the CDC.

Commensurate with the existing randomized data prior to 2020,¹ the CDC had previously recommended against wearing masks to prevent respiratory infections. ^{2, 6} A shift in messaging for the public to wear masks to control the pandemic came July 15th from the CDC director. This change came following the report of two hairdressers wearing masks while working, which concluded that masks were “likely a contributing factor in preventing transmission of SARS-CoV-2 during the close-contact interactions between stylists and clients.”⁶ In this instance, public health recommendations shifted largely based on anecdotal data in MMWR.

Randomized studies are the most reliable method of determining whether an intervention is efficacious. None of the studies identified in MMWR were randomized and none cited randomized data. Due to a high likelihood of confounding variables and/or spurious findings,¹⁸ observational studies of masking are unlikely to provide reliable information about the ability of masks to prevent infection with or transmission of SARS-CoV-2 or other respiratory viruses and are, with few exceptions, inappropriate for causal inference.

Only one study mentioned conflicting data on masking efficacy, in spite of the existing overall negative randomized data.^{1, 12} Interestingly the focus of this study was influenza and it was an international study.¹¹

Taken together, the absence of randomized data, the lack of acknowledgment of conflicting or randomized data on mask efficacy, and the tendency to conclude masks are efficacious either without any, or sufficient, data to make causal claims, is suggestive of bias within the journal. Our findings may also help explain why CDC remains an international outlier in continuing to recommend masks for COVID-19 under certain circumstances, including for children down to age 2.¹⁹

Concerns about publication bias within MMWR have been raised previously, when follow-up data to a 2-week study with a limited sample were submitted, which failed to identify evidence of school mask mandate effectiveness, was rejected from the MMWR.^{18, 20} In another school masking study, errors in data analysis and methodology, which normally would warrant retraction, were not addressed by the journal.²¹

One Journal of the American Medical Association (JAMA) Viewpoint²² described how, starting September 11, 2020, political appointees may have “demanded the ability to review and revise scientific reports” in MMWR, and concern was raised about “political appointees trying to influence the scientific process.” The extent to which this happened or is still happening is unclear. However, even prior to this, unlike other peer-reviewed scientific journals, MMWR publications have not and do not undergo any external peer review. Rather, they undergo a “clearance process”^{10, 23}, which is sometimes referred to as “internal peer review”.²² Both political involvement and lack of input from external domain experts could influence the journal’s ability to objectively evaluate scientific data. But the extent to which either of these explain our findings is beyond the scope of this paper.

However, the process by which scientific data are interpreted and published in MMWR and then promoted by the CDC is not transparently communicated to the public. Because the CDC uses data published in the MMWR journal to develop its guidelines, the quality of scientific data and data interpretation within the journal have major implications for public health and well-being in the United States, extending far beyond masks.

Our search did not identify any mask articles published prior to 2020, though our search was not restricted by date. Ninety percent of mask studies published in the MMWR had one or more authors with CDC affiliations. There was a median of 13 authors per paper and, though there were some authors who co-authored multiple papers, there were a total of 1544 paper authors, which speaks to the large amount of effort that went into studying and publishing about this topic in the journal. It is thus disappointing that, due to the intrinsic limitation of the study designs, the sum of the work was inconclusive, yet strong conclusions were drawn and communicated to the public nonetheless.