Effectiveness of SARS-CoV-2 mRNA Vaccines for Preventing Covid-19 Hospitalizations in the United States

Design

The protocol and statistical analysis plan for this project are available in the supplementary materials. We conducted a prospective observational case-control evaluation of vaccine effectiveness by comparing the odds of antecedent SARS-CoV-2 vaccination in hospitalized case-patients with Covid-19 versus control-patients without Covid-19. We included two control groups: 1) “test-negative” controls presented with signs or symptoms of an acute respiratory illness but tested negative for SARS-CoV-2; and 2) “syndrome-negative” controls were selected from hospitalized adults without signs or symptoms of an acute respiratory illness and tested negative for SARS-CoV-2. Test-negative controls are commonly used in hospital-based vaccine effectiveness evaluations;[6-9] in this test-negative design, utilizing a comparison group with the same clinical syndrome and similar level of acuity as cases reduces bias due to differential healthcare seeking behavior. Because of the potential for misclassification of true cases as test-negative controls due to false-negative test results using the test-negative design, particularly for those presenting late in the course of illness,[10] we included the second control group of hospitalized patients without an acute respiratory illness. Consistent with methodologies for vaccine effectiveness evaluations recommended by the World Health Organization,[11] SARS-CoV-2 vaccine coverage in both control groups served as a proxy for background vaccination rates in the source population for COVID-19 cases.

Setting

This surveillance activity was conducted by the IVY Network, a CDC-funded collaborative of hospital-based investigators across the US,[6, 12] and included patients hospitalized from March 11 through May 5, 2021, at 18 academic medical centers in 16 states. Each participating institution, including CDC, reviewed and conducted this activity consistent with applicable federal law and CDC policy (45 C.F.R. part 46.102(l)(2), 21 C.F.R. part 56; 42 U.S.C. §241(d); 5 U.S.C. §552a; 44 U.S.C. §3501 et seq.). The program was conducted as a public health surveillance activity without written informed consent.

Participants

Sites screened hospitalized adults ≥18 years old for potential eligibility through daily review of hospital admission logs and electronic medical records. Detailed eligibility criteria are shown in Supplementary Appendix B. In brief, Covid-19 cases included patients with a clinical syndrome consistent with acute Covid-19 and a positive test for SARS-CoV-2 within 10 days following symptom onset.[13-15] Test-negative controls had a clinical syndrome consistent with acute Covid-19 but tested negative for SARS-CoV-2. Syndrome-negative controls did not have a clinical syndrome consistent with Covid-19 and tested negative for SARS-CoV-2. Individual matching between cases and controls was not performed. Sites targeted a case: control ratio of approximately 1:1 for each control group and did not seek information on vaccination status until after patients were included.

Data Collection

Participants (or their proxies when participants could not answer medical questions) were interviewed by trained personnel to collect data on demographics, medical conditions, SARS-CoV-2 vaccination, and other patient characteristics. Additional information on underlying medical conditions and SARS-CoV-2 clinical testing was obtained through standardized medical record review.

Laboratory Analysis

Upper respiratory specimens (nasal swabs or saliva) were collected, frozen, and shipped to a central laboratory at Vanderbilt University Medical Center (Nashville, Tennessee). Specimens underwent reverse transcription polymerase chain reaction (RT-PCR) testing for the SARS-CoV-2 nucleocapsid gene using standardized methods and interpretive criteria.[16] Specimens positive for SARS-CoV-2 with a cycle threshold <32 for at least one of the two RT-PCR targets were shipped to the University of Michigan (Ann Arbor, Michigan) for viral whole genome sequencing using the ARTIC Network version 3 protocol on an Oxford Nanopore Technologies instrument (Supplementary Appendix B).[17] SARS-CoV-2 lineages were assigned with >80% coverage using Pangolin genomes.[18]

Classification of Case-Control Status

Final classification of case-control status was determined with consideration of both clinical SARS-CoV-2 testing completed at local hospital laboratories and RT-PCR testing completed at the central laboratory. Cases tested positive for SARS-CoV-2 by a clinical test or central laboratory RT-PCR test. Cases with SARS-CoV-2 detected by RT-PCR with a cycle threshold >32 were included in the analysis, but viral sequencing information was not available for these cases. Test-negative controls tested negative for SARS-CoV-2 by all clinical and central testing. Patients initially classified as a test-negative control who subsequently tested positive for SARS-CoV-2 by central RT-PCR testing were reclassified as cases. Syndrome-negative controls tested negative for SARS-CoV-2 by all clinical and central testing. Patients included as syndrome-negative controls who subsequently tested positive for SARS-CoV-2 in the central laboratory were excluded from the analysis.

Classification of Vaccination Status

Details of SARS-CoV-2 vaccination, including dates and location of vaccination, vaccine product, and lot number, were ascertained through a systematic process including patient or proxy interview and source verification. Sources of documentation included vaccination card, hospital records, state vaccine registries which were searched at the time of interview and again approximately 28 days later, and vaccine records requested from clinics and pharmacies. Vaccine doses were classified as administered if source documentation was identified or if the patient/proxy reported a vaccine dose with a plausible date and location of vaccination.

The SARS-CoV-2 mRNA vaccines are administered as a two-dose series; participants were considered fully vaccinated 14 days after receipt of the second vaccine dose.[19] Vaccination status was classified based on the number of mRNA vaccine doses received before a reference date, which was the date of symptom onset for cases and test-negative controls and date of hospital admission for syndrome-negative controls. Participants were classified as: unvaccinated if they had received no vaccine doses prior to the reference date; partially vaccinated if they received one dose ≥14 days before the reference date; and fully vaccinated if they received both doses ≥14 days before the reference date. As protective immunity from SARS-CoV-2 vaccines is not expected immediately after the first dose,[12] patients who received a first dose <14 days before the reference date were excluded from the analysis. Patients who received a SARS-CoV-2 vaccine that had not been authorized in the US were excluded. Due to recent introduction of the Janssen (Johnson & Johnson) SARS-CoV-2 vaccine following its EUA in February 2021,[2] patients who received this vaccine were also excluded.

Statistical Analysis

Vaccine effectiveness and 95% confidence intervals (95% CI) were determined by comparing the odds of prior SARS-CoV-2 vaccination in case-patients and control-patients, calculated as: vaccine effectiveness = (1 – odds ratio) × 100%.[20]

Primary vaccine effectiveness estimates were calculated in adults of all ages for full vaccination versus unvaccinated and for partial vaccination versus unvaccinated. Unadjusted vaccine effectiveness was calculated with simple logistic regression and then a model building approach was applied to estimate adjusted vaccine effectiveness accounting for potential confounders. Prespecified covariates in base multivariable logistic regression models included calendar time in biweekly intervals, US Department of Health and Human Services region, age, sex, and self-reported race and Hispanic ethnicity. We repeated the regression by adding health status indicators and SARS-CoV-2 exposure variables potentially associated with the likelihood of vaccination and risk of Covid-19 hospitalization (detailed in Supplementary Appendix B). No variable generated an absolute change in the odds ratio of vaccination of more than 5% in either direction, which was a prespecified cutoff, so no additional variables were added to the base model. Potential effect modification of prior SARS-CoV-2 infection (at least 14 days prior to the current illness) was assessed using a likelihood ratio test (with a P-value <0.15 suggestive of effect modification).[21]

Separate assessments were initially performed using the test-negative control and the syndrome-negative control group to assess comparability of estimates. Effectiveness estimates were very similar using the test-negative and syndrome-negative control groups. Therefore, control groups were combined to improve precision.

Vaccine effectiveness estimates were stratified by age group (18–49, 50–64, or ≥65 years), SARS-CoV-2 vaccine product (Pfizer-BioNTech or Moderna), SARS-CoV-2 variant, and underlying medical conditions with a prevalence ≥20% in the population, including immunocompromising conditions[6], diabetes mellitus, chronic lung disease, chronic cardiovascular disease, and obesity (definitions provided in Supplementary Appendix B). Sensitivity analyses are described in Supplementary Appendix B. Stata Version 16 (College Station, TX) and SAS 9.4 (Cary, NC) were used for statistical analysis.

Participants

We included 1210 patients (590 cases, 334 test-negative controls, and 286 syndrome-negative controls) (Figure S1) enrolled from 18 clinical sites (Figure 1) over the course of 51 days (Figure S2). Overall, median age was 58 years, 276 (22.8%) were non-Hispanic Black, 167 (13.8%) were Hispanic, and 248 (20.6%) had an immunocompromising condition (Table 1).

• Download figure
• Open in new tab

Figure 1.

Map of continental United States with incidence of Covid-19 hospitalizations by state in April 2021 indicated by color (red). Participating sites are shown on the map with circles; the size of each circle represents the number of Covid-19 cases included from each site in this analysis -- IVY Network, United States, March–May 2021.*

^* Sources of state Covid-19 hospitalization data and population census data were HealthData.gov and United States Census Bureau. [31, 32]

After excluding patients with a first vaccine dose 0-13 days before the reference date, full SARS-CoV-2 vaccination had been received by 45 (8.3%) cases, 115 (36.7%) test-negative controls, and 100 (35.6%) syndrome-negative controls (Figure 2); 454 (83.5%) cases were unvaccinated. Among fully vaccinated patients, median time between the last vaccine dose and onset of Covid-like symptoms was 44 days (interquartile range [IQR] 25 to 54 days) for 45 cases and 42 days (IQR 27 to 60 days) for 115 test-negative controls. Among fully vaccinated patients, 251 (96.5%) had source verification of vaccine doses. SARS-CoV-2 whole genome sequencing was completed for 231 cases, with variant of concern B.1.1.7 the most commonly identified lineage (138/231, 59.7%) (Table 2).

• Download figure
• Open in new tab

Figure 2.

Vaccination status of case patients (N=544), test-negative controls (N=313), and syndrome-negative controls (N=279) — IVY Network, United States, March–May 2021.^*

^* Vaccination status was classified based on the number of mRNA vaccine doses received before a reference date, which was the date of symptom onset (for cases and test-negative controls) or date of hospital admission (for syndrome-negative controls). Unvaccinated patients received no doses of mRNA vaccine before the reference date, partially vaccinated patients received one of two doses of mRNA vaccine ≥14 days before the reference date or both doses with the second dose received <14 days before the reference date, and fully vaccinated patients received both doses of vaccine ≥14 days before the reference date. Patients who received the first dose of a SARS-CoV-2 vaccine 0-13 days before the reference date (46 cases; 21 test-negative controls; 7 syndrome-negative controls) were excluded from this analysis.

Vaccine effectiveness

Vaccine effectiveness results for full vaccination were very similar using the test-negative control group (86.4%, 95% CI: 78.7–91.3%) and syndrome-negative control group (86.8%, 95% CI: 78.7–91.8%) (Figure S3 and Figure S4). After combining control groups, SARS-CoV-2 vaccine effectiveness for full vaccination to prevent Covid-19 hospitalizations was 86.9% (95% CI: 80.4–91.2%) (Figure 3). Vaccine effectiveness for full vaccination was similar for Pfizer-BioNTech (84.3%, 95% CI: 74.6–90.3%) and Moderna (90.0%, 95% CI: 82.0–94.4) vaccines. Point estimates were higher for people aged 18–49 years (97.3%, 95% CI: 78.9–99.7%) than aged 50–64 years (74.7%, 95% CI: 47.2–87.9%) and aged ≥65 years (87.2%, 95% CI: 77.6–92.7%). Among adults aged ≥65 years, vaccine effectiveness was similar in those 65-74 years (87.9%, 95% CI: 74.0–94.4%) and ≥75 years (90.5%, 95% CI: 73.2–96.7%). Vaccine effectiveness against SARS-CoV-2 B.1.1.7 lineage was 92.8% (95% CI: 83.0–96.9%).

• Download figure
• Open in new tab

Figure 3.

Vaccine effectiveness of SARS-CoV-2 mRNA vaccines for the prevention of Covid-19 hospitalizations overall and by subgroups — IVY Network, United States, March–May 2021.^*

^* The analysis included case patients with Covid-19-like illness who tested positive for SARS-CoV-2 infection and control patients combined from two groups, including a (1) test-negative control group with Covid-19-like illness and negative SARS-CoV-2 testing and (2) a syndrome-negative control group without Covid-19-like illness and negative for SARS-CoV-2. Vaccine effectiveness models were adjusted for calendar time in biweekly intervals, US Department of Health and Human Services region, age in years, sex, race and ethnicity. Vaccination status was classified based on the number of mRNA vaccine doses received before a reference date, which was defined as the date of symptom onset for cases and test-negative controls and date of hospital admission for syndrome-negative controls. Unvaccinated patients received no doses of vaccine before the reference date, partially vaccinated patients received one of two doses of vaccine ≥14 days before the reference date or both doses with the second dose received <14 days before the reference date, and fully vaccinated patients received both doses of vaccine ≥14 days before the reference date.

Vaccine effectiveness was significantly reduced for patients with immunocompromising conditions (59.2%, 95% CI: 11.9–81.1%) compared to individuals without an immunocompromising condition (91.3%, 95% CI: 85.5-94.7%) (Figure 3). Restricted to immunocompromised patients with an active solid organ or hematologic malignancy or solid organ transplant, vaccine effectiveness was 51.2% (95% CI: -30.7–81.8%). Vaccine effectiveness point estimates were lower for patients with underlying cardiovascular disease (82.8%, 95% CI: 72.3–89.3%), chronic lung disease (82.1%, 95% CI: 60.3–91.9%), and diabetes mellitus (81.6%, 95% CI: 64.5–90.5%) compared to patients without these underlying conditions but 95% confidence limits overlapped. Partial vaccination had a vaccine effectiveness of 76.0% (95% CI: 63.7–84.1%). Evidence of effect modification by prior laboratory-confirmed SARS-CoV-2 infection was not observed (likelihood ratio test p-value = 0.55). Sensitivity analyses produced results similar to the primary analysis (Table S2).

Breakthrough vaccine Covid-19 hospitalizations

Forty-five Covid-19 case patients met our definition of being fully vaccinated before symptom onset. Among these, median age was 68 years (IQR 62–77 years), median time between the final vaccine dose and symptom onset was 44 days (IQR: 25 to 54 days), and 20 (44.4%) had an immunocompromising condition, including active solid organ or hematologic malignancy (n=9), and prior solid organ transplant (n=7). These breakthrough cases included 28 patients vaccinated with the Pfizer-BioNTech product and 17 patients vaccinated with the Moderna product.