Abstract
Background The decline in COVID-19 mRNA vaccine effectiveness (VE) is well established, however the impact of variant-specific immune evasion and waning protection remains unclear. Here, we use whole-genome-sequencing (WGS) to tease apart the contribution of these factors on the decline observed following the introduction of the Delta variant. Further, we evaluate the utility of calendar-period-based variant classification as an alternative to WGS.
Methods We conducted a test-negative-case-control study among people who received SARS-CoV-2 RT-PCR testing in the Yale New Haven Health System between April 1 and August 24, 2021. Variant classification was performed using WGS and secondarily by calendar-period. We estimated VE as one minus the ratio comparing the odds of infection among vaccinated and unvaccinated people.
Results Overall, 2,029 cases (RT-PCR positive, sequenced samples) and 343,985 controls (negative RT-PCRs) were included. VE 14-89 days after 2nd dose was significantly higher against WGS-classified Alpha infection (84.4%, 95% confidence interval: 75.6-90.0%) than Delta infection (68.9%, CI: 58.0-77.1%, p-value: 0.013). The odds of WGS-classified Delta infection were significantly higher 90-149 than 14-89 days after 2nd dose (Odds ratio: 1.6, CI: 1.2-2.3). While estimates of VE against calendar-period-classified infections approximated estimates against WGS-classified infections, calendar-period-based classification was subject to outcome misclassification (35% during Alpha period, 4% during Delta period).
Conclusions These findings suggest that both waning protection and variant-specific immune evasion contributed to the lower effectiveness. While estimates of VE against calendar-period-classified infections mirrored that against WGS-classified infections, our analysis highlights the need for WGS when variants are co-circulating and misclassification is likely.
Summary of main points Using whole genome sequencing, we provide direct evidence of waning vaccine effectiveness and variant-specific immune evasion during the Delta wave. Effectiveness estimates against calendar-period-classified infections approximated estimates against WGS-classified infections, however, calendar-period classification was associated with a variant misclassification.
Introduction
As the number of cases and deaths among vaccinated and unvaccinated people rose following the introduction of the Delta variant of SARS-CoV-2 in the summer of 2021, concerns regarding variant specific immune evasion surfaced.[1,2] While some studies found vaccine effectiveness against Delta was lower than previously circulating variants (such as Alpha)[3–11], other studies found the differences were likely driven by waning levels of protection, not variant-specific immune evasion.[12–17]
However, interpreting and comparing studies of vaccine effectiveness is challenging due to the use of inconsistent variant-specific outcome definitions. In the absence of whole genome sequencing (WGS), studies commonly rely on calendar-period-based variant classification.[6,8,10,11,15,18,19] Though studies reliant on this approach have incorporated sensitivity analyses with varying time periods, this approach remains subject to unaccountable outcome (variant) misclassification. Despite its frequent use, a comparison of vaccine effectiveness resulting from WGS and calendar-period-based variant classification has not been completed. As a result, the reliability of calendar-period-based estimates remains uncertain.
In this retrospective analysis, we leveraged data from a large cohort of individuals enrolled in the Yale New Haven Health System (YNHH) who underwent RT-PCR testing to estimate and compare the effectiveness of mRNA vaccines (mRNA-1273-Moderna and BNT162b2-Pfizer) against infection, symptomatic infection and COVID-19 associated hospitalization with WGS-classified Alpha, Delta, and other co-circulating variants. Further, we examined whether protection conferred by primary series vaccination declined over time since series completion (second dose receipt). Finally, to examine the reliability of calendar-period-based variant classification, we compared estimates of vaccine effectiveness during the calendar period of Alpha and Delta variant predominance with those obtained from WGS.
Methods
Study Design and Setting
We conducted a test-negative-case-control (TNCC) analysis using RT-PCR tests collected between April 1 and August 24, 2021, as part of the larger Studying COVID-19 Outcomes after SARS-CoV-2 Infection and Vaccination (SUCCESS) Study at YNHH. The YNHH is a large academic health system comprising four delivery networks in Connecticut, southeastern New York, and western Rhode Island. We chose a TNCC design because it mitigates the risk of confounding introduced by care-seeking and testing access and has been shown to provide estimates of vaccine effectiveness consistent with those from randomized control trials.[20–24]
RT-PCR Collection and Sequencing
SARS-CoV-2 RT-PCR testing was broadly implemented at YNHH to screen people with suspected SARS-CoV-2 exposure and patients who underwent procedures or were admitted for hospitalization. Specimens were collected using nasopharyngeal/oropharyngeal swabs and primarily evaluated with QuantStudio 7 Flex Real-Time PCR System (Applied Biosystems, Waltham, MA), BHG Probe Panther (Hologic, Marlborough, MA), and Cobas 6800 System (Roche Diagnostics, Basel, Switzerland). Excess positive nasopharyngeal/oropharyngeal swabs were stored at -80C prior to processing for WGS.
Specimens were processed using custom NEB/Roche reagents the SARS-CoV-2 v2 SNAP kit + Omicron spike-in from IDT/Swift and were sequenced on the Illumina NovaSeq 6000 platform on SP or S2 flow cells. Variant calls were made for each specimen based on the proportion of variant-defining mutations present in the specimen (cutoff: ≥40%). Specimens that did not meet a variant-defining criterion were classified as an “unidentified variant.” Specimens with ≤65% of targeted bases covered at 100x were considered to have failed quality control and excluded. This cutoff was selected internally as it corresponded with the inclusion of 5% of unclassifiable samples (e.Table1).
Data Access
Demographic, comorbidity, COVID-19 vaccination, SARS-CoV-2 testing, and hospitalization data were extracted from the electronic health records (HER) using the Yale Computational Health Platform.[25] Records for vaccinations that occurred outside of YNHH were obtained from the state vaccination registry and extracted using the same platform. COVID-19 symptom data were collected from medical notes using a Natural Language Processor (NLP).[26] This study was approved by the Yale Institutional Review Board (ID#2000030222).
Study Sample
We identified all SARS-CoV-2 RT-PCRs collected among vaccine eligible people (ages ≥16 years) in the YNHH EHR between April 1 and August 24, 2021. We excluded tests from people who received a vaccine prior to state distribution (December 14, 2020), had a previous positive SARS-CoV-2 RT-PCR or rapid antigen test, or had missing covariate information (see Statistical Analysis). Additionally, we excluded tests that were performed after receiving a booster (3rd) dose or an Ad26.COV2 vaccine dose.
Outcome (Case) Definition and Control Selection
Our primary outcomes of interest were WGS-classified Alpha-, Delta-, and Other-variants infection, symptomatic infection, and COVID-19 associated hospitalizations. We defined infection as positive, sequenced specimens that passed quality control. Infections were categorized as Alpha, Delta, or Other-Variant based on their WGS classification, where the Other-Variant category comprised all non-Alpha, non-Delta samples. Controls were defined as negative RT-PCRs. We selected up to three negative tests (controls) per person. If an individual had more than one negative test within a seven-day period, one random test was selected.
Symptomatic SARS-CoV-2 infection was defined as the subset of SARS-CoV-2 infections identified among symptomatic people (people with at least one NLP-captured COVID-19-related symptom recorded within 0-14 days prior to or following testing). We defined COVID-19 associated hospitalizations as the subset of symptomatic SARS-CoV-2 infections collected in the 21 days prior to or three days following hospitalization. Controls for these outcomes were selected among negative RT-PCR test results from symptomatic people.
Our secondary outcomes were calendar period-classified Alpha and Delta infections. Specifically, we defined Alpha and Delta infections as positive RT-PCR tests collected during periods when the variant comprised ≥50% of sequenced samples from Connecticut deposited in the global initiative on sharing all influenza data (GISAID; https://www.gisaid.org).[27,28] Alpha accounted for ≥50% of sequenced samples from the beginning of the study (April 1) through May 28, 2021.[27,28] Delta contributed ≥50% of the sequenced samples beginning on June 27, 2021.[27,28] Controls were then limited to negative RT-PCRs collected during the period used for variant classification.
Statistical Analysis
We visually summarized the number of infections, or positive RT-PCRs collected in the absence of a positive test in the previous 90 days, recorded in the YNHH between April 1 and August 24, 2021, and vaccine coverage among selected cases and controls by day. The number of sequenced samples that passed quality control were visually summarized by day and variant classification.
Vaccine Effectiveness Against WGS-Classified SARS-CoV-2 Outcomes
We estimated the association between mRNA COVID-19 vaccination (<14 days, 14-89 days, 90-149 days, and ≥150 days since 2nd dose) and WGS-classified Alpha, Delta, and Other-Variant infection, symptomatic infection, and COVID-19 associated hospitalization using generalized additive multinomial, logistic regressions. We included the following a priori selected covariates: date of test (continuous), age (continuous), sex, race/ethnicity, Charlson comorbidity score [29] (continuous), number of non-emergent YNHH encounters in the year prior to vaccine rollout in Connecticut (December 2020; categorized as 0, 1-2, 3-4, 5+), insurance group (uninsured, Medicaid, Medicare, other), social vulnerability index of residential zip code (continuous) and residential county. Continuous factors were modeled using a natural spline with 3 knots.[30,31] From the model we estimated vaccine effectiveness as one minus the odds ratio (OR) of infection comparing vaccinated to unvaccinated people.
Duration of Protection from Primary Vaccination
We tested for declines in the level of protection over time by comparing the odds of infection among recently vaccinated people (14-89 days since 2nd dose) to the odds of infection among people who received their 2nd dose 90-149, and ≥150 days prior to testing.[32] We evaluated this association using a generalized additive logistic regression and accounted for the same confounders as in the WGS-classified vaccine effectiveness analysis.
Calendar Period-Classified, Variant-Specific Vaccine Effectiveness
We estimated period defined variant specific vaccine effectiveness using generalized additive logistic regressions and included the same confounders as in the WGS-classified vaccine effectiveness analysis. All analyses were conducted in R, version 4.1.2.[33]
Sensitivity Analyses
We performed multiple sensitivity analyses testing the robustness of our findings to alternative study design, data cleaning and modeling assumptions. Specifically, we examined the following scenarios: matched analysis (1:4 matching with replacement) and various quality control definitions. Additionally, we estimated vaccine effectiveness and tested for differences in the level of protection during periods when variants were co-circulating. Finally, to ensure that any differences observed between the period-classified and WGS-classified effectiveness estimates were not the result of bias introduced through the selection of sequenced samples, we performed the period-classified analysis among sequenced samples. For a detailed description, see Supplement: Sensitivity Analyses.
Results
The regions served by YNHH experienced two successive waves of SARS-CoV-2 infections (or positive RT-PCRs) between April 1 and August 24, 2021 (Figure1.A). According to the 4,125 samples available for sequencing during this period (1,076 of which failed quality control and were omitted), the first wave was comprised of Alpha and Other-Variant (non-Alpha/Delta) infections and the second was predominantly comprised of Delta infections (Figure1.B). The proportion of tests collected among vaccinated people increased during April but remained around 50% for the rest of the study period (Figure1.C).
The trends in (A) SARS-CoV-2 infections, (B) whole genome sequence (WGS) defined variant specific infections that passed quality control overlayed on the Alpha and Delta predominant periods (Alpha: April 1 – May 28, 2021, Delta: June 27 – August 24, 2021), and (C) mRNA vaccination status over the study period (April 1, 2021, through August 24, 2022). SARS-CoV-2 infections were defined as positive RT-PCRs collected among people without a prior positive SARS-CoV-2 RT-PCR recorded in the Yale New Haven Health System. Infections were stratified by if they were sequenced (dark grey) or not (light grey). Variant predominant periods were defined as periods where the variant comprised at least 50% of CT samples in GISAID.
Population and Sample
Between April 1 and August 24, 2021, 502,618 RT-PCRs were collected among 268,045 vaccine eligible people. Following the restriction to RT-PCRs that met the inclusion criteria (n=441,356 tests among 241,654 people), the sample contained 10,349 positives (cases), 2,565 (25%) of which were sequenced (1,560 sequenced samples did not meet our studies inclusion criteria) and 2,029 passed quality control. From the 431,007 negative RT-PCRs that met our inclusion criteria, we randomly selected up to three per person, resulting in the inclusion of 343,727 negative RT-PCRs as controls (Figure2).
The sample was limited to viral RNA samples collected between April 1 and August 24, 2021, among vaccine eligible individuals (people aged 16 years or greatera) and had recorded confounder data (2,032 and 64 tests were collected among people without SVI and sex data, respectively). Whole genome sequencing was performed on available specimens and variant calls were made based on the frequency of variant defining mutations. The other-variant classification incorporated all non-Alpha/Delta samplesc. Period based variant classifications were defined as cases collected during variant dominate periods, or periods when at least 50% of sequenced samples deposited in GISAID were that variant.[28] The Alpha dominate period was April 1 through May 28, 2021 and the Delta dominate period was July 10 through August 24, 2021d. People were allowed to contribute up to three negative tests to the control samplee.
Cases (positive RT-PCRs), sequenced cases and controls were similar with respect to age, gender, SVI of residential zip code and Charlson comorbidity score. However, a larger proportion of controls occurred among non-Hispanic White people (61% of controls vs 40% of Alpha infections, 49% of Delta infections, and 41% of Other-Variant infections). Among vaccinated people, the median time between 2nd dose administration and testing was shorter for Alpha cases (55 days [1st-3rd Quartile (1-3Q): 25-66 days]) than Other-Variant cases (85 days [1-3Q: 54-127 days]) and Delta cases (135 days [1-3Q: 110-169 days], Table1).
Vaccine Effectiveness Against WGS-classified SARS-CoV-2 Outcomes
The effectiveness of primary mRNA vaccination 14-89 days after 2nd dose administration was 84.4% (95% Confidence Interval [CI]: 75.6-90.0%) and 68.9% (CI: 58.0-77.1%) against Alpha and Delta infection, respectively, and 85.5% (CI: 65.5-93.9%) and 89.0% (CI: 52.9-97.4%) against Alpha and Delta associated hospitalizations, respectively (Figure3.A/C). The effectiveness of primary vaccination did not differ between Alpha-, Delta-, and Other-Variant classified infections for people who received their second dose <14 days prior to testing. However, the level of protection offered against Alpha infection was significantly higher than that offered against Delta infection for people who received their second dose 14-89 days prior to testing (Pvalue: 0.013). Only one Alpha infection was recorded among people vaccinated ≥90 days prior to testing and we were unable to reliably compare the level of protection ≥90 days after second dose receipt. The level of protection offered against Other-Variant infection was lower than against Alpha infection and higher than Delta infection but was not significantly different from either. The level of protection did not vary significantly by variant for symptomatic infection or COVID-19 associated hospitalization (Figure2).
The effectiveness of primary mRNA vaccination against variant defined (A) SARS-CoV-2 infection, (B) symptomatic infection, and (C) COVID-19 associated hospitalization. (A) SARS-CoV-2 infection was defined as a RT-PCR positive, sequenced viral RNA samples collected among a vaccine eligible person (≥16 years old at testing) and passed quality control (≥65% bases covered at 100x). (B) Symptomatic infection was defined as a SARS-CoV-2 infection collected within 12 days of a record of COVID-19 symptoms (captured using a natural language processor). (C) COVID-19 associated hospitalization was defined as a symptomatic infection that was collected within the three days after to or 21 prior to hospitalization. Cases were defined as Alpha, Delta, and Other-Variant based on whole genome sequencing. The other-variant classification incorporated all non-Alpha/Delta infections. Rows containing no cases were omitted. *The levels of protection offered against the examined variants were compared and significance was defined with an alpha of 0.05. (RT-PCR: reverse transcription polymerase chain reaction)
Duration of Protection from Primary Vaccination
The odds of Delta infection and symptomatic infection were significantly higher 90-149 and ≥150 days after 2nd dose administration than 14-89 days after administration (e.Table2). However, the odds of Delta associated hospitalization, as well as Alpha and Other-Variant infection, symptomatic infection, or COVID-19 associated hospitalization, did not increase significantly. The precision around these estimates was, however, low.
Calendar Period-Classified, Variant-Specific Vaccine Effectiveness
A total of 6,200 cases and 168,248 controls were collected during the Alpha predominant period (April 1-May 28, 2021). During the Delta predominant period (June 27-August 24, 2021), 3,882 cases and 128,926 controls were collected (Figure2). Cases collected during the variant predominant periods were similar to sequenced cases with respect to demographic and clinical factors. However, the median time between second dose receipt and calendar-period classified Alpha infections was shorter (37 days) than WGS-classified Alpha infections (55 days, eTable3).
Among the samples collected during the Alpha predominant period, 65% were Alpha, 0.9% were Delta, and 34.5% were Other-Variant. Conversely during the Delta period, 96% of sequenced samples were Delta (eTable4). The effectiveness of primary vaccination 14-89 days after 2nd dose administration was 88.2% (95% CI: 86.3-89.8%) and 64.6% (95% CI: 58.0-70.4%) during the period of Alpha and Delta predominance, respectively (Figure4).
The effectiveness of primary mRNA vaccination against variant defined SARS-CoV-2 infection. Infection was defined as a RT-PCR positive samples collected among a vaccine eligible person (≥16 years old at testing). Infections were classified based on periods of variant predominance (≥50% of Connecticut samples deposited in GISAID). The Alpha dominate period was April 1 through May 28, 2021 and the Delta dominate period was July 10 through August 24, 2021.
Sensitivity Analyses
The effectiveness of primary vaccination (14-89 days after 2nd dose administration) ranged between 81.5-89.9% against Alpha and 67.3-70.4% against Delta in the sensitivity analyses (eFigure1-7/eTable5). Under each examined scenario, the level of protection ≥90 days after 2nd dose administration was significantly higher against Alpha than against Delta. The effectiveness of vaccination ≥14 days after 2nd dose administration was significantly higher against Alpha than Delta for all examined scenarios except when quality control was defined at 100% bases covered at 100x (eFigure2). The level of protection offered against Other-Variant infection was higher than that offered against Delta infection for the matched and quality control defined as 50% bases covered at 100x analyses (e.Figure1/3). The effectiveness against calendar-period classified infections were similar following the restriction to sequenced samples (14-89 days Alpha: 84.4% [95% CI: 77.1-89.4%]; Delta: 69.3% [95% CI: 58.6-77.2%], e.Table5) as the primary analysis (14-89 days Alpha: 88.2% [95% CI: 86.3-89.8]; Delta: 64.6% [95% CI: 58.0-70.1%], Table2).
Discussion
In this retrospective analysis, we estimated the effectiveness of primary mRNA COVID-19 vaccination against WGS-classified, variant-specific SARS-CoV-2 infection, symptomatic infection, and COVID-19 associated hospitalization. We found that mRNA COVID-19 vaccines provided less protection against Delta infection than Alpha infection and that the level of protection offered against Delta infection declined significantly over time since vaccine administration. However, we did not observe a significant difference in the level of protection offered against variant-specific symptomatic infection or hospitalization.
These findings build upon existing literature suggesting that the increase in SARS-CoV-2 cases among vaccinated people during the Delta wave of fall 2021 was the result of variant-specific immune evasion and waning levels of vaccine protection.[3–5,10–17,34–38] However, while there is ample evidence of both phenomena during the Delta period [3–5,10–17,34–38], few studies have disentangled the two processes.[3–5,10–17,34–38] Further, even when their cooccurrence was reported, the authors have typically concluded that one of the two phenomena was the primary driver of the reduced effectiveness.[13,14,16] In contrast, we, in alignment with Britton et al., observed strong evidence for both declining levels of protection and variant-specific immune evasion.[18]
While many factors likely contribute to the variability of vaccine effectiveness estimates in the literature, differences in variant classification methodology are a potential component.[39] To examine how the vaccine effectiveness estimates from calendar-period-based variant classifications compares to WGS, we estimated the effectiveness of primary vaccination against Alpha and Delta cases defined by periods of variant predominance. We found that estimates of vaccine effectiveness against SARS-CoV-2 infection during periods of Alpha and Delta predominant were similar to those estimated using WGS.
These findings are not surprising. While there was a high degree (35% of sequenced samples) of misclassification during the Alpha predominance period, effectiveness against Alpha and other-variant (the major variant category co-circulating during the Alpha predominant period) infection did not vary significantly in the WGS analysis. Thus, we would not expect that this degree of misclassification would significantly alter vaccine effectiveness estimates. However, this finding is unlikely to hold under situations with similarly large amounts of misclassification and underlying differences in variant-specific vaccine effectiveness.
Unlike Alpha, data from the WGS analysis suggests that the effectiveness against Delta infection compared to other-variant infections may differ (differed significantly for multiple sensitivity analyses but not primary analysis). As a result, a large amount of variant misclassification during the Delta predominant period may impact vaccine effectiveness estimates. However, Delta rapidly became the predominant variant accounting for 50% of samples from Connecticut in GISAID on June 27, 2021, 75% on July 10, 2021, and 100% on September 6, 2021.[27,28] It is, thus, unsurprising that the period-classified vaccine effectiveness estimates mirrored the WGS-classified estimates.
These findings suggest that, even when the effectiveness of the variants is expected to differ, calendar-period-based variant classification may yield reliable estimates, when there is limited superimposition of variant waves, and the proportion of misclassified cases is expected to be low. However, under scenarios when variants or subvariants are co-circulating, as is the case with the Omicron subvariants, calendar-period-based classification provides limited utility. Not only would subvariant calendar-period-based classification result is meaningful amounts of misclassification, but it also prevents researchers from comparing the effectiveness of variants as they co-circulate. As a result, the need for WGS in future vaccine effectiveness and severity analysis has only increased.
Limitations
Our study was subject to several limitations. First, specimens available for sequencing were limited. However, the storage and sequencing of specimens was considered random and, while it reduced our power, this restriction was unlikely to have introduced bias. Further, our sensitivity analysis limiting the calendar-period classified infections to sequenced samples mirrored the main analysis. Second, to classify variants using WGS we had to exclude cases that did not meet our quality control definition. While this further reduced our sample, we show that this restriction was unlikely the driver of our results in sensitivity analyses. Third, sequenced samples have lower CT values than those for which sequencing is unsuccessful. For this reason, the vaccine effectiveness estimates against infection may be biased towards more severe illness, which in turn may impact the generalizability of our findings. Finally, we had few to no Alpha outcomes for the ≥90 days after 2nd dose administration categories and were unable to provide reliable estimates.
Conclusion
In this study, the use of WGS revealed differences in variant-specific COVID-19 vaccine effectiveness, with the effectiveness of primary mRNA vaccination against Delta infection being significantly lower than that against Alpha infection. However, the effectiveness of vaccination was found to be moderate against symptomatic COVID and high against COVID-19 associated hospitalization regardless of variant. Although we observed broad agreement between estimates of variant-specific vaccine effectiveness when WGS and calendar-period were used to define Alpha and Delta variants, there was a significant degree of misclassification associated with calendar-period classifications, which may limit its application in future SARS-CoV-2 epidemic waves and in settings with heterogeneity in variant-specific vaccine effectiveness.
Data Availability
Data cannot be shared publicly because of the presence of potentially identifiable health information. Requests for access can be directed to the corresponding author and Yale Human Research Protection Program (hrpp{at}yale.edu).
Data Use Agreement
Data cannot be shared publicly because of the presence of potentially identifiable health information. Requests for access can be directed to the corresponding author and Yale Human Research Protection Program (hrpp{at}yale.edu).
Funding
Funding for the Studying COVID-19 Outcomes after SARS-CoV-2 Infection and Vaccination (SUCCESS) Study was provided by Regeneron Pharmaceuticals, Inc. (CLINICAL COLLABORATION STUDY AGREEMENT R0000-COV-CES-2116), the Beatrice Kleinberg Neuwirth and Sendas Family Funds and the Yale Schools of Public Health and Medicine. Some of the study results were obtained with the support of a research grant from the Investigator-Initiated Studies Program of Merck Sharp & Dohme Corp (Title: Harnessing Large Cohorts and a Rapid Knowledge Pipeline to Elucidate Immunity to SARS-CoV-2 Infection; MISP Database No. 60487).
Author Contributions
Margaret L. Lind, Albert I. Ko and Wade L. Schulz have full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Lind, Hitchings, Schulz, Ko, Cummings, Hamilton, McCarthy, Copin
Acquisition and interpretation of data: Lind, Warner, Coppi, Duckwall, Ferguson, Coppi
Drafting of the manuscript: Lind, Hitchings, McCarthy, Copin, Ko
Critical revision of the results: All authors.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Lind
Administrative, technical, or material support: Duckwall, Borg
Supervision: Ko, Hitchings, Schulz, Cummings, Hamilton
Conflict of Interest Disclosures
AIK serves as an expert panel member for Reckitt Global Hygiene Institute, scientific advisory board member for Revelar Biotherapeutics and a consultant for Tata Medical and Diagnostics and Regeneron Pharmaceuticals, and has received grants from Merck, Regeneron Pharmaceuticals and Tata Medical and Diagnostics for research related to COVID-19. W.L.S. was an investigator for a research agreement, through Yale University, from the Shenzhen Center for Health Information for work to advance intelligent disease prevention and health promotion; collaborates with the National Center for Cardiovascular Diseases in Beijing; is a technical consultant to Hugo Health, a personal health information platform, and co-founder of Refactor Health, an AI-augmented data management platform for healthcare; and has received grants from Merck and Regeneron Pharmaceutical for research related to COVID-19. Other authors declare no conflict of interest. JDH, RC, SM, DF, SH, and AZ are employees and shareholders of Regeneron Pharmaceuticals, Inc
Supplement
eTable1: Variant call distribution by Percent Bases Covered (100x)
eTable2: Risk of Variant Specific SARS-CoV-2 Infection among Vaccinated People*, According to Time after Receiving a Second Primary Vaccine Dose
eTable3: Misclassified Alpha and Delta SARS-CoV-2 Cases Define using Calendar Period
Sensitivity Analyses
eFigure1: Forest Plot of Vaccine Effectiveness (Matching)
eFigure2: Forest Plot of Vaccine Effectiveness (Quality Control – 100% bases covered at 100x)
eFigure3: Forest Plot of Vaccine Effectiveness (Quality Control – 50% bases covered at 100x)
eFigure4: Number of Sequenced Samples by Variant 21signation and Date (Daily Count and 5-Day Rolling Average)
eFigure5: Forest Plot of Vaccine Effectiveness (Alpha and Delta Cocirculating period: May 13 – August 5, 2021)
eFigure6: Forest Plot of Vaccine Effectiveness (Alpha and Other Cocirculating period: April 2 – August 5, 2021)
eFigure7: Forest Plot of Vaccine Effectiveness (Delta and Other Cocirculating period: May 13 – August 21, 2021)
eTable 4: Effectiveness of Primary Vaccination Against Calendar Period Defined SARS-CoV-2 Variant Infections (Sequenced Only Cases)
Sensitivity Analyses
We tested the robustness of our findings to multiple alternative data cleaning and study designs. Under each examined scenario, we estimated vaccine effectiveness against variant specific SARS-CoV-2 infection. Each analysis mirrored the primary analyses beyond the stated changes.
Matching (1:4 with replacement)
To allow for a single analytic sample from which to perform our analyses, we did not perform a match for our primary analysis. However, our fully adjusted (un-matched) model may suffer from positivity violations. To test if matching resulted in increased precision, we performed a 1:4 match with replacement on date of test (+/- 7 day), age group, and county. All cases successfully matched to controls.
Varied quality control definition
In the primary analysis we limited to samples that passed a quality control definition of >=65% bases covered at 100x. This quality control definition was selected as it correlated with an inclusion of a 5% potential error rate, or 5% of the included samples not having a variant call. Here we present the results from alternative quality control definitions:
100% bases covered at 100x
50% bases covered at 100x
At a threshold for inclusion of 100% bases covered (at 100x), seven sequenced samples are included without a variant designation (0.8% potential error rate) (see eTable1)
At a threshold for inclusion of 50% bases covered (at 100x), 79 sequenced samples are included without a variant designation (8.8% potential error rate).
Co-circulation time restricted vaccine effectiveness
In the primary analysis, we included all samples and study time into a single model and modeled calendar time flexibly. While this allowed for maximum power, the variants groups did not co-circulate the full time and our results may be impacted unaccountable, time-dependent, factors (such as changes in testing). Here, we compare the level of vaccine offered protection during periods of known variant co-circulation. We defined known circulation as the period for which we had both variants present in our included sample (based on first and last observed variant).
Alpha and Delta: May 13 – August 5, 2021
Alpha and Other: April 2 – August 5, 2021
Delta and Other: May 13 – August 21, 2021
Period defined variants limited to sequenced samples
In the primary analysis, we estimated variant specific vaccine effectiveness using whole genome sequencing (WGS) and period designations. For the latter, we included all positive RT-PCRs that met our inclusion criteria (regardless of if they were sequenced). Because bias may have been introduced by the selection of specimens available for sequencing for the WGS analysis (primary), any differences we observed between these variant designation approaches may be the result of that bias. Here, we restrict the period defined analysis to sequenced samples.
References
