ABSTRACT
BACKGROUND The current understanding of the long-term effectiveness of the BNT162b2 vaccine across diverse U.S. pediatric populations is limited. Using data from the PEDSnet collaboration, we assessed the effectiveness of BNT162b2 against various strains of the SARS-CoV-2 virus.
METHODS We emulated three target trials to assess the real-world effectiveness of BNT162b2: adolescents aged 12 to 20 years during the Delta variant period (Target trial 1), children aged 5 to 11 years (Target trial 2) and adolescents aged 12 to 20 years during the Omicron variant period (Target trial 3). The outcomes included documented infection, COVID-19 illness severity, admission to an intensive care unit (ICU), and two cardiac-related outcomes, myocarditis and pericarditis. We implemented a novel trial emulation pipeline accounting for possible misclassification bias in vaccine documentation in EHRs.
RESULTS During the Delta period, the BNT162b2 vaccine demonstrated an overall effectiveness 98.4% against documented infection among adolescents, with no significant waning after receipt of the first dose. During the Omicron period, the overall effectiveness in preventing documented infection among children was estimated to be 74.3%. Higher levels of effectiveness of 75.5% and 84.9% were observed against moderate or severe COVID-19 and ICU admission with COVID-19, respectively. In the adolescent population, the overall effectiveness in preventing documented Omicron infection was 85.5%, with effectiveness of 84.8% against moderate or severe COVID-19, and 91.5% against ICU admission with COVID-19. The effectiveness of the BNT162b2 vaccine against the Omicron variant declined after 4 months following the first dose and then stabilized. Across all three cohorts, the risk of cardiac outcomes was approximately 65% to 85% lower in the vaccinated group than that of the unvaccinated group.
CONCLUSIONS Our study suggests that BNT162b2 is effective for various COVID-19-related outcomes in children and adolescents during Delta and Omicron periods, with lower cardiac risk. Waning effectiveness indicates potential need for future revaccination.
INTRODUCTION
The Food and Drug Administration (FDA) expanded the emergency use authorization of the BNT162b2 mRNA COVID-19 vaccine (Pfizer-BioNTech) to 12-15-year-olds on May 10, 2021, and to 5-11-year-olds on October 29, 2021. As of April 5, 2023, the Centers for Disease Control and Prevention (CDC) reports indicate that 40% of U.S. children aged 5-to-11-year-olds and 72% of adolescents aged 12-to-18-year-olds had received at least one dose of the vaccine. The emergence of the Omicron variant (B.1.1.529) and its subvariants in early 2022 led to a new surge in COVID-19 cases worldwide1. The randomized trials of the BNT162b2 vaccine which demonstrated high efficacy of 2 doses against COVID-19 (100% and 91% among those aged 12-15 and 5-11 years, respectively) were conducted before the emergence of the Omicron variant2, 3.
Several observational studies have been conducted to investigate the effectiveness of vaccination in real-world settings4–8. However, prior studies have had limited follow-up periods, covering the Delta variant or earlier subvariants of Omicron periods only. Studies evaluating the Omicron variant have only assessed the short-term effects of the vaccine, with only one study involving children evaluating the effect beyond 3 months9. There is limited information on the long-term durability of vaccine protection during the Omicron period. Few existing studies on U.S. pediatric populations have covered both hospitalized patients and those with mild or asymptomatic conditions. Furthermore, while studies have acknowledged limitations due to misclassification in vaccination status in real-world effectiveness studies, none have rigorously evaluated the impacts of such misclassification nor accounted for the potential bias it may introduce.
To address these gaps in our knowledge of the pediatric effectiveness of SARS-CoV-2 vaccination, we designed this study to assess the real-world effectiveness of BNT162b2 among children and adolescents during the Delta and Omicron variant-predominant periods using electronic health record (EHR) data from a national network of U.S. pediatric medical centers. Our study used a trial emulation design and adjusted for misclassification issues in vaccination status and has several attractive features that strengthen credibility of our inference. First, it is the largest study to date in the U.S. estimating vaccine effectiveness in children and adolescents, covering a broad spectrum of the U.S. pediatric population. Second, the study examined the effectiveness against infection over a longer follow-up period than any previous study, enabling evaluation of the durability of vaccine protection. Third, the study included a diverse representation of U.S. pediatric populations from primary care, specialty care, emergency department, testing centers, and inpatient settings. Fourth, the study was the first to account for the incomplete capture of vaccination status by health systems in the U.S. Finally, besides infection and severe disease endpoints, we also studied the effect of vaccination on the incidence of myocarditis and pericarditis to assess the effect of vaccination relative to this known potential risk.
METHODS
DATA SOURCES
This study used EHR data from PEDSnet10, which is a national collaboration of pediatric health systems that share EHR data, conduct research, and improve outcomes together. Participating institutions in this study included: Children’s Hospital of Philadelphia, Cincinnati Children’s Hospital Medical Center, Children’s Hospital Colorado, Ann & Robert H. Lurie Children’s Hospital of Chicago, Nationwide Children’s Hospital, Nemours Children’s Health System (inclusive of the Delaware and Florida health system), Seattle Children’s Hospital, and Stanford Children’s Health. Data were extracted from the PEDSnet COVID-19 Database Version Week 14111.
SPECIFICATION AND EMULATION OF TARGET TRIALS
Target trials are hypothetical randomized controlled trials (RCT) that guide the design of observational studies to assess the real-world effectiveness of treatments12. We designed and conducted emulation of three target trials to investigate the effectiveness of the BNT162b2 vaccine in preventing infection with various strains of the SARS-CoV-2 virus in children and adolescents in the United States (Table 1). The three target trials focused on documented SARS-CoV-2 infection and outcomes in:
Target trial 1 (Delta study in adolescents): adolescents aged 12 to 20 years during the period when the Delta variant was prevalent from July 1, 2021, to November 30, 2021.
Target trial 2 (Omicron study in children): children aged 5 to 11 years during the period when the Omicron variant was prevalent from January 1, 2022, to November 30, 2022.
Target trial 3 (Omicron study in adolescents): adolescents aged 12 to 20 years during the period when the Omicron variant was prevalent from January 1, 2022, to November 30, 2022.
The design of target trials and trial emulation procedures in real-world data are summarized in Table 1.
Eligibility criteria included age of 5 to 11 years for children or 12 to 20 years for adolescents at the start of the study period and no previous COVID-19 vaccination or documented SARS-CoV-2 infection. Additionally, participants were required to have a prior encounter (including telephone or telehealth encounters) within 18 months of cohort entrance to ensure that they had an ongoing interaction with the health system.
The intervention of interest was vaccination, in comparison with no receipt of any type of COVID-19 vaccine. Since the BNT162b2 vaccine covered more than 85% of documented vaccinations among children and adolescents in the PEDSnet database, in this study we focused primarily on studying the effectiveness of this vaccine, although the supplementary appendix reports a sensitivity analysis investigating all types of reported COVID-19 assessed in the U.S., with 85.6% BNT162b2, 1.2% mRNA-1273, and 12.4% unspecified COVID-19 vaccine.
In the emulation of the target trials, the cohort entrance date for the intervention group was defined as the date of the first dose of the BNT162b2 vaccine, while for the comparator group, it was a randomly selected date from visits, chosen to ensure the distribution of index dates for the control group matched the distribution of index dates for the vaccination group to control for time effects. The risk period for the study began 28 days after the index date such that infections within 28 days were excluded.
Randomized trials achieve balance across potential confounders by randomly allocating the treatment to intervention and comparator groups. In our trial emulation, we balanced the intervention and comparator groups by adjusting for a large number of measured confounders using propensity score stratification13. We built the propensity score model based on demographic factors including age, sex, race/ethnicity, clinical factors including obesity status, a chronic condition indicator as defined by the Pediatric Medical Complexity Algorithm (PMCA)14, and a list of pre-existing chronic conditions, and healthcare utilization factors including the number of inpatients, outpatients, ED visits, unique mediations, and the number of negative COVID-19 tests prior to the cohort entry. We stratified the patients into propensity score quintiles based on these factors. We refer to Table S1 in the Supplementary Appendix for detailed definitions of study variables.
The four COVID-19 outcomes of interest were: documented SARS-CoV-2 infection, mild COVID-19, moderate/severe COVID-19, and ICU admission with COVID-19. We did not evaluate death from COVID-19 as it was too rare among children and adolescents to study quantitatively. In the emulation of target trials, SARS-CoV-2 infections were defined by and occurrence of positive polymerase-chain-reaction (PCR), serology, or antigen tests or diagnoses of COVID-19, post-acute sequelae of SARS-CoV-2 (PASC), or multisystem inflammatory syndrome in children (MIS-C) regardless of the presence of symptoms. Classification of mild, moderate, or severe COVID-19 infections was defined based on the severity classification of COVID-19 in Forrest et al. (2022)15. ICU admission with COVID-19 was defined by any ICU visit 7 days prior to 13 days after documented SARS-CoV-2 infection. Additionally, we also considered the clinical outcomes of myocarditis or pericarditis incidence to evaluate the effect of the BNT162b2 vaccine in terms of cardiac outcomes.
STATISTICAL ANALYSIS
We evaluated covariate balancing after propensity score stratification by plotting the standardized mean differences (SMD) between variable values for the vaccinated and unvaccinated groups, with a difference of 0.1 or less indicating an acceptable balance. We used Poisson regression to estimate the relative risk between two treatment arms for the risk of each outcome while adjusting for different follow-up lengths among participants. Since immunization records are often captured and stored across multiple disconnected sources, resulting in incomplete vaccination records in patients’ EHRs, we mitigated the potential bias arising from this measurement error by incorporating an integration likelihood of the Poisson regression with a pre-specified range of misclassification rates. The vaccine effectiveness was defined as 100*(1-relative risk). The details of statistical methods are described in Section S1 of the Supplementary Appendix.
We conducted secondary analyses stratified by 2-month intervals since receipt of vaccination to investigate the durability of vaccine protection. Subgroup analyses were also performed to investigate differences in vaccine protection according to age groups (5-to-8, 9-to-11, 12-to-15, 16-to-20).
SENSITIVITY ANALYSES
Extensive sensitivity analyses were conducted to evaluate the robustness of the research findings; see Supplementary Appendix sections S3-10 for the impacts of cohort design. In scenarios in which any categorical covariates were unbalanced (with a standardized mean difference>0.1), we included a sensitivity analysis excluding participants in that category. Since the proportion of patients entered with ED visits is relatively low in the vaccinated cohort compared to the unvaccinated, we conducted a sensitivity analysis excluding all participants who entered the cohort due to an ED visit. We studied the dose-response of BNT162b2 by defining a single-dose vaccination as one-dose 28 days prior to infection and two-dose vaccination as the receipt of a second dose 14 days prior to infection. To further evaluate the risk of cardiac outcomes associated with vaccination, we estimated the vaccine effects on myocarditis and pericarditis stratified by gender and age groups. Residual study bias from unmeasured and systematic sources can still exist in observational studies after controlling for measured confounders; thus, we conducted negative control outcome experiments13, 16–17, where the null hypothesis of no effect was believed to be true using 40 negative control outcomes pre-specified by pediatric physicians. The empirical null distribution and calibrated effectiveness were reported as sensitivity analyses. We also reported the estimated vaccine effectiveness from all brands of COVID-19 vaccines.
MISSINGNESS IN VACCINE RECORDS
Vaccine status may be missing for individuals whose vaccine doses were administered by a site outside of the PEDSnet network care delivery sites. It is likely that patients recorded as vaccinated in the EHR are true positives, so specificity could be very high, but sensitivity would be reduced by undocumented vaccinations (false negatives). To account for potential bias from the misclassification of vaccination status, a range of possible sensitivities based on our prior study was pre-specified for each study. The sensitivity range was considered to be 0.8 to 1 for the study involving children and 0.7 to 0.9 for the studies involving adolescents. By accounting for misclassification and specifying a range of sensitivity, the study aimed to minimize the impact of bias caused by the misclassification in the estimation of the effectiveness of the COVID-19 vaccine among children and adolescents. The details of statistical methods are described in Section S1 of the Supplementary Appendix.
To further evaluate the robustness of statistical methods we used to account for misclassification, we varied the trial emulation procedure by considering various methods for bias correction, including the naive method (without adjusting for misclassification), using different ranges of misclassification rates, and using a fully Bayesian method19. To evaluate the impact of differential misclassification on effectiveness estimates, we conducted sensitivity analyses simulating vaccination status according to various differential misclassification scenarios. Results from these sensitivity analyses are summarized in section S11-13 of the Supplementary Appendix.
RESULTS
STUDY POPULATION
A total of 77,392 adolescents within the PEDSnet network were identified to study the effectiveness of vaccination against Delta infection and severe outcomes (see Table 2A for baseline characteristics). 111,539 children and 56,080 adolescents were included in the cohort to study the effectiveness of vaccination against the Omicron infections (see Table 2B for baseline characteristics). The vaccinated and unvaccinated groups had a slightly unbalanced distribution of testing rates before cohort entry across all three cohorts. After propensity-score stratification, all covariates were well balanced between vaccinated and unvaccinated groups with an SMD smaller than 0.1 in the Omicron study involving children (Figure S1) and involving adolescents (Figure S2). In the study evaluating vaccine effectiveness for adolescents during the Delta period, one site remained unbalanced after propensity-score stratification with an SMD larger than 0.1, and thus a sensitivity analysis was conducted by excluding participants from this site which gave consistent results with the primary analysis (see Figure S3 and Section S4 in Supplementary Appendix).
VACCINE EFFECTIVENESS
Table 3A summarizes the estimated vaccine effectiveness in three target-trial emulations and Figure 2 shows the durability of protection. The vaccine effectiveness was estimated to be 98.4% (95% CI, 98.1 to 98.7) among adolescents in the Delta period, 74.3% (95% CI, 72.2 to 76.2) against documented infection among children in the Omicron period, and 85.5% (95% CI, 83.8 to 87.1) among adolescents in the Omicron period. During the Delta period, the vaccine effectiveness against documented infection remained stable throughout the follow-up period of the study. After 4 months following the first dose, vaccine effectiveness against documented infection with Omicron declined from 82.3% (95% CI, 77.9 to 85.8) to 70.6% (95% CI, 65.9 to 74.6) among children, and from 91.3% (95% CI, 87.6 to 94.0) to 82.9% (95% CI, 79.0 to 86.1) among adolescents. Although vaccine effectiveness against documented infection stabilized after this initial decline, the corresponding confidence intervals were much wider indicating higher levels of uncertainty.
VACCINE EFFECTIVENESS: SEVERE ILLNESS AND COMPLICATIONS
During the Delta period, the vaccine was found to have high effectiveness against documented infection 98.4% (95% CI, 98.1, 98.7) and severe COVID-19 outcomes. The estimated relative risk reduction in percentage of the vaccine on myocarditis was 73.8% (95% CI, 43.8, 87.6) and pericarditis was 66.1% (95% CI, 14.8, 86.5). The estimated vaccine effectiveness against the Omicron variant in children was 73.5% (95% CI, 69.2 to 77.1) against mild COVID-19, 75.5% (95% CI, 69.0 to 81.0) against moderate or severe COVID-19, and 84.9% (95% CI, 64.8 to 93.5) against ICU admission with COVID-19. The estimated relative risk reduction in percentage of the vaccine on myocarditis was 71.4% (95% CI, 8.8, 91.0) and pericarditis was 69.1% (95% CI, -19.7, 92.0). In the Omicron study in adolescents, the vaccine effectiveness was estimated to be 87.0% (95% CI, 83.5 to 89.8) against mild COVID-19, 84.8% (95% CI, 77.3 to 89.9) against moderate or severe COVID-19, and 91.5% (95% CI, 69.5 to 97.6) against ICU admission with COVID-19. The estimated relative risk reduction in percentage of the vaccine on myocarditis was 82.0% (95% CI, 46.6, 94.0) and pericarditis was 83.7% (95% CI, 43.3, 95.3) in this cohort.
SENSITIVITY ANALYSES AND ADDRESSING MISCLASSIFICATION BIAS
Table S9 presents the proportion of vaccinated participants with single-dose and two-dose vaccination in three studies. Table 3B summarizes the single-dose and two-dose vaccine effectiveness indicating the two doses of the BNT162b2 vaccine are slightly more effective against the Omicron variant while much more effective against the Delta variant than one dose. Section S9 presents negative control experiments of three target-trial emulations using 40 negative control outcomes. After accounting for systematic error through calibration using negative control outcomes, our findings indicate a slight shift in point estimates accompanied by wider confidence intervals. This suggests the presence of a minor degree of systematic error, as well as additional uncertainty characterized by the estimated distribution derived from the negative control outcomes.
Section S11 shows effectiveness estimated from the naive method and proposed trial emulation pipeline with different ranges of sensitivity of vaccination status captured by EHR. The comparison results indicated that the vaccine effectiveness was reasonably consistent across different sensitivity ranges, suggesting that our primary analysis was robust to changes in the range of sensitivity considered. Section S12 shows the comparative results to a fully Bayesian method indicating nearly identical results. Section S13 shows sensitivity analyses on differential misclassifications which demonstrates the trial emulation pipeline corrects the bias even when the non-differential misclassification assumption does not hold.
DISCUSSION
We emulated three target trials to evaluate the effectiveness of BNT162b2 vaccines for the prevention of documented COVID-19 infections and severe disease in a national network of pediatric health systems in the U.S. During the period of time where the Omicron variant was dominant, the BNT162b2 vaccine was associated with a moderate protection against infection and severe outcomes with an effectiveness of approximately 70% for children and 85% for adolescents. The estimated protection decreased by roughly 10% around four months from the first dose and slightly waned over time. During the period of time where the Delta variant was dominant, the BNT162b2 vaccine in adolescents was associated with strong protection with effectiveness higher than 95% and with little evidence of waning during the follow-up period.
Our study has several strengths. First, we used a national network of academic medical centers that covered a diverse cohort being more representative of the general pediatric population, provided a robust sample size, and allowed for multiple subgroup analyses and detection of rare outcomes. Second, the richness of these EHR data allowed us to investigate the effectiveness against infection of different levels of severity as well as adjust for a broad set of confounders. Third, we conducted the negative control outcome experiments to assess the potential residual bias due to unmeasured confounders and other potential sources of systematic bias in the data. These experiments revealed a small amount of systematic error but with excessive uncertainty across different negative control outcomes, leading to wider confidence intervals of our estimated effectiveness that honestly reflects the impacts of unmeasured confounding and other potential sources of residual biases20. Finally, to the best of our knowledge, this is the first target trial emulation evaluating COVID-19 vaccines against infection and severe outcomes that explicitly handle exposure misclassification.
Our study also has several potential limitations. First, effectiveness was investigated against documented infection in a cohort without previous infection, but potential bias from undocumented infections cannot be ruled out, especially if these occurred differentially in the vaccinated and unvaccinated cohorts. Our inclusion of previous negative COVID-19 tests as a confounder can adjust for the propensity to get tested which could partially adjust for this factor. Moreover, the increasing availability of at-home rapid antigen testing kits over time could have further reduced the testing frequency captured by EHR. This factor is expected to have less impact on severe outcomes that are associated with specific symptoms of the infection. Baseline confounders were balanced between vaccinated and unvaccinated groups, which should adjust for between-cohort differences in exposure risks and risks of severe infections. Second, as in any observational study, assignment to the vaccine group was non-random and the validity of the results could be impacted by unmeasured confounders. To evaluate the impact of unmeasured confounders and residual bias, we conducted negative control experiments that quantified the robustness of our results.
Third, in this study, patients who had received vaccinations prior to the start of the study period were excluded. Due to missing vaccine records, some patients who had previously been vaccinated may have still entered the cohort, particularly in the unvaccinated group. However, the trial emulation pipeline used in this study adjusted for potential bias resulting from unrecorded vaccinated patients which could also reduce the bias resulting from this issue. Fourth, in the Omicron study involving adolescents, the cohort included adolescents who had their first vaccine after January 1, 2022. Since the use of BNT162b2 vaccines was authorized in adolescents aged 12-15 years on May 10, 2021, this cohort may represent a population with late vaccines which reduces the generalizability of the findings. Finally, in the dose-response analyses, the effectiveness of a single dose of the BNT162b2 vaccine was estimated by comparing patients who had a one-dose record in EHR to those with no vaccination, which may potentially reveal the effectiveness of mixing of single-and two-dose due to the missingness of the second dose. However, as the second dose is always scheduled at the receipt of the first dose, we consider the proportion of missingness on the second dose should be low.
Although this study provides evidence of a slight waning of vaccine effectiveness 4 months following the first dose against Omicron infection and the effectiveness is stabilized after 4 months, waning can be impacted by vaccines during the follow-up period and other factors. Patients who got boosters during the follow-up period were not excluded from the study. A sensitivity analysis evaluating the durability of two-dose vaccine effectiveness considering the third dose as censoring did not suggest a significantly different conclusion. A future study is warranted to investigate the effect of booster vaccination among children and adolescents. Furthermore, despite the recognized risk of myocarditis associated with COVID-19 vaccines in young men and teen boys, the study reveals a lower relative risk of myocarditis in vaccinated groups which can be explained by its reduced likelihood of infection21.
In summary, this study involving national pediatric cohorts in the U.S. estimates moderate effectiveness of the BNT162b2 vaccine for preventing infection and severe diseases of the SARS-CoV-2 Omicron variant, and high effectiveness against the Delta variant. This study reveals a low risk of cardiac-related outcomes among children and adolescents who were vaccinated. This study significantly contributes to our knowledge of the BNT162b2 vaccine in the U.S. pediatric population using a rigorously designed trial emulation pipeline accounting for the incomplete capture of vaccination status in EHR data in the U.S.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
Footnotes
↵^ Co-first authors
↵* Co-senior authors
Authorship Statement: Authorship has been determined according to ICMJE recommendations.
Funding source: This research was supported in part by National Institutes of Health (OT2HL161847-01, 1R01LM012607, 1R01AI130460, 1R01AG073435, 1R56AG074604, 1R01LM013519, 1R56AG069880, 1R01AG077820, 1U01TR003709). This work was supported partially by Patient-Centered Outcomes Research Institute (PCORI) Project Program Awards (ME-2019C3-18315 and ME-2018C3-14899). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.
Title and abstract updated