Abstract
Background The U.S. FDA authorized the monovalent third primary series or booster doses of COVID-19 mRNA vaccines in August 2021 for persons 18 years and older. Monitoring of outcomes following updated authorizations is critical to evaluate vaccine safety and can provide early detection of rare adverse events (AEs) not identified in pre-licensure trials.
Methods We evaluated the risk of 17 AEs following third doses of COVID-19 mRNA vaccines from August 2021 through early 2022 among adults aged 18-64 years in three commercial databases (Optum, Carelon Research, CVS Health) and adults aged >65 years in Medicare Fee-For-Service. We compared observed AE incidence rates to historical (expected) rates prior to the pandemic, estimated incidence rate ratios (IRRs) for the Medicare database and pooled IRR across the three commercial databases. Analyses were also stratified by prior history of COVID-19 diagnosis. Estimates exceeding a pre-defined threshold were considered statistical signals.
Results Four AEs met the threshold for statistical signals for BNT162b2 and mRNA-1273 vaccines including Bell’s Palsy and pulmonary embolism in Medicare, and anaphylaxis and myocarditis/pericarditis in commercial databases. Nine AEs and three AEs signaled among adults with and without prior COVID-19 diagnosis, respectively.
Conclusions This early monitoring study identified statistical signals for AEs following third doses of COVID-19 mRNA vaccination. Since this method is intended for screening purposes and generates crude results, results do not establish a causal association between the vaccines and AEs. FDA’s public health assessment remains consistent that the benefits of COVID-19 vaccination outweigh the risks of vaccination.
Introduction
In the United States (U.S.), the BNT162b2 (Pfizer-BioNTech) and mRNA-1273 (Moderna) mRNA vaccines are available to prevent severe Coronavirus disease 2019 (COVID-19).1,2 For monovalent formulations, third doses as primary series doses were recommended for persons with immunocompromising conditions as of August 12, 2021, and a first booster dose (outside of the primary series) was authorized through Emergency Use Authorization (EUA) by the U.S. Food and Drug Administration (FDA) for everyone aged 18 years and older from November 19, 2021 to April 18, 2023. Statistical signals have been observed in active surveillance of the monovalent mRNA vaccines.3–5 Thus, the FDA Biologics Effectiveness and Safety (BEST) Initiative, an active surveillance program for post-market surveillance of biologic products, conducted this study to investigate the safety of COVID-19 mRNA vaccines among individuals aged 18 to 64 years in the commercially insured population and individuals aged 65 years and older in the Medicare population by evaluating the rates of 17 adverse events (AEs) of interest following third doses by comparing them to historical control (expected) rates for their respective AEs. Results from this signal detection study do not establish a causal association between the vaccines and AEs.
Methods
Data Sources
This study used administrative claims data from the Centers for Medicare & Medicaid Services (CMS) Medicare (Fee-For-Service) database and from three commercial health insurance databases: CVS Health, Optum pre-adjudicated claims, and Carelon Research (Supplemental Table 1). The commercial data sources were supplemented with Immunization Information System (IIS) data from local and state jurisdictions to increase the capture of vaccinations.6
Study Population and Period
The study population included a commercially insured population (aged 18-64 years) and the publicly insured Medicare Fee-For-Service population (aged 65 years and older) who received a COVID-19 mRNA third dose vaccination from the study start date of August 12, 2021, when the FDA authorized an additional vaccine dose for certain immunocompromised individuals,7 through database-specific study end dates. Data are included through April 2022 for Optum and Carelon, February 2022 for CVS Health, and March 2022 for Medicare enrollees. For inclusion in AE-specific analyses, persons must have been continuously enrolled in a medical health insurance plan from the start of the AE-specific clean window through the date of their third COVID-19 vaccine dose. This AE-specific pre-vaccination clean window was defined as a specific number of days before vaccination date when a patient must not have had the AE (Table 1).
Exposure and Follow-Up
Exposure was defined as the receipt of a third dose of the BNT162b2 COVID-19 or the mRNA-1273 COVID-19 vaccine (Supplemental Table 2). Vaccinations were identified using brand and dose-specific Current Procedural Terminology (CPT)/Healthcare Common Procedure Coding System (HCPCS) codes or National Drug Codes (NDCs) in the professional, outpatient institutional, inpatient, or prescription drug care settings,8 and were identified by CVX (vaccine administered) product codes in IIS data sources.
Dose assignment was based on the chronological order in which vaccinations were observed for each person. Third doses were restricted to doses following a two-dose primary series of mRNA vaccines. If a patient was administered more than 3 doses of BNT162b2 or mRNA-1273 vaccine, follow-up time was censored at the time of the fourth dose administration.
Patients who received heterologous vaccine brands for the primary series and third dose and persons who received the first booster dose of Ad26.COV2.S (Janssen) vaccine were included in the descriptive analyses but not in the inferential analyses due to small sample size. First booster doses of the Ad26.COV2.S vaccine were restricted to doses following a one-dose primary series, and follow-up time was censored at the time of administration if a patient was administered more than 2 doses of Ad26.COV2.S.
Follow-up time began on the day after vaccination and was censored at death, disenrollment, end of AE-specific risk window (Table 1), end of study period, a subsequent COVID-19 vaccine dose, or at AE occurrence, whichever came first.
Adverse Events of Interest
Claims-based AE algorithms, pre-vaccination clean windows, and post-vaccination risk windows were selected based on literature review and clinical consultation. All AEs (Table 1) were identified using International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnosis codes.8 Claims from inpatient facilities (IP), emergency department encounters in outpatient facilities (OP-ED), and all outpatient facilities and individual providers or professionals (OP/PB) were used to identify AEs.
Statistical Analyses
The vaccine dose count was summarized by brand for each AE (Table 2). The count and proportion of heterologous vaccine brand administration of third doses was also summarized.
This study was designed as a retrospective cohort analysis with a historical comparator group. The primary analysis compared the observed rate of the AE of interest in the post-vaccination risk window to the rate in a database-specific historical control population, and incidence rate ratios (IRR) were estimated. The observed rate of each AE was adjusted based on observation delay for that outcome in 2019 using a similar approach as described in the literature for partially accrued data.9 We calculated database-specific comparator rates as the annual AE rates within the overall and influenza-vaccinated populations in the pre-COVID-19 (2017-2019) and during-COVID-19 pandemic (June 2020-December 10, 2020) periods. The comparator rate used for the analysis was selected to enhance the sensitivity of the test for signal detection. If annual rates in the historical period differed, as assessed by comparing the 95% confidence intervals (CI), we selected the lowest rate; otherwise, the median annual rate was selected. Pre-COVID-19 rates from 2019 were selected for most outcomes; however, when the during-COVID-19 rate (June-December 10, 2020) did not return to pre-COVID-19 levels, rates from 2020 were selected.8 There were no adjustments for baseline clinical characteristics. Comparator rates were generated stratified by age and sex when sufficient counts in the respective strata were available in all databases. Also, comparator rates were generated by nursing home residency status and race in the Medicare population only.
Analyses for myocarditis/pericarditis were stratified by care setting (all settings and IP/OP-ED settings) and age (18-45 years and 46-64 years) in the commercially insured population to more granularly assess risk based on prior evidence regarding increased risk of myocarditis/pericarditis following COVID-19 mRNA vaccination particularly amongst younger males.10–22
For each potential AE, Poisson models were used to estimate the incidence rate (IR) per 100,000 person-years with 95% CI and the IRR of observed IR vs. expected IR with 95% CI. One-sided tests were conducted with an overall alpha of 2.5% and with testing margins selected for each AE based on clinical significance and expert guidance to avoid detection of false-positive results and characterization of non-clinically meaningful, minimal risk increases. In the commercially insured population, the null hypotheses were IRR = 1.5 for anaphylaxis, transverse myelitis, and myocarditis/pericarditis; 2.5 for encephalitis or encephalomyelitis, narcolepsy, and Guillain-Barré syndrome (GBS); and 1.25 for all other outcomes. To detect any increase in risk in the Medicare population, in which outcomes are more common, more sensitive tests were conducted where the null hypothesis was IRR = 1 for all AEs. A statistical signal occurred if the observed IRR exceeded the null hypothesis and the null hypothesis was not within the IRR 95% CI which was used as a threshold for determining whether the result was likely to occur due to chance. Statistical testing was only conducted for individuals who received the same vaccine brand for primary series and third doses.
To account for prior COVID-19 infection as a potential confounder, we conducted a secondary analysis for each AE stratified by history of COVID-19 diagnosis identified by diagnosis of COVID-19 in any care setting (identified by U07.1 ICD-10-CM code) on or following April 1, 2020.
IRs and IRRs were generated for the Medicare database separately and pooled across the three commercial databases using meta-analysis with a random-effects and fixed-effects models. Between-study heterogeneity was assessed using a I2 statistic.23
Results
Descriptive Analyses
Across the commercial insurance databases, 3,803,442 beneficiaries received BNT162b2 and 2,400,502 beneficiaries received mRNA-1273 third dose vaccines. The count of vaccine doses was summarized by brand for each AE for all COVID-19 vaccines (Table 2) and AE-specific analyses (Supplemental Table 3). A large proportion of third doses was mRNA vaccines; we did not conduct inferential analyses on the Ad26.COV2.S vaccine due to the small number of doses administered. Approximately 64% - 83% of vaccinations across brands in commercial databases were eligible for the analysis and contributed to an AE-specific cohort. In the Medicare database, 3,336,862 beneficiaries received BNT162b2 and 3,399,048 beneficiaries received mRNA-1273 third dose vaccines; more than 90% of these vaccinations across brands were eligible for the analysis and contributed to an AE-specific cohort (Supplemental Table 3). Less than 13% of cases with an observed AE across databases had an associated prior COVID-19 diagnosis (Supplemental Table 4, Supplemental Table 5). Most people received the same vaccine brand for primary series and third doses (Supplemental Table 6).
Results of Primary and Secondary Analyses – Persons 18 to 64 years
Among the 17 AEs of interest evaluated in the commercially insured population, two AEs met the threshold for a statistical signal in primary analyses. A signal was observed post-BNT162b2 third dose vaccination for anaphylaxis (IRR = 4.92, 95% CI = 2.20–10.98). Following mRNA-1273 third dose vaccination, signals were observed for anaphylaxis (IRR = 5.70, 95% CI = 1.69– 19.21) and myocarditis/pericarditis: IP/OP-ED (18-45 years) (IRR = 3.38, 95% CI = 1.51–7.58) (Table 3).
In the secondary analyses stratified by prior COVID-19 diagnosis, statistical signals were observed in the population with a prior COVID-19 diagnosis following BNT162b2 third dose vaccination for anaphylaxis (IRR= 37.26, 95% CI = 6.69–207.58); encephalitis or encephalomyelitis (IRR= 26.85, 95% CI = 7.43 – 96.99); and unusual site TTS (IRR= 8.43, 95% CI = 1.51–46.96) (Supplemental Table 4). The following statistical signals were observed following mRNA-1273 third dose vaccination: myocarditis/pericarditis, all settings for 18-45 years (IRR= 7.22, 95% CI = 2.58–20.19) and 46-64 years (IRR= 6.61, 95% CI = 3.26–13.41); myocarditis/pericarditis, IP/OP-ED settings for 46-64 years (IRR= 11.35, 95% CI = 4.06–31.74); common site thrombosis with thrombocytopenia (TTS) (IRR= 5.83, 95% CI = 2.28–14.92); and GBS (IRR= 18.19, 95% CI = 3.26–101.35).
In the population without a prior history of COVID-19 diagnosis, a signal was observed post-BNT162b2 third dose vaccination for anaphylaxis (IRR= 6.21, 95% CI = 2.50–15.39). In addition, signals were observed post-mRNA-1273 third dose vaccinations for myocarditis/pericarditis, IP/OP-ED settings for 18-45 years (IRR= 3.63, 95% CI = 1.62–8.15); and anaphylaxis (IRR= 6.15, 95% CI = 1.82–20.71) (Supplemental Table 4).
Results of Primary and Secondary Analyses –Persons 65 years and older
Statistical signals were observed for two AEs in the primary analysis in the Medicare population following third dose vaccinations. Statistical signals were identified following BNT162b2 third dose vaccinations for Bell’s Palsy (BP) (IRR = 1.11, 95% CI = 1.03 – 1.19) and pulmonary embolism (PE) (IRR = 1.05, 95% CI = 1.001 – 1.100) (Table 4). No signals were identified following mRNA-1273 third dose vaccinations (Table 4).
In the secondary analyses stratified by prior COVID-19 diagnosis, five AEs met the threshold for a statistical signal. In the population with a prior COVID-19 diagnosis, signals were observed for PE following BNT162b2 (IRR = 1.36, 95% CI = 1.17–1.58) and mRNA-1273 third dose vaccinations (IRR = 1.43, 95% CI = 1.20–1.69); acute myocardial infarction (AMI) following BNT162b2 (IRR = 1.15, 95% CI = 1.02–1.29) and mRNA-1273 third dose vaccinations (IRR = 1.33, 95% CI = 1.16–1.52). In addition, following BNT162b2 third dose vaccination, signals were observed for immune thrombocytopenia (ITP) (IRR = 1.66, 95% CI = 1.17–2.29) and myocarditis/pericarditis (IRR = 1.89, 95% CI = 1.18–2.86).
In the population without a prior history of COVID-19 diagnosis, a signal was observed for BP (IRR = 1.11, 95% CI = 1.03–1.19) following BNT162b2 third dose vaccination (Supplemental Table 6). While more AEs of interest with elevated risk were observed in the secondary analysis compared to the primary analysis, smaller sample sizes reflected wider 95% CI estimates (Supplemental Tables 4–5).
Discussion
In this large population-based study we investigated the safety of third doses of COVID-19 mRNA vaccines by examining the risk of 17 AEs following vaccination in a signal detection study as the first step in the safety surveillance process.
The primary analyses of the study found statistical signals for four AEs. Myocarditis/pericarditis and anaphylaxis have been reported to be associated with the primary doses of vaccines.3,4,8,10–22,24 There is a high level of outcome misclassification (a non-case being identified as a case or vice versa) associated with the detection of BP cases and hence uncertainty with its signal as results from a medical record review (MRR) of a sample of BP cases in our other COVID-19 vaccine studies showed a low positive predictive value (PPV) (<20%, data not shown) of the claims-based diagnosis codes used to identify cases indicating that most cases resulting in the signal were not true cases. The PE signal was marginal and similar to inconsistent results detected in previous COVID-19 vaccine safety studies where a protective effect following third doses of both mRNA brands and a small statistically significant elevated risk following BNT162b2 primary series were observed for PE.24
Among those with a prior COVID-19 diagnosis, nine AEs signaled. Most of these AEs have been reported to be associated with COVID-19 infection itself.25–27 In adults aged 18-64 years, all signals in populations with and without prior COVID-19 diagnosis arose from a small number of cases. Similar to BP, our MRR of ITP cases in other COVID-19 vaccines safety studies suggested a low PPV (<10%, data not shown) of the claims-based diagnosis codes used to identify cases indicating that most cases identified for inclusion were not true cases. Moreover, given the small number of cases, wide 95% CIs, and IRRs that are unadjusted for confounders, these signals carry a high level of uncertainty and should be interpreted cautiously.
In the population without a prior COVID-19 diagnosis, three AEs signaled. The BP signal may be explained by potential outcome misclassification, and the anaphylaxis and myocarditis/pericarditis signals are consistent with the current literature.3,10–15
This study has several strengths. The study included a large U.S. adult population covered by commercial insurance or Fee-For-Service Medicare. Because of the large sample size, the study had higher power to detect a smaller increase in risk of rare AEs after vaccination that may not be captured in pre-authorization clinical trials. Linkage of the commercial administrative claims databases to the IIS databases captured more COVID-19 vaccines administered. The retrospective cohort design with the use of historical comparator group allowed a rapid safety evaluation of vaccines as the first step in the active surveillance process.
There are several limitations in this study. Aggregate historical rates rather than individuals were used as the comparator which increases the potential for confounding and bias. Thus, the study results are crude or unadjusted for confounders, and do not establish a causal relationship between the vaccines and AEs. The stratified results should also be interpreted in the context of uncertainty and with caution. Although this study design was unable to disentangle the effect of prior COVID-19 infection and COVID-19 vaccines on the rates of AEs, several AEs which signaled in the population with a prior history of COVID-19 diagnosis are known to be associated with COVID-19 infection.25–27 Therefore, it is likely that a number of detected signals from these crude results may be associated with the prior history of COVID-19 diagnosis.
In this study, AEs were identified by diagnosis codes in claims databases which are subject to coding errors, and the study did not conduct MRR for all AEs to verify the true cases. Thus, outcome misclassification cannot be ruled out and could also bias results. The misclassification also applies to detection of prior COVID-19 infection in claims databases for the same reason as well as others such as use of home tests or not seeking medical care for the infection. Several identified signals arose from a very small number of AE cases. AE risk intervals were pre-specified based on literature review and clinical input, but their misspecification could lead to bias. The large sample size, inclusion of many AEs, and the number of statistical tests could increase the probability of false-positive results. Additionally, our study may not have captured all COVID-19 vaccines administered as some people may have received vaccines from sources that did not submit claims to insurance carriers, such as mass vaccination sites. Lastly, the results of this study may not be generalizable to the portion of the U.S. population who is uninsured or carries other types of health insurance.
Since the authorization or approval of the mRNA vaccines that were evaluated in this study, there have been additional FDA authorizations of COVID-19 vaccines. This study did not include second booster doses authorized for those 50 years and older or those 12 years and older with certain immunocompromised conditions (authorized March 29, 2022), third doses for children ages 5 to 11 years (authorized May 17, 2022), or bivalent booster doses for individuals 12 years and older (authorized August 31, 2022) as these vaccines were not yet authorized for use at the time this study was conducted.1,28–30 Active safety surveillance is ongoing to expand the monitoring of AEs to include individuals who receive these vaccinations.
In conclusion, this signal detection study detected multiple signals after exposure to the third dose of COVID-19 mRNA vaccines. The results of this study must be interpreted in the context of the study design, statistical methods, and their limitations; thus, some of the safety signals detected may be false positive results. While this study has major limitations, it contributes to the signal detection phase in the multi-step surveillance process for COVID-19 mRNA vaccines. FDA’s public health assessment remains consistent that the benefits of COVID-19 vaccination outweigh the risks of COVID-19 disease, and safety surveillance of new COVID-19 vaccines is ongoing.
Tables
Conflicts of Interest
All authors have no conflicts of interest to disclose.
Ethics Statement
As a public health surveillance activity, this study was not subject to requirements for Institutional Review Board review.
Authorship Contributions
All authors contributed to the concept and design of the study and interpretation of the data. A. Shoaibi drafted the manuscript and all authors reviewed it. K. Matuska, H. Lyu, R. McEvoy, Z. Wan, M. Hu, D. Beachler, A. Secora, N. Selvam, D. Djibo, C. McMahill-Walraven, J. Seeger, K. Amend, J. Song, R. Clifford, R. Forshee, and S. Anderson acquired and managed the data and performed analyses. The views expressed here are those of the authors and not necessarily those of the U.S. Food and Drug Administration, the Centers for Medicare and Medicaid Services, the Department of Health and Human Services, CVS Health/Aetna, Carelon Research, IQVIA, or Optum Epidemiology. This work has not been presented publicly, previously published, or is under current consideration by another journal.
Funding
There were no extra-institutional sources of funding.
Data Availability
The study protocol was publicly posted as referenced in the manuscript before data analyses, and related documents can be made available where needed by contacting the corresponding author. De-identified participant data will not be shared without approval from the data partners.
Acknowledgments
We would like to thank Gyanada Acharya, Derick Ambarsoomzadeh, Mahasweta Mitra, Michelle Ondari, Ellie Smith, and Zoe Wu from Acumen LLC; Shiva Vojjala, Ramya Avula, Shiva Chaudhary, Qian Si, Dianna Hayden, Shanthi P Sagare, Ramin Riahi, Brian Greenwald, and Grace Stockbower from Carelon Research; Michael Goodman, Michael Bruhn, and Ruth Weed from IQVIA; Anne Marie Kline, Nancy B. Shaik, Ana M. Martinez-Baquero, Smita Bhatia, Carla Brannan, Vaibhav Sharma, Eugenio Abente, and Jonathan Deshazo from CVS Health; and Grace Yang, Sarah Sargen, Alexandra Stone, Wafa Tarazi, Megan Ketchell, Kathryn Federici, Amaka Ume, Emily Myers, Eli Wolter, Jackson Slaney, Bobby Smith, Lauren Peetluk, and Elizabeth Bell from Optum for their assistance with data validation, analysis, manuscript preparation, and project coordination.
Footnotes
↵* First Authors