Abstract
Background Monitoring safety outcomes following COVID-19 vaccination is critical for understanding vaccine safety especially when used in key populations such as elderly persons age 65 years and older who can benefit greatly from vaccination. We present new findings from a nationally representative early warning system that may expand the safety knowledge base to further public trust and inform decision making on vaccine safety by government agencies, healthcare providers, interested stakeholders, and the public.
Methods We evaluated 14 outcomes of interest following COVID-19 vaccination using the US Centers for Medicare & Medicaid Services (CMS) data covering 30,712,101 elderly persons. The CMS data from December 11, 2020 through Jan 15, 2022 included 17,411,342 COVID-19 vaccinees who received a total of 34,639,937 doses. We conducted weekly sequential testing and generated rate ratios (RR) of observed outcome rates compared to historical (or expected) rates prior to COVID-19 vaccination.
Findings Four outcomes met the threshold for a statistical signal following Pfizer-BioNTech vaccination including pulmonary embolism (PE; RR=1.54), acute myocardial infarction (AMI; RR=1.42), disseminated intravascular coagulation (DIC; RR=1.91), and immune thrombocytopenia (ITP; RR=1.44). After further evaluation, only the RR for PE still met the statistical threshold for a signal; however, the RRs for AMI, DIC, and ITP no longer did. No statistical signals were identified following vaccination with either the Moderna or Janssen vaccines.
Interpretation This early warning system is the first to identify temporal associations for PE, AMI, DIC, and ITP following Pfizer-BioNTech vaccination in the elderly. Because an early warning system does not prove that the vaccines cause these outcomes, more robust epidemiologic studies with adjustment for confounding factors, including age and nursing home residency, are underway to further evaluate these signals. FDA strongly believes the potential benefits of COVID-19 vaccination outweigh the potential risks of COVID-19 infection.
Introduction
The US Food and Drug Administration (FDA) is monitoring the safety of three vaccines for Coronavirus Disease 2019 (COVID-19) currently available in the US. These include the licensed Pfizer BioNTech vaccine (Comirnaty) for persons 16 years and older and authorized under emergency use authorization (EUA) for those 12-15 years (BNT162b2), as well as the authorized Moderna (mRNA-1273), and Janssen (Ad26.COV2.S) vaccines for persons 18 years and older. Pre-authorization clinical studies provided useful information on the safety of COVID-19 vaccines, but they have limitations such as sample size and follow-up that may be addressed in post-authorization safety studies in large healthcare databases. Accordingly, FDA and the Centers for Medicare & Medicaid Services (CMS) are using the Medicare health insurance database, covering more than 25 million elderly persons aged 65 years and older, to conduct near real-time safety monitoring of 14 outcomes on a weekly basis following COVID-19 vaccine administration.
The US elderly population, including persons in Long-Term Care Facilities or nursing homes [1], were disproportionately affected by COVID-19, as they were among the first US populations to be infected. Because they suffered a higher rate of infection, serious disease, and severe outcomes including death [2], they were among the first groups recommended by the Advisory Committee on Immunization Practices to receive the vaccine [3]. Available information concerning the safety of the vaccine in elderly persons is limited. However, the near real-time surveillance method used by FDA and CMS continues to expand the available knowledge base and further advances our understanding of the safety profile of these new COVID-19 vaccines. The FDA-CMS near real-time active surveillance program complements other FDA and US Government vaccine safety surveillance systems by rapidly detecting safety concerns that may not have been voluntarily reported to passive surveillance systems such as Vaccine Adverse Event Reporting System. Routinely used by FDA in safety surveillance for annual influenza vaccines in the past decade, this method is designed to be sensitive enough to quickly screen safety signals for further evaluation in robust epidemiologic studies [4, 5]. This rapid screening method performs hypothesis testing, sequentially, in a prospective manner as the vaccine data accrue to detect potential safety signals earlier in the course of surveillance, but signals must be further evaluated in more robust studies with adjustment for confounding. However, results detected by near real-time surveillance do not establish a causal association between the outcomes and vaccination because the method has limited adjustments for confounding.
In this report, we summarize the results of weekly sequential testing analyses for 14 outcomes where formal testing was initiated. We also describe how the identified signals were evaluated and how the epidemiological studies currently underway will provide more robust adjustment for confounding determine if any are true signals.
Methods
Data Sources
We used the US Medicare Fee-for-Service (FFS) Parts A (inpatient services) and B (outpatient care) claims and enrollment data to define inclusion criteria, exposures, outcomes, and patient characteristics. The number of 65+ beneficiaries with at least 1 day of Medicare FFS enrollment in the study period is 30,712,101. The Minimum Data Set identified nursing home residency status.
Study Period and Population
The study included Medicare FFS beneficiaries aged 65 years or older who received a COVID-19 vaccine since December 11, 2020. To be included, individuals needed to be enrolled on the vaccination date, and continuously enrolled during an outcome-specific pre-vaccination clean window [6].
Exposure and Follow-Up
Exposure was defined as receipt of a Pfizer-BioNTech, Moderna, or Janssen COVID-19 vaccination, identified using brand and dose-specific Current Procedural Terminology / Healthcare Common Procedure Coding System codes [7] (Table S1). The primary analysis included all observed doses by brand. Dose-specific analyses are described in supplemental materials. Follow-up included all time in the prespecified risk windows; the first-dose risk window was censored at the time of the second-dose vaccination.
Outcomes
The list of 14 outcomes, pre-vaccination clean windows, and post-vaccination risk windows are detailed in Table S2. Claims-based outcome algorithms were developed based on literature review and in consultation with clinical experts [8].
Vaccine Safety Surveillance
Weekly vaccine uptake was monitored by brand. The near real-time surveillance compared the observed number of each outcome in the COVID-19 vaccinated population to an expected number based on the background rate of the outcome in a similar COVID-19 unvaccinated population prior to the pandemic, adjusted for the delay in claims processing and standardized by nursing home residency status, age, sex, and race. We calculated annual background rates within the strata of the standardized variables, where possible, during 2017-2019 (pre-COVID-19) and peri-COVID-19 (January 1, 2020–December 10, 2020) [9] (Table S3). If annual rates in the historical period differed substantially from each other, we selected the minimum rate as a more conservative approach. Otherwise, the median annual rate was selected.
Statistical Analysis
Poisson Maximized Sequential Probability Ratio Test (PMaxSPRT) was used to detect increased outcome risk following vaccination compared to a historical baseline for 14 outcomes [10-12] (Table S3). Weekly sequential testing for each outcome commenced when a minimum of three cases accrued. One-tailed tests were used, with a null hypothesis that the observed rate was no greater than the historical comparator beyond a prespecified test margin with an overall alpha of 1%. The test margin was selected for each outcome based on expert guidance to avoid minimal risk increases that were unlikely to be clinically relevant. The alpha level was selected to address a large number of tests. A statistical signal occurred if the log likelihood ratio exceeded the critical value, or a threshold for determining whether the result was likely to occur due to chance. Additional details are provided in Table S3 and the study protocol [6].
Signal Evaluation
Prespecified signal evaluation analyses were conducted after a statistical signal was observed. Data quality was checked to rule out database errors or changes in event observation as potential sources of the signal. Sensitivity analyses evaluated whether the increase in risk was observed with alternate expected rates in sequential testing – (i) rates from calendar months in the historical period corresponding to those in the study period to address monthly variations in the background rates, and (ii) rates among a subset of Medicare beneficiaries with influenza vaccination in the past year to assess differential healthcare utilization. For pulmonary embolism (PE), additional ad hoc analyses were conducted with (i) the outcome limited to the inpatient setting and (ii) rates from November 1–December 11, 2020 to address changes in rates in the peri-COVID period as alternate expected rates.
Signal characterization assessed the cases and the potential elevated risk. Analyses included (i) identifying clusters in the risk window following dose 1 through temporal scans, (ii) estimating relative risks within demographic strata, (iii) assessing distribution of care settings and diagnosis codes in observation and historical periods, (iv) comparing patient characteristics among vaccinated and overall elderly populations, and (v) reviewing patterns of reimbursement codes by clinicians on a random sample of up to 100 cases per outcome to contextualize the medical histories of patients.
Statistical Software
All analyses were conducted using R 4.0.3 (R Foundation for Statistical Computing, Vienna, Austria), SAS v. 9.4 (SAS Institute Inc., Cary, NC, United States), and SaTScan v9.6 (Martin Kulldorff, Boston, MA, United States).
Ethical Considerations
This surveillance activity was conducted as part of the FDA public health surveillance mandate.
Results
Descriptive Statistics of the COVID-19 Vaccinated Population
From December 11, 2020 through January 15, 2022, 17,088,796 Pfizer-BioNTech, 16,898,376 Moderna, and 634,019 Janssen COVID-19 vaccine doses were found among 30,712,101 eligible individuals in the study period (34,621,191 total vaccine doses including vaccination days with multiple products). The population of Pfizer-BioNTech vaccinees had a slightly higher proportion of older individuals (85 years and older), nursing home residents, and individuals residing in urban areas compared to the general elderly Medicare population (Table 1; Figure 1). Moderna vaccinees exhibited similar characteristics to the general Medicare population, while the population receiving the Janssen vaccine was younger and had fewer nursing home residents.
Comparing data cuts through April 24, 2021 (Table S6), March 13, 2021 (Table S7) and February 27, 2021 (Table S8), the populations differed for those who were vaccinated from mid December 2020 to February 2021 versus the later months. Pfizer vaccinees experienced more hospitalizations in the prior year and higher proportions of underlying medical conditions (e.g., Charlson comorbidity index > 0), when compared to the overall elderly Medicare, Moderna, or Janssen populations.
PMaxSPRT Sequential Testing Results
All outcomes with primary analyses for PMaxSPRT testing met the prespecified criteria for initiation of analyses (Table S3). No statistical signals were identified following vaccination with either the Moderna or Janssen vaccines. AMI (RR=1.42), PE (RR=1.54), disseminated intravascular coagulation (DIC; RR=1.91), and immune thrombocytopenia (ITP; RR=1.44) following Pfizer-BioNTech vaccination met the statistical threshold for a signal (Table 2). Dose-specific results can be seen in Tables S4-5.
Signal Evaluation
None of the prespecified data quality assurance checks, including claims duplication and unusual variability in claim accrual, raised data quality concerns (Table S9). Primary findings for signal robustness and signal characterization analyses are summarized in Table 3. Adjustment for monthly variation in the background rates resulted in statistically non-significant associations for AMI, DIC, and ITP following Pfizer-BioNTech vaccination. With background rates from the flu-vaccinated population as the historical comparator, DIC and ITP no longer met the signal threshold, while signals for AMI (RR=1.41) and PE (RR=1.48) remained. When rates during the peri-COVID period were used as the historical comparator, PE and DIC no longer met the signal threshold. We conducted an additional ad hoc sensitivity analysis for PE. When PE events were restricted to the inpatient setting, the statistical signal remained (RR=2.17).
Clinical subject matter experts reviewed claims-based diagnoses and procedures for selected patients from one year prior through one year after the dates of sampled outcome events. They found outcome-specific comorbidities were present in 41%, 45%, 95%, and 66% of AMI, PE, DIC, and ITP cases, respectively. Additionally, only a single diagnosis without other mention of the outcome occurred in 26%, 37%, 61%, and 49% of cases for AMI, PE, DIC, and ITP, respectively. Outcome-specific treatments or diagnostics were observed in 34%, 53%, 40%, and 18% of AMI, PE, DIC, and ITP cases, respectively.
Evaluation of claims-based diagnosis codes showed differences in coding patterns for AMI and PE between the study period and the historical period used to calculate expected rates (Table 3). Type II AMI codes, which are indicative of a mismatch between myocardial oxygen supply and demand (as opposed to acute coronary thrombosis), were more common in outcomes identified during the study period (46%) than in the historical period (28%). Additionally, for both AMI and PE outcomes, a higher proportion of inpatient claims-based codes occurred in a non-primary diagnosis position in the study period. No differences in coding patterns between the study period and historical period were noted for either ITP or DIC.
Finally, temporal scans conducted at the time of signal did not identify any clustering of cases within the post-vaccination risk window for PE, DIC, or ITP post Pfizer’s first dose. A scan for clusters 1-18 days long identified a statistically significant cluster of 1-17 days for AMI; however, the observed rate of events in the cluster was only 4% higher than if events had been evenly distributed in the risk window (Figure 2). Another scan for clusters 1-10 days long that also accounted for censoring due to end of follow-up did not identify a statistically significant cluster for AMI (Figure 2).
Discussion
Our early warning safety system is the first to identify four new statistical signals for modestly elevated risks (RR < 2) of four serious outcomes of AMI, PE, DIC, and ITP following Pfizer-BioNTech vaccination. This FDA and CMS COVID-19 vaccine safety study is one of the largest studies of elderly persons aged 65 years and above including approximately 34 million doses administered to more than 17 million Medicare insured persons. Our surveillance monitoring did not detect statistical signals for the Moderna and Janssen vaccines for any of the 14 monitored outcomes.
The statistical signals of four serious outcomes are not necessarily causal and may be due to factors potentially unrelated to vaccination. Additional analyses indicated that the potential association was less than twice the historical rates and may be associated with factors not accounted for in the near real-time surveillance methods. For example, the elderly Medicare population that received the Pfizer-BioNTech vaccine differed from other elderly COVID-19 vaccinated populations, including a preponderance of nursing home residents and populations with a higher comorbidity burden. These demographic and medical differences were not fully accounted for, since expected rates were only standardized to a subset of characteristics – age, sex, race, and nursing home residency status. Further, the AMI, DIC, and ITP signals were not robust when additional baseline rates were evaluated, while the PE signal might be explained by differences in rates between the pre-COVID-19 and peri-COVID-19 periods. In addition, the clinical assessment of patterns of reimbursement codes indicated that a substantial fraction had pre-existing outcome-specific comorbidities and risk factors, and that some outcomes may be due to follow-up care to an existing condition preceding the vaccination.
Our study has several strengths. This is the largest study of a population of more than 25 million elderly persons who are vulnerable to COVID-19 infections and complications-including residents of long-term care facilities. By using the large Medicare nationwide database with longitudinal linkage of vaccination, health services, and demographic information for millions of elderly persons, we can detect even small increases in the relative risk of rare outcomes for multiple vaccines that may not be captured in pre-authorization clinical trials. In addition, this near real-time surveillance benefits from the experience and knowledge obtained during more than a dozen years of successful collaboration between FDA and CMS conducting vaccine safety analyses using the Medicare database [13], including near real-time surveillance analyses for Guillain-Barré syndrome after influenza vaccination [14-16]. Furthermore, the weekly data updates and analyses allow for signal detection across 14 outcomes using near-real time monitoring. This further expands our knowledge of COVID-19 vaccine safety for informing timely regulatory action, if warranted as well as decision-making by healthcare providers, patients and the general public
We acknowledge our analysis has limitations. The near real-time analysis did not adjust for underlying risk factors such as comorbidities among recipients in the early vaccination campaign leading to falsely positive or negative signals. Furthermore, the early warning system may falsely identify a signal (false positive) or signals because of the high number of statistical tests performed or possible misspecification of parameters. Conversely, true safety signals (false negatives) may be missed due to mispecified parameters in the analyses. Diagnosis billing codes in claims data may underestimate or overestimate certain clinical conditions because of reimbursement priorities. We also note that results of this near real-time surveillance in elderly persons may not be generalizable to those younger than 65 years and adults who are uninsured or received only commercial health insurance. To address several of these limitations we are conducting further epidemiological studies along with medical record review to adjudicate outcomes identified by claims-based definitions.
In conclusion, we demonstrate that this FDA-CMS early warning safety system is working to rapidly identify potential new and important safety concerns following COVID-19 vaccination for consideration and to support potential decision-making by regulatory and public health authorities, healthcare professionals and the general public. Our new findings of statistical signals for four important outcomes for the Pfizer-BioNTech vaccine should be interpreted cautiously because the early warning system does not prove that vaccines cause the safety outcomes. FDA strongly believes the potential benefits of COVID-19 vaccination outweigh the potential risks of COVID-19 infection. Per FDA communication of these findings, FDA is currently not taking any regulatory actions based on these signal detection activities because these signals are still under investigation and require more robust study. The FDA active surveillance systems, including the CMS partnership, are a major part of a larger US federal surveillance effort to increase knowledge of COVID-19 vaccine safety to support decision-making that further protects public health during the pandemic.
Data Availability
The study protocol was publicly posted as referenced in the manuscript before data analyses and related documents can be made available where needed, by contacting the corresponding author. De-identified participant data will not be shared without approval from CMS.
Acknowledgements
We thank Joyce Obidi, PhD, Kristin “Kristine” Sepúlveda, MBA and Tainya Clarke, PhD, MPH, MSc for program management and J. Rosser Matthews, PhD, MPP, MPH for writing support. We would like to acknowledge John Hornberger, MD, MS, Nirmal Choradia, MD, Marna Bogan, RN, CPC, Susan Siford, PharmD, MBA, Christina Jessee, RN, CPC for their clinical and coding expertise. Additionally, we acknowledge Anchi Lo, MS, MPH, Jing Wang, BA, Yue Wu, MS, Zhiruo Wan, MSE, Shanlai Shangguan, MPH, Rowan McEvoy, BS, Arnstein Lindaas, MA, and Chianti Shi, MS for providing statistical programming support, and Yixin Jiao, MPP, Manzi Ngaiza, MPH, and Ellie Smith, BS for providing writing support. We acknowledge Michael Sklar, PhD, Lorene Nelson, PhD, MS, Julia Simard, ScD, SM, and Steve Goodman, MD, MHS, PhD for their epidemiologic expertise. Finally, we acknowledge Ivair Silva, PhD for his methodological expertise and development of the ‘Sequential’ R package used to conduct analyses in this manuscript.
Footnotes
Conflict of Interest Statement All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: Hui-Lee Wong, Cindy Zhou, Deborah Thompson, Rositsa Dimova, Patricia Lloyd, Richard Forshee, and Steven Anderson are/were salaried employees of the U.S. Food and Drug Administration; Ellen Tworkoski, Mao Hu, Bradley Lufkin, Rose Do, Laurie Feinberg, Yoganand Chillarige, and Thomas MaCurdy are/were salaried employees of Acumen, LLC, which is a contractor with the U.S. Food and Drug Administration; Jeffrey Kelman is a salaried employee of the Centers for Medicare & Medicaid Services; Ellen Tworkoski is currently a salaried employee of Biogen, which includes stock options provided to her as part of employee benefits; no other relationships or activities that could appear to have influenced the submitted work.
References
