Abstract
Background Routinely collected healthcare data such as administrative claims and electronic health records (EHR) can complement clinical trials and spontaneous reports when ensuring the safety of vaccines, but uncertainty remains about what epidemiological design to use.
Methods Using 3 claims and 1 EHR database, we evaluate several variants of the case-control, comparative cohort, historical comparator, and self-controlled designs against historical vaccinations with real negative control outcomes (outcomes with no evidence to suggest that they could be caused by the vaccines) and simulated positive controls.
Results Most methods show large type 1 error, often identifying false positive signals. The cohort method appears either positively or negatively biased, depending on the choice of comparator index date. Empirical calibration using effect-size estimates for negative control outcomes can restore type 1 error to close to nominal, often at the cost of increasing type 2 error. After calibration, the self-controlled case series (SCCS) design shows the shortest time to detection for small true effect sizes, while the historical comparator performs well for strong effects.
Conclusions When applying any method for vaccine safety surveillance we recommend considering the potential for systematic error, especially due to confounding, which for many designs appears to be substantial. Adjusting for age and sex alone is likely not sufficient to address the differences between vaccinated and unvaccinated, and for the cohort method the choice of index date plays an important role in the comparability of the groups Inclusion of negative control outcomes allows both quantification of the systematic error and, if so desired, subsequent empirical calibration to restore type 1 error to its nominal value. In order to detect weaker signals, one may have to accept a higher type 1 error.
Highlights
Most methods used in vaccine safety surveillance show large type 1 error, which could lead to many false safety signals.
Empirical calibration using effect-size estimates for negative control outcomes can restore type 1 error to close to nominal, often at the cost of marginal increases in type 2 error.
After calibration, the self-controlled case series (SCCS) design shows the shortest time to detection for small true effect sizes, while the historical comparator appears best for large true effect sizes.
Implementing negative control outcomes in a safety surveillance system is recommended to identify vulnerability to systematic error.
Competing Interest Statement
All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: GH receives grant funding from the US National Institutes of Health and the US Food & Drug Administration. PBR, SF, and MJS are employees of Janssen Research and Development and shareholders in Johnson & Johnson. DPA reports grants and other from Amgen, grants, non-financial support and other from UCB Biopharma, grants from Les Laboratoires Servier, outside the submitted work; and Janssen, on behalf of IMI-funded EHDEN and EMIF consortiums, and Synapse Management Partners have supported training programs organized by DPA’s department and open for external participants. MAS receives grant funding from the US National Institutes of Health and the US Food & Drug Administration and contracts from the US Department of Veterans Affairs and Janssen Research and Development. FN was an employee of AstraZeneca until 2019 and owns some AstraZeneca shares. NP receives grant funding from the Australian National Health and Medical Research Council (GNT1157506).
Clinical Trial
EUPAS40259
Funding Statement
UK National Institute of Health Research (NIHR), European Medicines Agency, Innovative Medicines Initiative 2 (806968), US Food and Drug Administration CBER BEST Initiative (75F40120D00039), and US National Library of Medicine (R01 LM006910).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The use of Optum and IBM Marketscan databases was reviewed by the New England Institution Review Board (IRB) and was determined to be exempt from broad IRB approval, as this research project did not involve human subjects research.
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Abbreviations
- CCAE
- IBM MarketScan Commercial Claims and Encounters
- CI
- Confidence Interval
- COVID-19
- COronaVIrus Disease 2019
- CV
- Critical Value
- H1N1pdm
- Hemagglutinin Type 1 and Neuraminidase Type 1 (2009 pandemic influenza)
- HPV
- Human papillomavirus
- LLR
- Log Likelihood Ratio
- MaxSPRT
- Maximum Sequential Probability Ratio Testing
- MDCR
- IBM MarketScan Medicare Supplemental Database
- MDCD
- IBM MarketScan Multi-State Medicaid Database
- OHDSI
- Observational Health Data Sciences and Informatics
- PS
- Propensity Score
- SCCS
- Self-Controlled Case Series
- SCRI
- Self-Controlled Risk Interval
- TaR
- Time-at-Risk
The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, University of Washington, and Vrije Universiteit Amsterdam.