Abstract
As the COVID-19 vaccination campaign unfolds as one of the most rapid and widespread in history, it is important to continuously assess the real-world safety of the FDA-authorized vaccines. Curation from large-scale electronic health records (EHRs) allows for near real-time safety evaluations that were not previously possible. Here, we advance context- and sentiment-aware deep neural networks over the multi-state Mayo Clinic enterprise (Minnesota, Arizona, Florida, Wisconsin) for automatically curating the adverse effects mentioned by healthcare providers in over 108,000 EHR clinical notes between December 1st 2020 and February 8th 2021. We retrospectively compared the clinical notes of 31,029 individuals who received at least one dose of the Pfizer/BioNTech or Moderna mRNA vaccine to those of 30,933 unvaccinated individuals who were propensity matched by demographics, residential location, and history of prior SARS-CoV-2 testing. We find that vaccinated and unvaccinated individuals were seen in the clinic at similar rates within 21 days of the first or second actual or assigned vaccination date (first dose Odds Ratio = 1.14, 95% CI: 1.10-1.18; second dose Odds Ratio = 0.91, 95% CI: 0.86-0.96). Further, the incidence rates of all surveyed adverse effects were similar or lower in vaccinated individuals compared to unvaccinated individuals after either vaccine dose, although myalgia was modestly increased within 7 days of the second dose when considering only pairs of matched individuals who each had at least one clinical note in this time window (Incidence Rate Ratio = 2.5, 95% CI: 1.1-6.7). Finally, the most frequently documented adverse effects within 7 days of each vaccine dose were fatigue (Dose 1: 1.75%, Dose 2: 1.18%), nausea (Dose 1: 1.03%, Dose 2: 0.84%), myalgia (Dose 1: 0.41%; Dose 2: 0.43%), diarrhea (Dose 1: 0.65%; Dose 2: 0.45%), arthralgia (Dose 1: 0.64%; Dose 2: 0.57%), erythema (Dose 1: 0.56%; Dose 2: 0.44%), vomiting (Dose 1: 0.44%, Dose 2: 0.29%) and fever (Dose 1: 0.21%; Dose 2: 0.18%). These frequencies of adverse event documentation in EHR notes are 2.1 times (95% CI: [1.5, 3.0]) to 1500 times (95% CI: [670, 2800]) lower than the frequencies of adverse events recorded via active solicitation during clinical trials or post-marketing surveillance, with headache after second vaccination showing the highest ratio of trial reporting to EHR documentation. This rapid and timely analysis of EHR notes from 31,029 vaccinated individuals highlights the rarity of vaccine-associated adverse effects requiring clinical attention and reaffirms the tolerability of the FDA-authorized COVID-19 vaccines in practice.
Introduction
Following their Emergency Use Authorizations by the Food and Drug Administration (FDA) in December of 2020, more than 52 million doses of BNT162b2 (Pfizer/BioNTech) and mRNA-1273 (Moderna) COVID-19 vaccines have been administered in the United States1–3. Phase 3 trials demonstrate strong efficacy and favorable safety profiles for these vaccines in the cohorts studied. Specifically, the trials showed 95.0% (95% CI, 90.3 to 97.6) and 94.1% (95% CI, 89.3 to 96.8%) efficacy for BNT162b2 and mRNA-1273 respectively. While self-resolving mild to moderate adverse effects were common in vaccinated participants, serious adverse effects occurred rarely and with a frequency comparable to placebo4,5. Local adverse effects of any severity reported in these trials included injection site pain (84.1-92%), injection site swelling (10.5-70%), and injection site erythema (9.5-14.6%). Systemic effects of any severity included fatigue (62.9-70%), headache (55.1-64.7%), myalgia (38.3-61.5%), arthralgia (23.6-46.4%), chills (31.9-45.4%), fever (14.2-15.5%), and nausea/vomiting (1.1-23.0%)6,7.
Consistent with CDC recommendations, early vaccination efforts (Phase 1a) in the United States have targeted healthcare workers, residents and staff at long term care facilities, who are at elevated risk for COVID-19 relative to the general public8–11. As the vaccines continue to be administered more broadly, it will be critical to continuously evaluate real world safety and efficacy data from all those who have received these vaccines. This approach may validate existing findings or highlight differences in the larger population compared to the clinical trial experience with respect to vaccine efficacy and adverse effects. Of note, the populations undergoing vaccination may differ meaningfully from the trial populations. For example, healthcare workers and long term care residents receiving the first wave of vaccines in Phase 1a likely comprised a small fraction of the Phase 3 trial populations. Additionally, some pregnant women are receiving COVID-19 vaccines in the real world12, while they were excluded from both trials4,5. Finally, continuous monitoring will help to identify and better quantify the frequency of rare severe adverse effects such as anaphylaxis, which was widely reported but only observed in a small number of individuals after approval of both vaccines13–17.
As part of the Vaccine Adverse Events Reporting System (VAERS) maintained by the FDA, V-safe has recently been established to specifically monitor the safety profiles of COVID-19 vaccines as they are administered to the public18. V-safe is a voluntary program into which vaccinated individuals can enroll. Once enrolled, participants receive frequent reminders to complete surveys which document their side effects electronically. Thus, this program will create an excellent resource of clinical trial-like safety data derived from significantly larger and more diverse patient populations than those who were enrolled in the Phase 1/2 or Phase 3 trials.
An important complementary approach to post-authorization surveillance of vaccine efficacy and safety is via the real-time analysis of patient data stored in electronic health record (EHR) systems. We have previously developed and described augmented curation methods to rapidly create and compare cohorts of COVID-19 patients within a large EHR system, and have recently applied these methods to assess the real world efficacy of both BNT162b2 and mRNA-1273 in over 31,000 individuals receiving these vaccines at the Mayo Clinic and associated health system19–23. Here we expand on these efforts to study the adverse effects experienced by individuals and documented in the EHR after COVID-19 vaccination in the clinical environment.
It should be noted that the monitoring of vaccine-associated adverse effects in clinical trials or post-marketing surveillance efforts like V-safe is quite different from such monitoring in routine clinical practice. In clinical trials, participants are aware that they are receiving an experimental product, and adverse effect reporting is encouraged or solicited. Similarly, as was previously noted, V-safe participants complete frequent surveys to document their side effects after vaccination. On the other hand, individuals receiving a COVID-19 vaccine during the mass vaccination campaign are informed of adverse effects that are likely to be experienced and can even be discouraged from seeking medical attention unless the symptom is particularly severe. Thus, adverse effects are likely to be captured only in the EHR of individuals who experience symptoms which are severe or persistent enough to warrant a return to clinic, or who happen to have a previously scheduled routine clinical visit in the post vaccination time period. Accordingly, the intention of our analysis is not to determine whether real world data recapitulates the adverse effect frequencies reported in prior trials. Instead, it is to (1) establish the rates at which individuals actually report potential vaccine-associated adverse effects to health care practitioners (HCPs) in several defined time intervals after vaccination, and (2) determine whether these rates of adverse effect reporting are unexpectedly high.
To address the latter point, it is critical to establish the baseline frequency at which each adverse effect is expected to be documented in the clinical notes of our vaccinated cohort. We thus built upon our previous vaccine efficacy analysis which employed a 1-to-1 propensity matching procedure to derive a cohort of unvaccinated individuals who are balanced for demographic factors, residential location, and history of SARS-CoV-2 PCR testing (see Methods)24. Here we curated the clinical notes of these vaccinated and matched unvaccinated individuals over several time periods to define the frequencies of various symptoms of interest. In our analyses, the propensity matched group serves a purpose similar to that of the placebo arm in clinical trial safety assessments, allowing us to contextualize and better interpret the absolute rates of adverse effects documented in the notes of vaccinated individuals.
We find that individuals undergoing COVID-19 vaccination do not return to the clinic, including the emergency department, at unexpectedly high rates in the 7, 14, or 21 days following either vaccine dose. Further, the rates of adverse effects (headache, myalgia, arthralgia, fatigue, fever, diarrhea, nausea/vomiting, anaphylaxis, facial paralysis, lymphadenopathy, and erythema) documented in EHR notes over these time intervals are generally not higher in vaccinated individuals compared to the unvaccinated control cohort. The absolute rates of adverse effect documentation in the EHR are markedly lower than the rates observed in settings of solicited reporting (i.e. clinical trials, post-marketing surveillance). This study supports the safety and tolerability of BNT162b2 and mRNA-1273 in practice, further strengthening the case for the rapid and broad distribution of vaccines to the public.
Methods
Study design, setting and population
This is a retrospective study of individuals who underwent polymerase chain reaction (PCR) testing for suspected SARS-CoV-2 infection at the Mayo Clinic and hospitals affiliated with the Mayo Clinic Health System. This study was reviewed by the Mayo Clinic Institutional Review Board (IRB) and determined to be exempt from the requirement for IRB approval (45 CFR 46.104d, category 4). Subjects were excluded if they did not have a research authorization on file.
The cohorts of vaccinated and unvaccinated individuals eligible for inclusion in this study are identical to the cohorts considered in a previous analysis: “FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system”24. In total, there were 507,525 individuals in the Mayo electronic health record (EHR) database who received a PCR test between February 15, 2020 and February 8, 2021. To obtain the eligible study population, we defined the following inclusion criteria: (1) at least 18 years old; (2) no positive SARS-CoV-2 PCR test before December 1, 2020; (3) resides in a locale (based on Zip code) with at least 25 individuals who have received BNT162b2 or mRNA-1273. This population included 249,708 individuals, of whom 31,623 have received BNT162b2 or mRNA-1273 and 218,085 have no record of COVID-19 vaccination. Vaccinated individuals who had tested positive for SARS-CoV-2 by PCR between December 1, 2020 and the date of their first vaccine dose were excluded, resulting in 31,299 individuals. Individuals with zero follow-up days after vaccination (i.e. those who received the first vaccine dose on the date of data collection) were also excluded, leaving 31,069 individuals in the final vaccinated cohort.
The propensity matched unvaccinated cohort was selected from the previously derived set of 218,085 unvaccinated individuals. The purpose of this cohort was to establish the baseline frequency of EHR documentation for each adverse effect of interest in a cohort which is clinically similar to our vaccinated cohort. These baseline, or expected, frequencies can then be compared to the observed frequencies to determine whether or not these adverse effects are reported at unexpectedly high rates among patients receiving a COVID-19 vaccine.
From these cohorts of 31,069 vaccinated and propensity matched unvaccinated individuals, we removed individuals who received a positive SARS-CoV-2 PCR test within the first four days after their first actual or assigned vaccine dose (see below). This filtering step was done because it is known that the onset of disease associated symptoms, some of which overlap with vaccine side effects, can precede the confirmed diagnosis of COVID-19 by PCR25. Accordingly, the cause of symptoms observed during this time period for such patients would be ambiguous, as they could reasonably stem from vaccination or infection. After this filter, there were 31,029 and 30,933 individuals in the final vaccinated and unvaccinated cohorts, respectively.
The selection algorithm and its associated counts are summarized in Figure 1. More details on the propensity score matching procedure are provided in the prior manuscript24. This includes a table showing that the 1-to-1 matched cohorts (n = 31,069 each) were indeed balanced for all covariates included in propensity score matching (standardized mean difference < 0.1 for each one) and figures illustrating the distributions of age and total follow-up time between the first vaccine dose and the end of the data collection period.
Schematic illustration of participant selection and study design. A vaccinated cohort and 1-to-1 propensity matched unvaccinated cohort (n = 31,069 each) were previously derived to assess vaccination efficacy in a separate study. With the additional specification that an individual must not have had a positive SARS-CoV-2 PCR test with the first four days after their actual or assigned vaccination date, we were left with 31,029 vaccinated individuals and 30,933 unvaccinated individuals for this safety analysis. For each cohort, the incidence rates of several adverse effects (e.g. arthralgia) were calculated for the seven days following the first dose and, separately, for the seven days following the second dose. For a given adverse effect, the incidence rate ratio and its corresponding 95% confidence interval (CI) were calculated to determine whether one cohort was more likely to experience the event than the other. Incidence rates and incidence rate ratios were also calculated for the 14 and 21 days following each vaccine dose.
Definition of time intervals for safety analyses
For each vaccinated individual, we defined the date of their first vaccine dose as Day V1 and the date of their second vaccine dose as Day V2. In the Results section, these are referred to as “actual” dates of vaccination. For each unvaccinated individual, Day V1 and Day V2 were designated as identical to their matched vaccinated individual. In the Results section, these are referred to as “assigned” dates of vaccination. Individuals were considered for the post-second dose analyses if they (1) were part of the initial cohorts, (2) received a second vaccine dose (vaccinated cohort) or were matched to an individual who received a second vaccine dose (unvaccinated cohort), and (3) were not diagnosed with COVID-19 before Day V2 or within the first four days after Day V2. The third criterion was applied for a similar reason as explained above for the initial cohorts: if an individual tested positive for SARS-CoV-2 on or before the fourth day after the second vaccination, then any symptoms observed after the second actual or assigned vaccination date would have an ambiguous cause.
Definition of adverse effects of interest
The adverse effects considered were primarily derived from those assessed in Phase III trials of BNT162b2 and mRNA-12734,5, including fatigue, fever, chills, myalgia, arthralgia, headache, lymphadenopathy, erythema, diarrhea, vomiting, and local pain and swelling. We also included anaphylaxis and facial paralysis (Bell’s palsy), as these rare events have been reported in individuals receiving COVID-19 vaccines as well. Each adverse effect was mapped to a set of synonyms intended to capture the most common ways that a given phenotype would be referenced in the context of a clinical note.
Curation of adverse effects from clinical notes
To curate the adverse effects experienced by each patient from the electronic health record, we used a BERT-based neural network model26 to classify the sentiment for the phenotypes (described above) mentioned in the clinical notes. Specifically, this classification model categorizes phenotype-containing sentences into one of four categories: (1) confirmed diagnosis, (2) ruled-out diagnosis, (3) possibility of disease, and (4) alternate context (e.g. family history). This classification model was trained on 18,500 sentences and has shown an out-of-sample accuracy of 93.6% with precision and recall scores above 95%25. For each individual, we applied the sentiment model to the clinical notes in the Mayo Clinic electronic health record during our defined intervals of interest for each individual: (1) Day V1 to 7, 14, or 21 days after Day V1, and (3) Day V2 to 7, 14, or 21 days after Day V2. For each phenotype, we identified the first date on which the given individual had at least one sentence in which the phenotype was categorized as “confirmed diagnosis” with a confidence score of at least 90%. For the severe phenotype anaphylaxis, each such sentence was manually reviewed to verify the positive sentiment (i.e. confirmed diagnosis) and to assess the tense of this sentiment (i.e. past vs. present). Only sentences which confirm a present diagnosis were used to count anaphylaxis events in this study.
Evaluating rates of return to clinic after vaccination
To evaluate the likelihood of returning to the clinic after vaccination, we counted the number of individuals who had at least one clinical note in the 7, 14, and 21 days after Day V1 and Day V2. The fraction of individuals with clinical follow-up was calculated as the number of individuals with at least clinical note in the time window divided by the total number of individuals in each group. The difference in clinical follow-up rates was assessed by calculating the odds ratio (OR) along with its corresponding 95% confidence interval (CI). The null hypothesis was that the OR falls between 0.91 and 1.1, inclusive (i.e. the larger rate is at most 10% larger than the smaller rate); thus, an OR was considered significant if the upper bound of the 95% CI was less than 0.91 or the lower bound of the 95% CI was greater than 1.1.
To evaluate emergent clinical follow-up, we compared the number and percentage of vaccinated and unvaccinated individuals who contributed at least one Emergency Department (ED) note in the 1, 7, 14, or 21 days after each actual or assigned vaccine administration date. The odds ratio and corresponding 95% CI was calculated for each time window, with the null hypothesis stating that the OR falls between 0.91 and 1.1, inclusive (as above). We also determined the total number of ED notes from vaccinated and unvaccinated individuals in these same time windows, along with the fraction of ED notes relative to all clinical notes. The difference in the fraction of ED notes was assessed by computing an odds ratio and corresponding 95% CI, with the null hypothesis stating that the OR falls between 0.91 and 1.1, inclusive. ED notes included those under the following categories in the EHR system: ED Notes, ED Procedure Note, ED Provider Notes, ED Provider Triage Note, ED Re-evaluation Note, and ED Triage Notes. All odds ratios and 95% confidence intervals were calculated using the fisher.test function in R version 4.0.327.
Evaluating vaccine adverse effects among the total cohorts
To evaluate adverse effects associated with receiving a COVID-19 vaccine in the clinical setting, we compared the two populations described above and summarized in Figure 1: (1) 31,029 individuals with follow-up who received BNT162b2 or mRNA-1273 and did not have a positive SARS-CoV-2 PCR test prior to vaccination (“vaccinated”), and (2) 30,933 propensity matched individuals who have never received a COVID-19 vaccine and did not have a positive SARS-CoV-2 PCR test before the first vaccination date of their matched individual (“unvaccinated”).
The association of a given adverse effect to vaccination status was assessed by computing the incidence rate ratio (IRR) of the vaccinated and unvaccinated cohorts during several defined time intervals. Specifically, we evaluated adverse effects which were documented in clinical notes within 7, 14, or 21 days of receiving the first vaccine dose (Day V1 to 7, 14, or 21 days after Day V1) or the second vaccine dose (Day V2 to 7, 14, or 21 days after Day V2). For each cohort in a defined time period, incidence rates were calculated as the number of individuals experiencing the given adverse effect in that time period divided by the total number of at risk person-days contributed in that time period. For each individual, at-risk person-days are defined as the number of days from the start of the time period until the day on which the individual experienced the adverse effect or died, or four days prior to testing positive for SARS-CoV-2. The IRR was calculated as the incidence rate of the vaccinated cohort divided by the incidence rate of the unvaccinated cohort, and its 95% CI was computed using an exact approach described previously28. The IRR was considered to be statistically significant if the 95% CI did not include 1.
Comparison of adverse effect rates per EHR versus clinical trial and V-safe reporting
We aimed to compare the percentage of patients with an EHR-documented occurrence of each adverse event to the percentage of patients who reported the corresponding adverse event in the Phase 3 trials of BNT162b2 and mRNA-1273, and in V-safe (a voluntary and active post-marketing surveillance effort to assess the safety of COVID-19 vaccines)4,5. To do so, we extracted the number of patients reporting each adverse effect (of any severity level) within 7 days of the first or second vaccine doses from the clinical trial Solicited Safety Set (mRNA-1273; NDose 1 = 14,808, NDose 2 = 14,733) or Reactogenicity Subset (BNT162b2; NDose 1 = 4,093, NDose 2 = 3,758). We also extracted the percentage of patients reporting each adverse effect through January 14, 2021 in response to daily text message-based health screenings as part of V-safe for the week following each dose of vaccination29. We compared these clinical trial and post-marketing surveillance adverse effect reporting rates to the rates of EHR-documented adverse effects within 7 days of the first or second doses. Reporting-to-Documentation ratios were calculated for each adverse effect in each time interval by dividing the 7 day reporting rate in the given clinical trial (or V-safe) by the 7 day EHR documentation rate in our cohort. 95% confidence intervals around these ratios were computed with a standard delta-method-based approach30.
Evaluating vaccine adverse effects in individuals who returned to clinic
Because only a fraction of individuals in the vaccinated and unvaccinated cohorts contributed clinical notes during the defined intervals after their actual or assigned vaccination dates, we also assessed the rates of adverse effects documented in the EHR specifically among the individuals who had returned to clinic. To select the cohorts for this analysis, we considered any pair of matched vaccinated and unvaccinated individuals who both contributed at least one clinical note which contained at least one phenotype term in the 7, 14, or 21 days after the actual or assigned date of first or second vaccination. The number of such matched pairs in each time interval is provided in Table S1. For each time interval, we then computed incidence rates for each adverse effect in the vaccinated and unvaccinated cohorts, along with the incidence rate ratios and 95% CI as described above. The IRR was considered to be statistically significant if the 95% CI did not include 1.
Results
Vaccinated individuals do not return to the clinic or emergency department at unexpectedly high rates after either vaccine dose
To assess rates of clinical follow-up after study enrollment, we compared the number of vaccinated and unvaccinated individuals with EHR notes within 7, 14, or 21 days of each actual or assigned vaccination date. In the 7 days after the first vaccine dose, 6,081 of the 31,029 (19.6%) vaccinated individuals had at least one EHR note, compared to 5,653 of 30,933 (18.3%) unvaccinated individuals (Odds Ratio = 1.09; 95% CI: [1.05-1.13]) (Table 1). This difference was not considered significant as the lower bound of the 95% CI was less than 1.1 (see Methods). The rates of return to clinic were also similar between these cohorts within 14 and 21 days of the first dose (Odds Ratio14 Days = 1.13 [1.09-1.17]; Odds Ratio21 Days = 1.14 [1.10-1.18]) (Table 1).
Rates of return to clinic in vaccinated and unvaccinated individuals after first vaccination. The first two columns give the numbers and percent of individuals in each group who had at least one phenotype-containing EHR note with 7, 14, or 21 days of the actual or assigned first vaccination date. To assess the magnitude and significance of difference between the follow-up rates, the odds ratio (OR) and corresponding 95% CI are shown. With the null hypothesis that the OR falls between 0.91 and 1.1, a difference was considered significant if the upper bound of 95% CI was less than 0.91 or the lower bound of the 95% CI was greater than 1.1.
Within 7 days of the second dose, 2,496 of 16,982 (14.7%) vaccinated individuals contributed EHR notes, compared to 2,630 (15.8%) of 16,623 unvaccinated individuals (Odds Ratio = 0.92 [0.86-0.97) (Table 2). This difference was not considered significant as the upper bound of the 95% CI was greater than 0.91 (see Methods). Similarly, the rates of return to clinic were comparable between these cohorts within 14 and 21 days of the second dose (Odds Ratio14 Days = 0.90 [0.85-0.95]; Odds Ratio21 Days = 0.91 [0.86-0.96]) (Table 2).
Rates of return to clinic in vaccinated and unvaccinated individuals after second vaccination. The first two columns give the numbers and percent of individuals in each group who had at least one phenotype-containing EHR note with 7, 14, or 21 days of the actual or assigned first vaccination date. To assess the difference in follow-up rates between groups, the odds ratio (OR) and corresponding 95% CI are shown. With the null hypothesis that the OR falls between 0.91 and 1.1, a difference was considered significant if the upper bound of 95% CI was less than 0.91 or the lower bound of the 95% CI was greater than 1.1.
To specifically assess the rates of emergent clinical follow-up, we also compared the number of emergency department (ED) EHR notes contributed by each group. The vaccinated cohort contributed a similar or lower number of ED notes than the unvaccinated cohort in the 1, 7, 14, and 21 days after both the first and second actual or assigned vaccination dates (Tables S2-S3). Consistent with this, fewer vaccinated than unvaccinated individuals contributed at least one ED EHR note within any of these time intervals (Tables S4-S5).
In summary, these findings show that individuals receiving COVID-19 vaccines do not tend to return to the clinic, including the emergency department, in the subsequent weeks at higher-than-expected rates, which suggests favorable tolerability of these vaccines.
Documented vaccine associated adverse effects in EHR notes are rare compared to the frequencies reported in clinical trials
Among the 31,029 and 30,933 individuals in the vaccinated and unvaccinated cohorts, the most commonly documented symptoms in the 7 days after the first vaccine dose included fatigue (542 [1.75%] vaccinated individuals; 592 [1.91%] unvaccinated individuals), nausea (321 [1.03%]; 493 [1.59%]), diarrhea (203 [0.65%]; 247 [0.80%]), myalgia (128 [0.41%]; 123 [0.40%]), arthralgia (198 [0.64%]; 168 [0.54%]), erythema (174 [0.56%]; 184 [0.60%]), vomiting (138 [0.44%]; 261 [0.84%]), and fever (66 [0.21%]; 124 [0.40%]) (Table 3).
For each adverse effect, incidence rates were calculated for the vaccinated (n = 31,029) and propensity matched unvaccinated (n = 30,933) cohorts as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 7-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 128 cases of myalgia were recorded over a total of 216,320 person-days (contributed by 31,029 vaccinated individuals), corresponding to an incidence rate of 0.59 cases per 1000 person-days.
The same events were also the most frequently documented in the 7 days following the second vaccine dose (n = 16,982 in the vaccinated cohort; n = 16,623 for the unvaccinated cohort): fatigue (200 [1.17%] vaccinated individuals; 321 [1.93%] unvaccinated individuals), nausea (143 [0.84%]; 283 [1.70%]), diarrhea (76 [0.45%]; 117 [0.70%]), myalgia (73 [0.43%]; 61 [0.37%]), arthralgia (97 [0.57%]; 91 [0.55%]), erythema (75 [0.44%]; 89 [0.54%]), vomiting (49 [0.29%]; 165 [0.99%]), and fever (31 [0.18%]; 63 [0.38%]) (Table 4).
For each adverse effect, incidence rates were calculated for the vaccinated (n = 16,982) and propensity matched unvaccinated (n = 16,623) cohorts as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 7-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 73 cases of myalgia were recorded over a total of 118,575 person-days (contributed by 16,982 vaccinated individuals), corresponding to an incidence rate of 0.62 cases per 1000 person-days.
Notably, these rates of adverse effects documented in EHR notes were markedly lower than the rates of adverse effects observed in clinical trials and those currently captured in V-safe 18,29,31,32. Specifically, the rates of EHR documentation of adverse effects within 7 days of the first dose were 2.1 (95% CI: 1.5-3) to 910 (95% CI: 500-1,500) times lower than the corresponding reporting rates among the safety populations of the Phase 3 trials in the same time interval, with headache showing the largest discrepancy between these sources (Table 5). These differences were even more pronounced in the week following the second vaccine dose, during which EHR documentation rates were 4.9 (95% CI: 3.3-7.2) to 1,500 (95% CI: 670-2,800) times lower than corresponding reporting rates in the trials (Table 5). This discrepancy is to be expected, as individuals vaccinated outside of the trial or post-marketing surveillance setting are advised that it is normal to experience these adverse effects, and so they are less likely to report them to a healthcare provider. As such, the vaccine associated adverse effects which are captured in EHR notes are likely to be those that are severe or persistent enough to cause an individual to return to the clinic or otherwise notify their health care provider.
Clinical trial reporting rates for each adverse effect were extracted from the FDA Emergency Use Authorization documents for mRNA-1273 (Moderna) and BNT162b2 (Pfizer/BioNTech). For each cohort, the values shown are the patient counts (with percentages) who reported each symptom within 7 days of either the first or second vaccine dose. EHR documentation rates for each adverse effect were derived from Tables 3 and 4, which include the number of individuals who contributed at least one EHR note documenting the occurrence of the given symptom within 7 days of the first (Table 3) or second (Table 4) dose. To obtain the “Reporting to Documentation Ratio”, we divide the appropriate trial-derived reporting percentage by the corresponding EHR-derived documentation percentage.
Adverse effects are not documented more frequently in the EHR notes of vaccinated individuals compared to those of propensity-matched unvaccinated individuals
To test whether documentation of each adverse effect was actually associated with receiving a COVID-19 vaccine, we computed the incidence rate ratio (IRR) for each adverse effect between the vaccinated and propensity matched unvaccinated cohorts. Several symptoms or events including headache, myalgia, arthralgia, erythema, and anaphylaxis had similar incidence rates during the 7 days after the first or second actual or assigned vaccine dose, with the confidence intervals including 1 (Tables 3-4). Other symptoms showed significantly lower incidence rates in the 7 days after both doses in the vaccinated cohort, including fever (IRRdose 1= 0.53 [0.39,-0.72]; IRRdose 2 = 0.48 [0.30-0.74]), nausea (IRRdose 1 = 0.65 [0.56-0.75]; IRRdose 2 = 0.49 [0.40-0.60]), vomiting (IRRdose 1 = 0.53 [0.42-0.65]; IRRdose 2 = 0.29 [0.2-0.4]), diarrhea (IRRdose 1 = 0.82 [0.68-0.99]; IRRdose 2 = 0.63 [0.47-0.85]), fatigue (IRRdose 1 = 0.91 [0.81-1.0]; IRRdose 2 = 0.6 [0.50-0.72]), chills (IRRdose 1 = 0.61 [0.42-0.88]; IRRdose 2 = 0.67 [0.43-1.0]), and lymphadenopathy (IRRdose 1 = 0.65 [0.48-0.87]; IRRdose 2 = 0.52 [0.33-0.81]).
These trends of vaccinated individuals showing similar or lower incidence rates for each considered adverse effect were also true in the 14 and 21 days following actual or assigned vaccination dates (Tables S6-S9). In summary, this data supports the tolerability of BNT162b2 and mRNA-1273, as individuals receiving COVID-19 vaccines do not return to the clinic to report adverse effects at higher rates than propensity matched unvaccinated individuals.
Among matched pairs who return to clinic, adverse effects are reported at similar rates in the vaccinated and unvaccinated cohorts
To complement our previous analysis, we also considered the rates of adverse effect documentation in only the subsets of matched pairs who each contributed at least one EHR note in the 7, 14, or 21 days following the first or second actual or assigned vaccine dose. In the 7 days after the first actual or assigned vaccination date (n = 1,399 propensity matched pairs) the IRR 95% CIs for most adverse effects included 1 (Table 6). Only nausea, vomiting and diarrhea showed differences in incidence rates, with lower incidence rates in the vaccinated cohort (IRRnausea = 0.59 [0.41-0.83]; IRRdiarrhea = 0.50 [0.30-0.80]; IRRvomiting = 0.59 [0.37-0.94]). (Table 6). After the second vaccination date (n = 897 propensity matched pairs), myalgia was documented more frequently among vaccinated individuals (IRRmyalgia = 2.5 [1.1-6.7]), while all other adverse effects were recorded at similar or lower rates in vaccinated compared to unvaccinated individuals (Table 7).
For each adverse effect, incidence rates were calculated for the vaccinated and propensity matched unvaccinated cohorts (n = 1,399 pairs) as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 7-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 31 cases of myalgia were recorded over a total of 9,623 person-days (contributed by 1,399 vaccinated individuals), corresponding to an incidence rate of 3.2 cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the vaccinated and propensity matched unvaccinated cohorts (n = 897 pairs) as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 7-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 21 cases of myalgia were recorded over a total of 6,191 person-days (contributed by 897 vaccinated individuals), corresponding to an incidence rate of 3.4 cases per 1000 person-days.
Similarly, vaccinated individuals who returned to the clinic also showed comparable or lower incidence rates for each considered adverse effect in the 14 and 21 days post vaccination compared to their matched unvaccinated individuals who also returned to the clinic (Tables S10-S13). Taken together, this analysis further corroborates the conclusion that most adverse effects, with the potential exception of myalgia in the week following second vaccination, are not documented in the EHRs of vaccinated individuals at unexpectedly high rates.
Discussion
This study demonstrates that the two currently FDA-authorized COVID-19 vaccines, BNT162b2 and mRNA-1273, are safe and tolerated beyond the confines of a clinical trial setting. This conclusion is consistent with the extensive safety and tolerability assessments conducted in Phase 1/2 and Phase 3 trials over the past nine months and in the post-marketing surveillance efforts which are currently under way4,5,18,33,34. Here we assessed real world safety and tolerability by longitudinally curating the EHR documentation of adverse effects in 31,029 individuals receiving at least one dose of a COVID-19 vaccine compared to a propensity matched unvaccinated cohort of similar size. Compared to this control cohort, vaccinated individuals were not more likely to return to the clinic (including the emergency department) or to report any of the surveyed adverse effects within 7, 14, or 21 days after the first or second vaccine dose. When considering only the subsets of matched pairs who each had at least one clinical note in these defined time intervals, myalgia was the only symptom with a modest increase in documentation frequency among vaccinated individuals, while all other potential side effects occurred at similar or lower rates in this cohort.
The purpose of using propensity matching in this study was to establish an expected frequency for each potential vaccine associated adverse effect in a group of individuals with similar demographic and geographic characteristics, akin to a placebo group in a randomized clinical trial. Our finding that EHR notes from vaccinated and propensity matched unvaccinated individuals record similar rates of potential vaccine associated adverse effects differs from the data obtained in Phase 3 trials, wherein vaccinated participants experienced higher rates of symptoms than those receiving placebo. Further, the absolute rates of adverse effects documented in these EHR notes are well below the rates reported in clinical trials (see Table 5) and in the active post-marketing surveillance of these vaccines (V-safe) (Table S14). These discrepancies are likely attributable to differences in the populations analyzed and the methodology of reporting symptoms.
Regarding the populations analyzed, it is important to realize that adverse effects experienced by healthcare workers, who comprise a significant proportion of the Phase 1a vaccinated population thus far, are generally less likely to be documented in EHR notes. At many institutions, healthcare workers experiencing potential vaccine side effects are directed to follow up with Occupational Health Services, which is an entity of the employer and thus does not document the reported symptoms in EHR notes. Several institutions have also established COVID-19 response lines, which will usually direct individuals experiencing nonspecific vaccine side effects toward self-care. Further, healthcare workers may be less likely to report adverse effects at baseline given their clinical expertise and ability to self-assess the severity of their illness.
Regarding reporting methods, both trials included a 7-day post-vaccination period in which symptoms were actively solicited from some or all individuals as well as longer periods in which unsolicited adverse effects and serious adverse reactions were recorded from all individuals. V-safe similarly solicits adverse effect reports from its voluntary participants on a regular basis. In contrast, our methods rely exclusively on the recording of unsolicited symptoms or events in the EHR. Given that individuals are warned of the likely vaccine associated adverse effects at the time of vaccination in the real world setting, it is likely that most mild or moderate symptoms are never actually reported and thus are not documented in an EHR note.
That said, serious safety concerns by definition require medical care and thus are likely to be documented in the EHR. For example, should an individual experience anaphylaxis, this individual will likely require emergent care, during which one or more clinical notes will be written and will mention this phenotype. Thus, our method should identify the symptoms and phenotypes that represent the most serious threats to vaccine safety and tolerability of practical significance. Indeed, this is the central reason why our analysis should be viewed as complementary to the data which has been obtained in the more controlled setting of clinical trials. It is extremely important to know and understand the frequencies of side effects which were observed via solicitation in the trial setting, and this study does not contradict or refute this data in any way. Rather, our assessment specifically aims to describe the frequencies of adverse effects that receive some form of clinical attention as evidenced by their documentation in the EHR.
In light of this, it is worth noting that the rate of severe side effects was actually similar among participants receiving vaccine or placebo in the Pfizer/BioNTech and Moderna Phase 3 trials at 2% or less, with the exception of severe fatigue in 3.8% of participants after their second dose of BNT162b24,5. This is consistent with our observations that vaccinated and unvaccinated individuals have similar EHR documentation rates for each surveyed potential vaccine associated symptom. And our finding that vaccinated individuals are not more likely to receive emergent care provides further orthogonal support for the safety of these vaccines.
There are several limitations of this study. First, while the analysis was conducted on a population derived from a large healthcare system, the cohort demographics are not representative of the American population. For example, both the vaccinated and unvaccinated cohorts were predominantly Caucasian (>90%) and female (>60%). These biases likely reflect both the populations who receive care at the various Mayo Clinic centers and the populations who have been prioritized in Phase 1a of the vaccine rollout. The enrichment of healthcare workers among individuals vaccinated in this phase likely leads to an underestimation of the rates of return to clinic, due to the factors previously discussed (i.e. access to an Occupational Health Services office and other institution-specific COVID-19 response centers). However, it is worth noting that even those individuals who utilize such services are likely to generate EHR notes if they are indeed determined to require clinical attention or intervention. Second, the BERT model used to curate EHR notes does not imply a direct link between COVID-19 vaccination and the experience of a phenotype. That is, we simply capture the occurrence of an adverse effect without ensuring that the clinical note indeed suggests or confirms that vaccination caused the symptom. This shortcoming is addressed by comparing vaccinated individuals to the unvaccinated control cohort, which establishes a baseline expected frequency for each symptom in the absence of vaccination. Finally, while sentences suggesting the occurrence of anaphylaxis were manually reviewed to confirm both the positive sentiment and the tense (see Methods), sentences for the other curated phenotypes were not reviewed. In the future, we will train natural language processing models to discriminate past from present tense, thereby circumventing the need for this manual review.
As the remainder of the US population undergoes COVID-19 vaccination, it will not be feasible to conduct solicited reporting of adverse effects in all vaccinated individuals. Augmented curation of EHR notes for real world safety monitoring presents a practical solution to this problem which can be deployed to complement other directed surveillance efforts like V-safe. The method demonstrated here represents a scalable approach to continuously monitor serious safety concerns associated with any authorized COVID-19 vaccines. Taken together with our recent study highlighting the real world efficacy of these vaccines24, this data reinforces that individuals, providers, and public health officials should proceed rapidly with vaccination efforts, with high confidence in their efficacy and safety.
Data Availability
After publication, the data will be made available upon reasonable requests to the corresponding author. A proposal with detailed description of study objectives and the statistical analysis plan will be needed for evaluation of the reasonability of requests. Deidentified data will be provided after approval from the corresponding author and the Mayo Clinic.
Data Availability
After publication, the data will be made available upon reasonable requests to the corresponding author. A proposal with detailed description of study objectives and the statistical analysis plan will be needed for evaluation of the reasonability of requests. Deidentified data will be provided after approval from the corresponding author and the Mayo Clinic.
Declaration of Interests
RM, PL, ES, AP, SA, CP, VA, AJV, PA, AR, CC, KC, DD, NK, ER, GB, AM, TW, and VS are employees of nference and have financial interests in the company and in the successful application of this research. JCO receives personal fees from Elsevier and Bates College, and receives small grants from nference, Inc, outside the submitted work. ADB is a consultant for Abbvie, is on scientific advisory boards for nference and Zentalis, and is founder and President of Splissen therapeutics. JH, JCO, GJG, AWW, AV, MDS, and ADB are employees of the Mayo Clinic. The Mayo Clinic may stand to gain financially from the successful outcome of the research. This research has been reviewed by the Mayo Clinic Conflict of Interest Review Board and is being conducted in compliance with Mayo Clinic Conflict of Interest policies.
Author Contributions
VS, PL, and SA conceived the study. RM, PL, AJV and VS wrote the manuscript and reviewed the findings. ES, AP, SA, CP, VA, PA, AR, CC, KC, DD, NK, ER, GB, AM, TW contributed methods, analysis, and software. JCOH, GJG, AWW, ADB, MDS, AV, and JH reviewed the study, findings, and the manuscript. All authors revised the manuscript.
Supplemental Material
The first column gives the time interval: within 7, 14, or 21 days. The second column gives the number of matched pairs with at least one clinical note in the defined time interval after the first actual or assigned vaccination date. The third column gives the number of matched pairs with at least one clinical note in the defined time interval after the second actual or assigned vaccination date.
The percentages shown were obtained by dividing the number of ED notes by the total number of clinical notes for the cohort in the given time interval, and multiplying by 100. To assess the magnitude and significance of difference between the percentages of ED notes, the odds ratio (OR) and corresponding 95% CI are shown. With the null hypothesis that the OR falls between 0.91 and 1.1, a difference was considered significant if the upper bound of 95% CI was less than 0.91 or the lower bound of the 95% CI was greater than 1.1.
The percentages shown were obtained by dividing the number of ED notes by the total number of clinical notes for the cohort in the given time interval, and multiplying by 100. To assess the magnitude and significance of difference between the percentages of ED notes, the odds ratio (OR) and corresponding 95% CI are shown. With the null hypothesis that the OR falls between 0.91 and 1.1, a difference was considered significant if the upper bound of 95% CI was less than 0.91 or the lower bound of the 95% CI was greater than 1.1.
Percentages shown are obtained by dividing the number of individuals with at least one ED note by the total number of individuals in the cohort, and multiplying by 100. To assess the magnitude and significance of difference between the percentages of patients with ED notes, the odds ratio (OR) and corresponding 95% CI are shown. With the null hypothesis that the OR falls between 0.91 and 1.1, a difference was considered significant if the upper bound of 95% CI was less than 0.91 or the lower bound of the 95% CI was greater than 1.1.
Percentages shown are obtained by dividing the number of individuals with at least one ED note by the total number of individuals in the cohort, and multiplying by 100. To assess the magnitude and significance of difference between the percentages of patients with ED notes, the odds ratio (OR) and corresponding 95% CI are shown. With the null hypothesis that the OR falls between 0.91 and 1.1, a difference was considered significant if the upper bound of 95% CI was less than 0.91 or the lower bound of the 95% CI was greater than 1.1.
For each adverse effect, incidence rates were calculated for the vaccinated (n = 31,029) and propensity matched unvaccinated (n = 30,933) cohorts as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 14-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 164 cases of myalgia were recorded over a total of 431,530 person-days (contributed by 31,029 vaccinated individuals), corresponding to an incidence rate of 0.38 cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the vaccinated (n = 16,982) and propensity matched unvaccinated (n = 16,623) cohorts as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 14-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 70 cases of myalgia were recorded over a total of 237,001 person-days (contributed by 16,982 vaccinated individuals), corresponding to an incidence rate of 0.3 cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the vaccinated (n = 31,029) and propensity matched unvaccinated (n = 30,933) cohorts as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 21-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 255 cases of myalgia were recorded over a total of 645,745 person-days (contributed by 31,029 vaccinated individuals), corresponding to an incidence rate of 0.39cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the vaccinated (n = 16,982) and propensity matched unvaccinated (n = 16,623) cohorts as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 21-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 98 cases of fatigue were recorded over a total of 355,085 person-days (contributed by 16,982 vaccinated individuals), corresponding to an incidence rate of 0.28 cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the propensity matched vaccinated and unvaccinated individuals (n = 2,643 pairs) as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 14-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 65 cases of myalgia were recorded over a total of 36,105 person-days (contributed by 2,643 vaccinated individuals), corresponding to an incidence rate of 1.8 cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the propensity matched vaccinated and unvaccinated individuals (n = 1,717 pairs) as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 14-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 36 cases of myalgia were recorded over a total of 23,720 person-days (contributed by 1,717 vaccinated individuals), corresponding to an incidence rate of 1.5 cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the propensity matched vaccinated and unvaccinated individuals (n = 3,553 pairs) as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 21-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 100 cases of myalgia were recorded over a total of 72,586 person-days (contributed by 3,553 vaccinated individuals), corresponding to an incidence rate of 1.4 cases per 1000 person-days.
For each adverse effect, incidence rates were calculated for the propensity matched vaccinated and unvaccinated individuals (n = 2,360 pairs) as the number of positive cases divided by the total number of at risk person days during this time period. Individuals were considered at risk for developing an adverse effect from their actual or assigned date of first vaccination until they experienced the event, died, or reached the end of the 21-day study period, or until four days prior to a positive SARS-CoV-2 test. For example, in the first row, we show that 45 cases of myalgia were recorded over a total of 48,944 person-days (contributed by 2,360 vaccinated individuals), corresponding to an incidence rate of 0.92 cases per 1000 person-days.
V-Safe reporting data summarized through January 14, 2021 were extracted from a publicly available Advisory Committee on Immunization Practices (ACIP) powerpoint for mRNA-1273 (Moderna) and BNT162b2 (Pfizer/BioNTech). For the Moderna and Pfizer/BioNTech cohorts, the values shown are the percentages of patients who reported each symptom within 7 days of either the first dose (for those receiving the Moderna vaccine) or the first or second vaccine dose (for those receiving the Pfizer/BioNTech vaccine). Patient counts are not provided for these cohorts because only percentages were available in the source V-Safe dataset. At the time of this most recent V-Safe data summary, 2nd doses of the Moderna vaccine had not been administered and thus are not included. EHR documentation rates for each adverse effect were derived from Tables 3 and 4, which include the number of individuals who contributed at least one EHR note documenting the occurrence of the given symptom within 7 days of the first (Table 3) or second (Table 4) dose. To obtain the “Reporting to Documentation Ratio”, we divide the appropriate trial-derived reporting percentage by the corresponding EHR-derived documentation percentage.
Acknowledgements
The authors thank Murali Aravamudan for the careful review and feedback on this manuscript.
Footnotes
↵+ Joint first authors
A discussion paragraph has been added the clinical trials to real world evidence comparison for adverse events. The title and abstract have been made consistent with the manuscript that is currently undergoing peer-review.