Disentangling post-vaccination symptoms from early COVID-19 =========================================================== * Liane S. Canas * Marc F. Österdahl * Jie Deng * Christina Hu * Somesh Selvachandran * Lorenzo Polidori * Anna May * Erika Molteni * Benjamin Murray * Liyuan Chen * Eric Kerfoot * Kerstin Klaser * Michela Antonelli * Alexander Hammers * Tim Spector * Sebastien Ourselin * Claire Steves * Carole H. Sudre * Marc Modat * Emma L. Duncan ## Abstract **Background** Identifying and testing individuals likely to have SARS-CoV-2 is critical for infection control, including post-vaccination. Vaccination is a major public health strategy to reduce SARS-CoV-2 infection globally. Some individuals experience systemic symptoms post-vaccination, which overlap with COVID-19 symptoms. This study compared early post-vaccination symptoms in individuals who subsequently tested positive or negative for SARS-CoV-2, using data from the COVID Symptom Study (CSS) app. **Design** We conducted a prospective observational study in UK CSS participants who were asymptomatic when vaccinated with Pfizer-BioNTech mRNA vaccine (BNT162b2) or Oxford-AstraZeneca adenovirus-vectored vaccine (ChAdOx1 nCoV-19) between 8 December 2020 and 17 May 2021, who subsequently reported symptoms within seven days (other than local symptoms at injection site) and were tested for SARS-CoV-2, aiming to differentiate vaccination side-effects *per se* from superimposed SARS-CoV-2 infection. The post-vaccination symptoms and SARS-CoV-2 test results were contemporaneously logged by participants. Demographic and clinical information (including comorbidities) were also recorded. Symptom profiles in individuals testing positive were compared with a 1:1 matched population testing negative, including using machine learning and multiple models including UK testing criteria. **Findings** Differentiating post-vaccination side-effects alone from early COVID-19 was challenging, with a sensitivity in identification of individuals testing positive of 0.6 at best. A majority of these individuals did not have fever, persistent cough, or anosmia/dysosmia, requisite symptoms for accessing UK testing; and many only had systemic symptoms commonly seen post-vaccination in individuals negative for SARS-CoV-2 (headache, myalgia, and fatigue). **Interpretation** Post-vaccination side-effects per se cannot be differentiated from COVID-19 with clinical robustness, either using symptom profiles or machine-derived models. Individuals presenting with systemic symptoms post-vaccination should be tested for SARS-CoV-2, to prevent community spread. **Funding** Zoe Limited, UK Government Department of Health and Social Care, Wellcome Trust, UK Engineering and Physical Sciences Research Council, UK National Institute for Health Research, UK Medical Research Council and British Heart Foundation, Alzheimer’s Society, Chronic Disease Research Foundation, Massachusetts Consortium on Pathogen Readiness (MassCPR). **Evidence before this study** There are now multiple surveillance platforms internationally interrogating COVID-19 and/or post-vaccination side-effects. We designed a study to examine for differences between vaccination side-effects and early symptoms of COVID-19. We searched PubMed for peer-reviewed articles published between 1 January 2020 and 21 June 2021, using keywords: “COVID-19” AND “Vaccination” AND (“mobile application” OR “web tool” OR “digital survey” OR “early detection” OR “Self-reported symptoms” OR “side-effects”). Of 185 results, 25 studies attempted to differentiate symptoms of COVID-19 vs. post-vaccination side-effects; however, none used artificial intelligence (AI) technologies (“machine learning”) coupled with real-time data collection that also included comprehensive and systematic symptom assessment. Additionally, none of these studies attempt to discriminate the early signs of infection from side-effects of vaccination (specifically here: Pfizer-BioNTech mRNA vaccine (BNT162b2) and Oxford-AstraZeneca adenovirus-vectored vaccine (ChAdOx1 nCoV-19)). Further, none of these studies sought to provide comparisons with current testing criteria used by healthcare services. **Added value of this study** This study, in a uniquely large community-based cohort, uses prospective data capture in a novel effort to identify individuals with COVID-19 in the immediate post-vaccination period. Our results show that early symptoms of SARS-CoV-2 cannot be differentiated from vaccination side-effects robustly. Thus, post-vaccination systemic symptoms should not be ignored, and testing should be considered to prevent COVID-19 dissemination by vaccinated individuals. **Implications of all the available evidence** Our study demonstrates the critical importance of testing symptomatic individuals - even if vaccinated – to ensure early detection of SARS-CoV-2 infection, helping to prevent future pandemic waves in the UK and elsewhere. Keywords * COVID-19 detection * Vaccination * Side-effects * Self-reported symptoms * Mobile technology * Early detection * severe acute respiratory syndrome-related coronavirus 2 (SARS-CoV-2) ## Introduction The havoc wrought by SARS-CoV-2 is unprecedented in living memory, with >184 million cases of COVID-19 world-wide and >4.0 million deaths by 8 July 2021.1,2 Extraordinary efforts directed towards rapid vaccine development meant that by late 2020 the UK Medicines and Healthcare products Regulatory Agency had authorized three vaccines: Pfizer-BioNTech mRNA (BNT162b2)3,4 Oxford-AstraZeneca adenovirus-vectored5–7 and Moderna mRNA (mRNA-1273)8,9 A fourth vaccine (Janssen adenovirus-vectored Ad26.COV2.S) was authorised on 28 May 2021.10 Vaccination with BNT162b2 (herein, PB) and ChAdOx1 nCoV-19 (herein, O-AZ) started in the UK on 8 December 202011 and 4 January 202112 respectively, during which time the UK was experiencing its third pandemic wave with widespread community transmission (peak UK positive specimens reported on 29 December 202013). Since then, and in the context of social distancing and stay-at-home directives, new infections, hospitalisations, and deaths from SARS-CoV-2 have fallen rapidly. 1,2,14 Local and systemic reactions have been observed after all vaccines for SARS-CoV-2. Considering the two vaccines used predominantly in the UK to date (O-AZ and PB), local reactions were common during their pivotal trials (76% of younger (<55 years) O-AZ recipients reported tenderness;5,6 83% of younger PB recipients reported pain).4 Systemic reactions were also common and included fatigue (O-AZ 76%; PB 59%), headache (O-AZ 65%; PB 52%), and fever (O-AZ 24%; PB 16%).4–6 Observational data from the COVID Symptom Study (CSS)15 also showed high incidence of local (62%) and systemic (26%) effects.16 The most serious side-effect to date, observed after both O-AZ and Janssen vaccines, is vaccine-induced immune thrombotic thrombocytopenia (VITT) associated with anti-PF4 antibody production.17,18 As of 9 June 2021, with 69,743,980 vaccinations administered in the UK (>45 million O-AZ, including first and second doses), 395 cases of VITT have been reported, with 70 deaths.19 Saliently, most vaccine-related side-effects (including VITT) are more common in younger individuals, whereas COVID-19 clinical severity increases with age.4–6,16 Prevention of SARS-CoV-2 dissemination requires rapid recognition followed by quarantining of infected individuals (along with appropriate health care). However, there is overlap between symptoms from COVID-1920,21 and early post-vaccination systemic symptoms.4–6,16 Moreover, immunity to SARS-CoV-2 does not occur immediately post-vaccination,22 with functional protection from approximately day 12.23 Quarantining and testing every individual with systemic symptoms early post-vaccination would be onerous, expensive, and labour-intensive – but given the impact of viral outbreaks might be unavoidable if SARS-CoV-2 infection cannot be excluded robustly.20,21 Here we aim to determine whether symptom profiles can be used to differentiate individuals with systemic side-effects of vaccination alone from individuals with superimposed SARS-COV-2 infection. ## Methods ### Study design and Participants Data were acquired prospectively from the CSS, using a mobile health application launched by ZOE Limited and King’s College London in March 2020 (app details and development given in Supplementary Methods).15,21 Briefly, individuals are asked daily to log their health status, health care access, SARS-CoV-2 testing and results, and vaccination details, with direct questions about symptoms associated with COVID-19 (Supplementary Table S1).14,15,21 Symptomatic individuals are prompted to undergo testing, either through standard care or through test request from ZOE/CSS.24 Data were acquired from UK participants aged 16-90 years, between 8 December 2020 (UK vaccination start date) and 17 May 2021, who were asymptomatic when vaccinated with PB or O-AZ (first or second dose), and subsequently reported: a) at least one predefined symptom (Supplementary Table S1) within seven days post-vaccination, and b) a SARS-CoV-2 test result (rtPCR or lateral flow antigen test [LFAT]) within ten days post-vaccination. The seven-day cut-off for symptom presentation was informed by: a) serial interval for COVID-19 (mean, 5.2 days25); b) incubation period for SARS-CoV-2 (mean, 5.8 days25,26); c) the timeline for acute post-vaccination side-effects in both pivotal trials (one-week4–6,8,10) and d) reported real-world experience of post-vaccination symptoms (peak prevalence day 1 post-vaccination; mean duration one day16). The ten-day cut-off for testing allowed three days’ delay in accessing testing.27 Early results indicated a large imbalance in numbers of individuals testing positive vs. negative post-vaccination, sufficient to bias analysis.28 A 1:1 population from the negative cohort (matching age, BMI, gender, occupation, week of testing, and comorbidities) was selected based on minimisation of Euclidean distance between positive and negative subjects considering these features, enabling a fair comparison between groups of equal size.29,30 However, to ensure robustness, analyses were repeated using: a) a one-hundred bootstrapping scheme selecting from the negative population; and b) the entire negative population. Individual symptoms (here, a symptom reported at any time within seven days post-vaccination, irrespective of duration) were compared between vaccinated individuals testing positive or negative for SARS-CoV-2, using Chi-squared tests per symptom. Duration of individual symptoms was calculated as days from first report of that symptom, until asymptomatic and/or seven days post-vaccination, noting that duration beyond seven days was not considered; however, as number of individuals experiencing each symptom were low in both groups, no statistical comparison was made. Symptom burden, defined as total symptom count per person [irrespective of symptom duration] were compared between groups using Mann-Whitney-U tests. We also considered symptom manifestation across the week post-vaccination, by dynamic profiling for each symptom (symptom frequency). Distribution of symptom duration was assessed using Mann-Whitney-U. Correlation of individual symptoms within both positive and negative individuals was assessed by computing a Spearman-rank correlation test. Local symptoms due to vaccination *per se* (Supplementary Table S2) were excluded from analysis as unlikely to be indicative of, or influenced by, SARS-CoV-2 infection. Machine learning was used to determine if post-vaccination symptoms *per se* could be separated from superimposed SARS-CoV-2 infection (including symptom combination, and cumulative symptom burden).16,21 We trained a set of binary classifiers to identify SARS-CoV-2-positive individuals. Models included random forest, logistic regression, and Bayesian mixed-effect models, exploiting their varying properties (Supplementary Table S3) to improve reliability of results. We also considered whether symptoms were after first or second vaccination and the impact of vaccination order, by including vaccination ordering as a covariate. Models were trained on data without stratifying by vaccine type, due to small sample sizes. We also did not discriminate between type of SARS-CoV-2 testing (PCR vs. LFAT), or mode of testing access (NHS vs. ZOE-request), either in model development or other analyses. We sought to reduce bias from assessing high numbers of individual symptoms by performing symptom-clustering using K-means. 31 However, a relevant/accurate number of clusters was not evident from the silhouette plot and entropy (data not shown); thus further analyses using machine clustering were not pursued. Symptoms were clustered manually into clinical groupings (reviewed by ELD, MO, TS, AH, CJS) (Supplementary Table S4), and analysed using the above models similarly. Lastly, clustering based on having at least one of the four symptoms required for accessing NHS testing during the timing of this study (viz., presence or absence of fever, persistent cough, anosmia and/or dysosmia)24 were assessed. Data were split into training and validation sets for random forest, logistic regression, and Bayesian mixed-effect models. Five folds were used on the training set, composed of 80% of the initial dataset randomly selected, to train the models in different subsamples of the population. The remaining 20% were then used to assess the performance of models, evaluating sensitivity, specificity, and balanced accuracy. The class ratio was maintained in both training and testing sets. For fair evaluation, both clinical clustering and categorisation using the NHS criteria were assessed on 20% of the data of each fold. ### Role of the funding source ZOE Limited developed the app for data collection as a not-for-profit endeavour. The funder had no role in study design, data analysis, data interpretation, or influence on report content. ### Ethical approval The app and CSS were approved in the UK by KCL’s ethics committee (REMAS no. 18210, review reference LRS-19/20–18210). All app users provided informed consent for use of their data for COVID-19 research. ## Results Figure 1 shows the flow chart for this study. Overall, 1,072,313 UK CSS app users were vaccinated with either O-AZ or PB (O-AZ: 713,651; PB: 358,662). Of these, 362,770 (33.8% (O-AZ: 264,587; PB: 98,183)) reported at least one symptom early post-vaccination, with SARS-CoV-2 testing in 14,842 (4.09%) of these individuals (O-AZ: 10,765; PB: 4,077). A positive test was reported by 150 (1.01%) individuals (O-AZ: 73/10,765 (0.68%); PB: 77/4077 (1.89%)). ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/22/2021.07.21.21260906/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2021/07/22/2021.07.21.21260906/F1) Figure 1. Flowchart of individuals included in this study. Symptoms* within 7 days excluded local symptoms related to injection site. SARS-CoV-2 test included both rtPCR and LFAT. Positive and negative refers to self-logged test results. Within the tested group, 3,525/14,842 (23.75%) reported at least one requisite symptom fulfilling UK testing criteria 24; 62 (1.76%) tested positive. Conversely, 11,317 tested individuals did not report any requisite symptom, of whom 88 (0.78%) tested positive. Individuals with requisite symptoms were more likely to test positive than those without (p-value<0.0001); none-the-less, the majority (88 of 150, 59%) who tested positive did not meet current UK testing criteria. For further analyses, one positive individual (vaccinated with PB) was excluded due to invalid data entry (invalid BMI), leaving 149 symptomatic positively-tested individuals. Table 1 describes the positive and matched negative cohorts. View this table: [Table 1.](http://medrxiv.org/content/early/2021/07/22/2021.07.21.21260906/T1) Table 1. Demographic information of vaccinated individuals testing positive or negative for SARS-CoV-2 infection. Data are presented as median value [IQR] for age and BMI; and numbers (percentages) for other values. *BMI: Body Mass Index. IQR: Inter-Quartile Range*. Four symptomatic individuals who tested positive did so within 10 days of their second vaccination; their symptoms after first vaccination were thus disregarded. As the matched negative control cohort (N=149) included matching for vaccination order, the controls also included four individuals reporting symptoms after second vaccination. Given the small sample size, all 149 subjects in each of the positive and negative groups were included in model training, which included regressing data from vaccination order. However, as we could not conduct a fair statistical analysis of symptoms after first vs. second vaccination for symptom profiling (given the very small numbers of infected individuals presenting after second vaccination) and as post-vaccination symptoms vary after first vs. second dose,16,32 we present the symptom profiles after first vaccination (N=145) in the main text with data from the entire cohort (N=149) in the Supplementary Materials (Supplementary Table S5). Individual symptom prevalence after first vaccination is shown in Figure 2 and Supplementary Table S5. Although some symptoms were more common in individuals testing positive vs. negative (sore throat (p-value = 0.0187), sneezing (p value = 0.0474) and persistent cough (p-value = 0.0396)), others were more common in the negative group (palpitations (p-value = 0.0284). The numbers of individuals reporting each symptom were small (e.g., sore throat, n=17; persistent cough, n =12) (Supplementary Table S5). ![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/22/2021.07.21.21260906/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2021/07/22/2021.07.21.21260906/F2) Figure 2. Profiles of illness in symptomatic individuals early post-vaccination, comparing symptom prevalence (symptom reported at any time during first week) in positive vs. negative cases (1:1 matched population; N=145 for each). *p < 0.05 **p < 0.01. Median day of onset for any symptom post-vaccination was Day 1 in both groups (noting all individuals were asymptomatic when vaccinated), with highest symptom burden on Day 3, again in both groups (Figure 3; and Supplementary Tables S6 and S7). There was no difference in symptom burden between individuals testing positive or negative (median: 12 in positive group, 10.5 in negative group, Mann-Whitney test p-value = 0.22). ![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/22/2021.07.21.21260906/F3.medium.gif) [Figure 3.](http://medrxiv.org/content/early/2021/07/22/2021.07.21.21260906/F3) Figure 3. Symptom prevalence and distribution during the first week after the first dose of vaccination, in symptomatic individuals testing positive or negative for SARS-CoV-2. The colour bar represents the percentage of symptomatic individuals reporting each symptom. Table 2 presents prevalence of each symptom over the week post-vaccination, divided into three windows. Some symptoms increased over time in both positive and negative individuals (e.g., headache, myalgia) whereas others increased in positive individuals only (e.g., sneezing, hoarse voice). Although fever and sore throat increased across the week in the negative individuals, there was a suggestion of a biphasic response in the positive individuals, also observed with persistent cough. The numbers of individuals were too small for formal testing; moreover, the exact date of infection in positive individuals was unknown. View this table: [Table 2.](http://medrxiv.org/content/early/2021/07/22/2021.07.21.21260906/T2) Table 2. Symptom prevalence and timing during the first week post-vaccination (N=145), in symptomatic individuals testing positive or negative for SARS-CoV-2, grouped by days post-vaccination. Number of subjects and percentage (value in parenthesis). Individual symptom duration is shown in Supplementary Table S8. There were no significant differences in symptom duration in positive vs. negative individuals after first vaccination. Importantly, symptom assessment was truncated at seven days, noting as above that some symptoms were increasing in prevalence with time. Interestingly, amongst individuals testing negative, dysosmia and delirium had the longest duration (median, 2 days for each). Similar results for symptom prevalence after first vaccination were obtained comparing the positive population with the constructed cohort (1:1 matched) of negative individuals selected by bootstrapping (Supplementary Table S9) (Supplementary Figure S1); and with the negative population as a whole (Supplementary Figure S2). Some symptoms were significantly more common in negative individuals when using the entire negative population (e.g., brain fog), driven by the extremely large negative sample size, which supports our use of a selected matched population to avoid bias from unbalanced sample size. There was no correlation between symptoms in either the positive or negative populations, assessed using Spearman-rank test (Supplementary Figure S3). As a sensitivity analysis we assessed the impact of a (self-logged) previous COVID-19 diagnosis; this made no difference to our results. ### SARS-CoV-2 test outcome prediction modelling Model performance including receiver operator curves, using all reported symptoms, are shown in Table 3 and Figure 4. The best performance was obtained with random forest, followed by logistic regression; however, neither reached clinical utility. Other models, including clinical symptom clustering (Supplementary Table S4) and categorisation of individuals using NHS screening criteria, were no better than chance. View this table: [Table 3.](http://medrxiv.org/content/early/2021/07/22/2021.07.21.21260906/T3) Table 3. Model performance in the classification of COVID-19 status according to post-vaccination symptoms. Median values and percentiles [0.25 and 0.75] are obtained across five folds. AUC - area under curve, in a receiver operating characteristic analysis. ![Figure 4.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/22/2021.07.21.21260906/F4.medium.gif) [Figure 4.](http://medrxiv.org/content/early/2021/07/22/2021.07.21.21260906/F4) Figure 4. ROC-AUC performance for the different models. Mean value (line) and 95% CI (shadow area) of the models’ performance, given the predictions obtained over the cross-validation scheme (5-folds) adopted for the validation. bMEM: Bayesian Mixed-effect Model (blue); LR: Logistic Regression (yellow); RF: Random Forest (red); Symptoms clustering (green) and NHS screening criteria (grey). ## Discussion Here we aimed to develop a clinically useful algorithm predictive of SARS-CoV-2 infection early post-vaccination, by parsing symptoms according to proven infection status in symptomatic individuals. Such an algorithm would be extremely useful, particularly in countries with limited health resources, as testing could be targeted towards those predicted positive, with quarantining of these individuals until an available result. To our knowledge, this is the first study with this aim. However, we were unable to differentiate post-vaccination symptoms *per se* from superimposed SARS-CoV-2 infection with robustness. Although two models showed ROC AUC significantly greater than 0.5, neither came close to approaching clinical utility - for most clinical tests, conventionally given as 0.8; but for a highly infectious agent with devastating consequences from community spread the necessary AUC is much higher. Although one third of the one million vaccinated app users reported symptoms previously associated with COVID-19 early post-vaccination, only 4% of symptomatic individuals reported testing for SARS-CoV-2 even with allowance for delayed testing access. Considering those individuals who reported at least one of the symptoms fulfilling NHS criteria for testing (266,502 overall), 40% (107,929) were tested. During the study period, testing was widely available in the UK and it is unclear why more symptomatic people (including those with the widely advertised core symptoms) were not tested.33 Conversely, of 149 individuals who tested positive, only 62 (41%) had symptoms that met current UK testing criteria. We do not know why the other 88 positive individuals were tested (e.g., contact tracing, routine workplace testing, direct personal request through the app). Our data also suggest sensitivity of using core symptoms for COVID-19 may be lower post-vaccination than in pre-vaccination times (here 48%, previously 73%.).34 Although individuals with core symptoms were more likely to test positive than those without, the overall sensitivity and AUC suggests current UK testing policy is suboptimal for pandemic management particularly now that rapid testing capacity is much greater than when these criteria were established. 34 Notably, current UK testing criteria are more limited than WHO guidelines2 and that of many other jurisdictions of similar GDP (including France, Germany, USA, and Australia). Although there were some differences in symptom prevalence and distribution between positive and negative individuals, these could not be used robustly to discriminate between groups, including using machine-learning. We also considered time of symptom onset and symptom duration post-vaccination, (previous trials and post-marketing observational data have examined these parameters but not with respect to SARS-CoV-2 status).4–6,8,16 Whether positive or negative, median symptom peak burden was day 3 in both groups, concordant with vaccination side-effect profiles reported previously. 4–6,8,16 As time progressed, some symptoms started to become more common in the positive group only (e.g., persistent cough, hoarse voice), which timing coincides with the serial interval and incubation period of SARS-CoV-2;26 however, the critical public health importance of identifying and isolating cases early does not allow the luxury of a watch-and-wait approach. We do not know the circumstances contributing to infection of the positive group (whether prior to, peri-, or immediately post-vaccination), noting here that we required rtPCR or LFAT results thus confirming recent infection. The serial interval and incubation period for SARS-CoV-2, and the high proportion of asymptomatic infection means that individuals could have been infected before vaccination. Although current UK vaccination guidelines do not require individuals to be completely asymptomatic at time of vaccination (only not “acutely unwell”35), our inclusion criteria required individuals to be asymptomatic at time of vaccination. It is also possible, that infection was contracted whilst getting vaccinated. Nosocomial infection with SARS-CoV-2 has been reported in the UK,36,37 including many health care workers infected at their workplace.38,39 Here we would emphasise strongly that data from ourselves and others indicate that even if infected peri- or post-vaccination, the course of COVID-19 is much less severe in vaccinated vs. unvaccinated individuals;40–44 and - acknowledging that a small percentage of vaccinated and symptomatic individuals were tested - our data demonstrated only 150 cases of confirmed infection (1%) in 14,842 tested individuals from over 1 million vaccinated app users. It is also possible that individuals changed their behaviour immediately after vaccination, increasing infection risk and contracting SARS-CoV-2 prior to acquiring adequate immunity. Here, data from previous infections suggest vaccination reduces adherence to other public health measures,45 with pre-prints suggesting that this is occurring after vaccination against SARS-CoV-2 also.46,47 Overall, CSS app users are not fully representative of the UK population (younger, more likely to be female, of higher educational status, and over-representative of healthcare workers)21. Although the population in the current study shares some of these biases, the median age of vaccinated individuals at the time of our analysis (64 years) was older than for app users overall (47 years), which is not surprising as the UK vaccination schedule began with the oldest individuals in the community. We considered the implications of this with respect to the likelihood of an infected person presenting for testing: although asymptomatic SARS-CoV-2 infection is well-recognised, it is less common in older people.25,26 We also acknowledge that different economic and cultural experiences may influence presentation of SARS-CoV-2 infection and reporting of post-vaccination side-effects.45 Our approach in comparing symptom profiles for individuals testing positive or negative for SARS-CoV-2 required a 1:1 matched population, so that comparison of symptom prevalence was fair and unbiased by the greatly different sample sizes of the two populations. However, this methodological choice is less reliable when used for the outcome of SARS-CoV-2 test prediction; and the forced balance of the classes can lead to an overestimation of the likelihood of being positive in the modelling. Thus we have also presented extended analyses, using both boot-strapping, and entire-cohort approaches in the Supplementary Results. We acknowledge that the predictive power of our optimised models may be hampered if there are brand-specific post-vaccination side-effects, which were not considered during model optimisation.16 However, although there were some differences in frequencies, most symptoms were reported in both PB and O-AZ pivotal trials.4,5,8,10 We also did not consider type of SARS-CoV-testing (PCR vs. LFAT), or mode of testing access (NHS vs. ZOE-request), which may also contribute variability to these models. Our analyses do not consider the impact of COVID-19 prevalence in UK at the time of the vaccination. Positive and negative predictive values (PPV, NPV) for a test depend not only on test sensitivity and specificity but also on population prevalence of disease. The rapidly changing prevalence of SARS-CoV-2 infection in the UK and the pace of vaccination delivery over the time period of this study limits our capacity to provide accurate PPV and NPV. Further analyses, particularly in populations with higher prevalence of infection and/or higher symptom burden and severity of COVID-19, may result in better differentiation of early signs of infection from post-vaccination side-effects. A strength of our study was our very large cohort of vaccinated participants, in a country that was an early adopter of vaccination; and our timeframe included the UK pandemic “third wave”. Prospective real-time symptom logging through the app minimised recall bias; and our symptom assessment included direct ascertainment of core symptoms for accessing UK testing. However, the sharp decline in cases in the first six months of 2021 resulted in only 150 positive cases to inform our modelling which number we acknowledge is small - though we were also able to draw upon large numbers of tested negative individuals for comparisons, reinforcing the consistency and generalisation of our results. We also acknowledge that the demographic features of the app population especially those parameters considered for model estimation (e.g., age, gender, BMI) may be different in other populations within the UK and elsewhere. The implication of our results will vary depending on the population prevalence of SARS-CoV-2 and pace of vaccination roll-out. At the time of writing Australia has negligible community spread of SARS-CoV-2 but is early in vaccination roll-out. It would be very interesting to repeat this study in these different circumstances. Testing of all symptomatic individuals comes at a cost (e.g., testing kits, infrastructure). Here, the UK is a resource-rich country; the impact of our results in countries with fewer health resources needs careful consideration. In conclusion, post-vaccination symptoms cannot be distinguished with clinical confidence from early SARS-CoV-2 infection. Our study highlights the critical importance of testing symptomatic individuals - even if recently vaccinated – to ensure early detection of SARS-CoV-2 infection and help prevent future waves of COVID-19. ## Supporting information Supplementary material [[supplements/260906_file03.pdf]](pending:yes) ## Data Availability Data collected in the COVID Symptom Study smartphone application are shared with other health researchers through the UK National Health Service-funded Health Data Research UK (HDRUK) and Secure Anonymised Information Linkage consortium, housed in the UK Secure Research Platform (Swansea, UK). Anonymised data are available to be shared with researchers according to their protocols in the public interest ([https://web.www.healthdatagateway.org/dataset/fddcb382-3051-4394-8436-b92295f14259](http://web.www.healthdatagateway.org/dataset/fddcb382-3051-4394-8436-b92295f14259)). The code is available in: [https://gitlab.com/KCL-BMEIS/covid-zoe/vaccination](http://gitlab.com/KCL-BMEIS/covid-zoe/vaccination). [https://web.www.healthdatagateway.org/dataset/fddcb382-3051-4394-8436-b92295f1425](https://web.www.healthdatagateway.org/dataset/fddcb382-3051-4394-8436-b92295f1425) ## Authors’ contribution LSC, MM and ELD contributed to study concept and design. CHS, JCP, BM, TS, CJS, SO contributed to acquisition of data. All the authors had access to the raw data underlying the study. LSC, JD, ELD contributed to data analysis and verified the underlying data. LSC, MO, MM and ELD contributed to drafting of the manuscript. All authors contributed to interpretation of data and critical revision of the manuscript. MM and ELD contributed to study supervision. ## Declaration of Interests ELD and CJS report grants from the Chronic Disease Research Foundation (CDRF) during the conduct of the study. CH, SS, LP, AM report other from ZOE Limited, during the conduct of the study. TS is a scientific advisor to ZOE Limited. CHS reports grants from Alzheimer’s Society, during the conduct of the study. SO reports grants from the Wellcome Trust, Innovate UK (UKRI), and Chronic Disease Research Foundation (CDRF), during the conduct of the study. ## Data sharing Data collected in the COVID Symptom Study smartphone application are shared with other health researchers through the UK National Health Service-funded Health Data Research UK (HDRUK) and Secure Anonymised Information Linkage consortium, housed in the UK Secure Research Platform (Swansea, UK). Anonymised data are available to be shared with researchers according to their protocols in the public interest ([https://web.www.healthdatagateway.org/dataset/fddcb382-3051-4394-8436-b92295f14259](https://web.www.healthdatagateway.org/dataset/fddcb382-3051-4394-8436-b92295f14259)). The code is available in: [https://gitlab.com/KCL-BMEIS/covid-zoe/vaccination](https://gitlab.com/KCL-BMEIS/covid-zoe/vaccination). ## Acknowledgements ZOE Limited provided in-kind support for all aspects of building, running and supporting the app and service to all users worldwide. This work is supported by the Wellcome EPSRC Centre for Medical Engineering at King’s College London (WT 203148/Z/16/Z) and the UK Department of Health via the National Institute for Health Research (NIHR) comprehensive Biomedical Research Centre award to Guy’s & St Thomas’ NHS Foundation Trust in partnership with King’s College London and King’s College Hospital NHS Foundation Trust. Investigators also received support from Medical Research Council (MRC), British Heart Foundation (BHF), Alzheimer’s Society, European Union, NIHR, COVID-19 Driver Relief Fund (CDRF) and the NIHR-funded BioResource, Clinical Research Facility and Biomedical Research Centre (BRC) based at GSTT NHS Foundation Trust in partnership with KCL. SO was supported by the French government, through the 3IA Côte d’Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002. This research was funded in part by the Wellcome Trust [215010/Z/18/Z]. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript (AAM) version arising from this submission. ## Abbreviations AUC : Area under the curve bMEM : Bayesian mixed-effect model BMI : Body mass index CI : confidence interval COVID-19 : Coronavirus disease 2019 CSS : COVID Symptom Study IQR : inter-quartile range KCL : King’s College London LFAT : Lateral flow antigen test LR : Logistic regression MD : Missing data NHS UK : National Health Service of the United Kingdom O-AZ : Oxford-AstraZeneca adenovirus-vectored vaccine PB : Pfizer-BioNTech mRNA vaccine RF : Random forest ROC : Receiver operating curve rtPCR : Reverse transcription polymerase chain reaction SARS-CoV-2 : severe acute respiratory syndrome-related coronavirus 2 UK : United Kingdom of Great Britain and Northern Ireland * Received July 21, 2021. * Revision received July 21, 2021. * Accepted July 22, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. Worldometer. COVID-19 Coronavirus Pandemic. [https://www.worldometers.info/coronavirus/](https://www.worldometers.info/coronavirus/) (accessed June 10, 2020). 2. world Health Organisation. WHO Coronavirus (COVID-19) Dashboard. 2021; published online May 25. 3. Medicines and Healthcare products Regulatory Agency. Public Assessment Report Authorisation for Temporary Supply COVID-19 mRNA Vaccine BNT162b2 (BNT162b2 RNA) concentrate for solution for injection. 2020 [https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/944544/COVID-19\_mRNA\_Vaccine\_BNT162b2UKPARPFIZER\_BIONTECH15Dec2020.pdf](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/944544/COVID-19\_mRNA_Vaccine_BNT162b2UKPARPFIZER_BIONTECH15Dec2020.pdf) (accessed May 26, 2021). 4. Polack FP, Thomas SJ, Kitchin N, et al. Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine. New England Journal of Medicine 2020; 383: 2603–15. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa2034577&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 5. Folegatti PM, Ewer KJ, Aley PK, et al. Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial. The Lancet 2020; 396: 467–78. 6. Ramasamy MN, Minassian AM, Ewer KJ, et al. Safety and immunogenicity of ChAdOx1 nCoV-19 vaccine administered in a prime-boost regimen in young and old adults (COV002): a single-blind, randomised, controlled, phase 2/3 trial. The Lancet 2020; 396: 1979–93. 7. Medicines and Healthcare products Regulatory Agency. Public Assessment Report Authorisation for Temporary Supply COVID-19 Vaccine AstraZeneca, solution for injection in multidose container COVID-19 Vaccine (ChAdOx1-S [recombinant]) Department of Health and Social Care (DHSC) AstraZeneca AB. 2020 [https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/963928/UKPAR\_COVID\_19\_Vaccine\_AstraZeneca\_23.02.2021.pdf](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/963928/UKPAR\_COVID\_19_Vaccine_AstraZeneca_23.02.2021.pdf) (accessed May 26, 2021). 8. Baden LR, el Sahly HM, Essink B, et al. Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine. New England Journal of Medicine 2021; 384: 403–16. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa2035389&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 9. Medicines and Healthcare products Regulatory Agency. Public Assessment Report National Procedure COVID-19 Vaccine Moderna, 0.20 mg/mL dispersion for injection (COVID-19 mRNA Vaccine [nucleoside modified]). 2020 [https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/977367/UKPAR\_COVID\_19\_Vaccine\_Moderna\_07.04.2021\_CMA\_Reliance\_PAR-final.pdf](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/977367/UKPAR\_COVID\_19\_Vaccine\_Moderna\_07.04.2021_CMA_Reliance_PAR-final.pdf) (accessed May 26, 2021). 10. Sadoff J, Gray G, Vandebosch A, et al. Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19. New England Journal of Medicine 2021; 384: 2187–201. 11. British Broadcasting Corporation. Covid-19 vaccine: First person receives Pfizer jab in UK. BBC News. 2020; published online Dec 8. 12. Department of Health and Social Care Press Release: First people to receive Oxford University. Press Release: First people to receive Oxford University/AstraZeneca COVID-19 vaccine today. DHSC. 2021; published online Jan 4. 13. Government of the United Kingdom. Coronavirus (COVID-19) in the UK - Cases by specimen date. GOV UK. 2021; published online Jan 29. [https://coronavirus.data.gov.uk/details/cases](https://coronavirus.data.gov.uk/details/cases) (accessed May 26, 2021). 14. Menni C, Sudre CH, Steves CJ, Ourselin S, Spector TD. Quantifying additional COVID-19 symptoms will save lives. Lancet (London, England) 2020; 395: e107–8. 15. Drew DA, Nguyen LH, Steves CJ, et al. Rapid implementation of mobile technology for real-time epidemiology of COVID-19. Science 2020; 368: 1362–7. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEzOiIzNjgvNjQ5Ny8xMzYyIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjEvMDcvMjIvMjAyMS4wNy4yMS4yMTI2MDkwNi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 16. Menni C, Klaser K, May A, et al. Vaccine side-effects and SARS-CoV-2 infection after vaccination in users of the COVID Symptom Study app in the UK: a prospective observational study. The Lancet Infectious Diseases 2021; published online May 20. DOI:10.1016/S1473-3099(21)00224-3. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1473-3099(21)00224-3&link_type=DOI) 17. Greinacher A, Thiele T, Warkentin TE, Weisser K, Kyrle PA, Eichinger S. Thrombotic Thrombocytopenia after ChAdOx1 nCov-19 Vaccination. New England Journal of Medicine 2021; published online April 9. DOI:10.1056/nejmoa2104840. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/nejmoa2104840&link_type=DOI) 18. Schultz NH, Sørvoll IH, Michelsen AE, et al. Thrombosis and Thrombocytopenia after ChAdOx1 nCoV-19 Vaccination. New England Journal of Medicine 2021; published online April 9. DOI:10.1056/nejmoa2104882. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/nejmoa2104882&link_type=DOI) 19. Medicines & Healthcare products Regulatory Agency. Coronavirus vaccine - weekly summary of Yellow Card reporting. London:MHRA. 2021; published online May 26. [https://www.gov.uk/government/publications/coronavirus-covid-19-vaccine-adverse-reactions/coronavirus-vaccine-summary-of-yellow-card-reporting](https://www.gov.uk/government/publications/coronavirus-covid-19-vaccine-adverse-reactions/coronavirus-vaccine-summary-of-yellow-card-reporting) (accessed June 9, 2021). 20. Struyf T, Deeks JJ, Dinnes J, et al. Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19 disease. The Cochrane database of systematic reviews 2020; 7: CD013665–CD013665. 21. Menni C, Valdes AM, Freidin MB, et al. Real-time tracking of self-reported symptoms to predict potential COVID-19. Nature Medicine 2020; 26: 1037–40. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41591-020-0916-2&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 22. Brooks Jackson J, Dempewolf S, Johnson AL, van Ert H, Maury W. Dynamics of Humoral Immunity Post COVID Vaccination – A Case Report. Clin Rev & Cases 2021; 3: 1–2. 23. Watson J, Richter A, Deeks J. Testing for SARS-CoV-2 antibodies. The BMJ 2020; 370. DOI:10.1136/bmj.m3325. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNzAvc2VwMDhfMi9tMzMyNSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIxLzA3LzIyLzIwMjEuMDcuMjEuMjEyNjA5MDYuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 24. NHS UK. Coronavirus (COVID-19). 2020; published online Nov 1. [https://www.nhs.uk/conditions/coronavirus-covid-19/](https://www.nhs.uk/conditions/coronavirus-covid-19/) (accessed Feb 2, 2021). 25. Rai B, Shukla A, Dwivedi LK. Estimates of serial interval for COVID-19: A systematic review and meta-analysis. Clinical epidemiology and global health 2021; 9: 157–61. 26. McAloon C, Collins Á, Hunt K, et al. Incubation period of COVID-19: a rapid systematic review and meta-analysis of observational research. BMJ open 2020; 10: e039652–e039652. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTAvOC9lMDM5NjUyIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjEvMDcvMjIvMjAyMS4wNy4yMS4yMTI2MDkwNi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 27. Sudre CH, Keshet A, Graham MS, et al. Anosmia and other SARS-CoV-2 positive test-associated symptoms, across three national, digital surveillance platforms as the COVID-19 pandemic and response unfolded: an observation study. medRxiv 2020; : 2020.12.15.20248096. 28. Alberto Fernández, Salvador GarcÍa, Mikel Galar, Ronaldo C. Prati, Bartosz Krawczyk, Francisco Herrera. Learning from Imbalanced Data Sets. Springer, Cham, 2018 DOI:[https://doi.org/10.1007/978-3-319-98074-4](https://doi.org/10.1007/978-3-319-98074-4). 29. Kulesa A, Krzywinski M, Blainey P, Altman N. Points of Significance: Sampling distributions and the bootstrap. Nature Methods 2015; 12: 477–8. 30. Austin PC, Small DS. The use of bootstrapping when using propensity-score matching without replacement: a simulation study. Statistics in medicine 2014; 33: 4306–19. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/sim.6276&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25087884&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 31. Christopher M. Bishop. Pattern Recognition and Machine Learning. New York, 2006 DOI:10.1198/tech.2007.s518. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1198/tech.2007.s518&link_type=DOI) 32. Antonelli M, Penfold RS, Merino J, et al. Post-vaccination SARS-CoV-2 infection: risk factors and illness profile in a prospective, observational community-based case-control study. medRxiv 2021; : 2021.05.24.21257738. 33. Graham MS, May A, Varsavsky T, et al. Knowledge barriers in the symptomatic-COVID-19 testing programme in the UK: an observational study. medRxiv 2021; : 2021.03.16.21253719. 34. Wells PM, Doores KJ, Couvreur S, et al. Estimates of the rate of infection and asymptomatic COVID-19 disease in a population sample from SE England. Journal of Infection 2020; 81: 931–6. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jinf.2020.10.011&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33068628&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 35. Public Health England. Green Book. In: COVID-19-SARS-CoV-2 Notifiable. 2021. [https://fingertips.phe.org.uk/static-reports/mortality-surveillance/excess-mortality-in-](https://fingertips.phe.org.uk/static-reports/mortality-surveillance/excess-mortality-in). 36. Harada S, Uno S, Ando T, et al. Control of a Nosocomial Outbreak of COVID-19 in a University Hospital. Open Forum Infectious Diseases 2020; 7: 1–9. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 37. Carter B, Collins JT, Barlow-Pay F, et al. Nosocomial COVID-19 infection: examining the risk of mortality. The COPE-Nosocomial Study (COVID in Older PEople). Journal of Hospital Infection 2020; 106: 376–84. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jhin.2020.07.013&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 38. Houlihan CF, Vora N, Byrne T, et al. Pandemic peak SARS-CoV-2 infection and seroconversion rates in London frontline health-care workers. The Lancet. 2020; 396: e6–7. 39. Cook TM, El-Boghdadly K, Brown J, Pickering AE. The safety of anaesthetists and intensivists during the first COVID-19 surge supports extension of use of airborne protection PPE to ward staff. Clinical Medicine, Journal of the Royal College of Physicians of London 2021; 21: E137–9. 40. Hall VJ, Foulkes S, Saei A, et al. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study. The Lancet 2021; 397: 1725–35. 41. Angel Y, Spitzer A, Henig O, et al. Association Between Vaccination With BNT162b2 and Incidence of Symptomatic and Asymptomatic SARS-CoV-2 Infections Among Health Care Workers. JAMA 2021; published online May 6. DOI:10.1001/jama.2021.7152. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2021.7152&link_type=DOI) 42. Haas EJ, Angulo FJ, McLaughlin JM, et al. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. The Lancet 2021; 397: 1819–29. 43. Bailly B, Guilpain L, Bouiller K, et al. BNT162b2 mRNA vaccination did not prevent an outbreak of SARS COV-2 variant 501Y.V2 in an elderly nursing home but reduced transmission and disease severity. Clinical Infectious Diseases 2021; published online May 16. DOI:10.1093/cid/ciab446. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciab446&link_type=DOI) 44. Zaqout A, Daghfal J, Alaqad I, et al. The initial impact of a national BNT162b2 mRNA COVID-19 vaccine rollout. International Journal of Infectious Diseases 2021; 108: 116–8. 45. R Angel, P Thoits. The impact of culture on the cognitive structure of illness. Cult Med Psychiatry 1987; 4: 465–94. 46. Reiber C, Shattuck EC, Fiore S, Alperin P, Davis V, Moore J. Change in Human Social Behavior in Response to a Common Vaccine. Annals of Epidemiology 2010; 20: 729–33. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.annepidem.2010.06.014&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20816312&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) 47. Brewer NT, Cuite CL, Herrington JE, Weinstein ND. Risk compensation and vaccination: Can getting vaccinated cause people to engage in risky behaviorsã Annals of Behavioral Medicine 2007; 34: 95–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/BF02879925&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17688401&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F22%2F2021.07.21.21260906.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000249663800011&link_type=ISI)