City-wide school-located influenza vaccination: a retrospective cohort study
============================================================================

* Jade Benjamin-Chung
* Benjamin F. Arnold
* Kunal Mishra
* Chris J. Kennedy
* Anna Nguyen
* Nolan N. Pokpongkiat
* Stephanie Djajadi
* Anmol Seth
* Nicola P. Klein
* Alan E. Hubbard
* Arthur Reingold
* John M. Colford, Jr.

## Abstract

**Background** We measured the effectiveness of a city-wide school-located influenza vaccination (SLIV) program implemented in over 102 elementary schools in Oakland, California.

**Methods** We conducted a retrospective cohort study among Kaiser Permanente Northern California (KPNC) members of all ages residing in either the intervention or a multivariate-matched comparison site from September 2011 - August 2017. Outcomes included medically attended acute respiratory illness (MAARI), influenza hospitalization, and Oseltamivir prescriptions. We estimated difference-in-differences (DIDs) in 2014-15, 2015-16, and 2016-17 using generalized linear models and adjusted for race, ethnicity, age, sex, health plan, and language.

**Results** Pre-intervention member characteristics were similar between sites. Among school-aged children, SLIV was associated with lower Oseltamivir prescriptions per 1,000 (DIDs: −3.5 (95% CI −5.5, −1.5) in 2015-16; −4.0 (95% CI −6.5, −1.6) in 2016-17) but not with other outcomes. SLIV was associated with lower MAARI per 1,000 in adults 65+ years (2014-15: −13.2, 95% CI −23.2, −3.2; 2015-16: −21.5, 95% CI −31.1, −11.9; 2016-17: −13.0, 95% CI −23.2, −2.9). There were few significant associations with other outcomes among adults.

**Conclusions** A city-wide SLIV intervention was associated with lower Oseltamivir prescriptions in school-aged children and lower MAARI among people over 65 years, suggesting possible indirect effects of SLIV among older adults.

Key words
*   influenza
*   influenza vaccination
*   school-located influenza vaccination

## Introduction

In the United States, seasonal influenza has been responsible for 140,000 – 590,000 hospitalizations and 12,000 – 61,000 deaths annually since 2010 [1]. Children are responsible for the majority of influenza transmission, and mathematical models suggest that vaccinating 50% – 70% of school-aged children for influenza can produce herd immunity [2].

School-located influenza vaccination (SLIV) interventions may increase vaccine coverage among schoolchildren and reduce influenza transmission community-wide [3]. SLIV is associated with higher influenza vaccination coverage [4–13] and lower medically attended acute respiratory illness (MAARI) [14], influenza-like illness [4] and laboratory-confirmed influenza [14,15] in schoolchildren. Some studies have reported indirect effects of SLIV among non-school aged individuals, while others found none [7,8,14,16–20]. With the exception of one study [4], prior SLIV evaluations have measured health outcomes using observational designs that did not rigorously account for systematic differences between intervention and comparison sites prior to intervention [5–8,13–20].

We previously reported the findings from a matched cohort study of a city-wide SLIV program called Shoo the Flu that was implemented in a diverse, primarily low-income population in Oakland, California [21]. The initial evaluation found higher vaccination coverage and lower influenza hospitalization among non-elementary school aged individuals in the intervention site.

Here, we report the results of a retrospective cohort study to measure associations between Shoo the Flu program on additional outcomes, including MAARI and Oseltamivir prescriptions, among Kaiser Permanente Northern California (KPNC) members residing within either the intervention or a matched comparison area. Using data from 2011 to 2017, we estimated associations with SLIV among school-aged individuals and assessed potential indirect effects in other age groups.

## Methods

### SLIV intervention

This study evaluated the Shoo the Flu intervention ([www.shootheflu.org](http://www.shootheflu.org)), which has delivered free influenza vaccinations to schools in Oakland, California since 2014. The intervention was delivered to children in all public and charter elementary schools in Oakland Unified School District (OUSD, the “intervention district”) and offered to all other charter and private pre-schools and elementary schools in Oakland. OUSD has a diverse population of over 26,000 elementary school students, and >70% are low-income. From 2014-2017, Shoo the Flu vaccinated 7,502 – 10,106 students annually (22 – 28% of eligible students) in 102-138 schools. Each influenza season, 23-24% of intervention participants reported KPNC health plan membership. Additional intervention details are in the Supplement 1.

### Study setting and population

KPNC is an integrated healthcare system that delivers care at 46 medical clinics and 21 hospitals operated by KPNC to approximately 4 million members. Members comprise at least 30% of the population and are representative of the race, ethnicity, and socioeconomic distribution of Northern California, although very low-income individuals are under-represented. Health care visits, diagnoses, prescriptions, immunizations, and laboratory results are captured in KPNC’s electronic medical record. Vaccines are offered free of charge to members, and the date, injection site, and vaccine brand, lot, and dose of each vaccination at KPNC are recorded. Whether to test patients for influenza A or B using polymerase chain reaction is at the discretion of KPNC clinicians.

All KPNC members who resided in the catchment areas of the intervention and comparison districts from September 1, 2011 - August 31, 2017 and had no more than a one-month gap in KPNC membership for each influenza season during the study period were included in this study.

### Vaccines

In 2014-15 and 2015-16, the intervention provided the live attenuated influenza vaccine (LAIV) to students [22,23]. Students with LAIV contraindications were offered the trivalent inactivated injectable influenza vaccine (IIV3), as were staff and teachers. Because LAIV effectiveness in children was low in 2014-15 and 2015-16 [24–26], the intervention offered IIV4 to all participants following the Advisory Committee on Immunization Practices’ recommendation to use IIV for all children in 2016-17 [25].

### Study design

We conducted a retrospective cohort study of KPNC health plan members who lived in the catchment areas of the intervention district and a comparison district (West Contra Costa Unified School District [WCCUSD]). We identified the comparison school district using a genetic multivariate matching algorithm [27] to pair-match public elementary schools in the intervention district and each candidate comparison district using pre-intervention school-level characteristics (additional details in Supplement 2); detailed methods are available elsewhere [21]. We selected WCCUSD as the comparison site because it had the smallest average generalized Mahalanobis distance between paired schools [27].

### Program data

KPNC electronic medical records did not include records for vaccinations administered at locations other than KPNC. We therefore estimated the number of vaccinations delivered by the program to KPNC members using data from the Shoo the Flu program, which tracks vaccination counts using the number of vaccination consent forms collected from the parents each year. Consent forms included information about insurance provider (e.g., KPNC or other provider), allowing us to estimate the number of children vaccinated by Shoo the Flu who were KPNC members.

### Population and school district data

We obtained demographic information about the general population in study sites from the U.S. 2010 Census using zip codes that overlapped with the intervention and comparison school districts. We also obtained data about school district populations from the California Department of Education for these zip codes for 2013.

### Outcomes

Outcomes included medically attended acute respiratory illness (MAARI) (see Supplement Table 1 for ICD-9 and ICD-10 codes), laboratory-confirmed influenza among tested individuals, influenza hospitalization, and filled Oseltamivir prescriptions. Individuals hospitalized with any of the following ICD-9-CM codes for otitis media and sinusitis (381–383, 461x), upper respiratory tract illness (79x, 460, 462–463, 465, 487.1), and lower respiratory tract illness (464x, 466x, 480x–487.0, 490x–496x, 510x–513x, 515x–516x, 518x, and 786.1) were classified as having an influenza hospitalization. We defined laboratory-confirmed influenza as a positive result from RT-PCR influenza diagnostic test.

### Definition of influenza season

We defined influenza season based on the percentage of medical visits for influenza-like illness in California using data from the California Department of Public Health. Influenza season started after two consecutive weeks in which the percentage of medical visits for influenza-like illness was greater than or equal to 2%; it ended after two consecutive weeks with a percentage under 2%.

### Statistical analysis

This study’s pre-analysis plan and replication scripts are available at [https://osf.io/rtsf2/](https://osf.io/rtsf2/).

We defined the cumulative incidence as the proportion of individuals with at least one outcome event in each season. We restricted analyses to influenza season, when the intervention would be expected to affect influenza-related outcomes. Our primary pre-specified parameter was mean difference-in-differences (DIDs) that compared the difference in cumulative incidence during influenza season to that in three seasons prior to the intervention (2011-2014) in each study group (the “pre-intervention DID”). Pre-intervention monthly incidences of each outcome were consistent with the equal trends assumption (Supplement Figure 2). DIDs remove time-invariant confounding but may be subject to time-dependent confounding [28]. To account for this, we conducted a post-hoc alternative “pre-season DID” analysis that compared the incidence in each influenza season to that in the period immediately preceding each season (May – September). While the pre-season DID does not account for pre-intervention differences, it is less subject to time-dependent confounding than the pre-intervention DID.

Models adjusted for available potential confounders with at least 5% prevalence in each analysis; these included race, ethnicity, sex, mediCAL, subsidized KPNC health plan (proxy for low socio-economic status), and primary language spoken. Enrollment in mediCAL or a subsidized KPNC health plan varied by season; other variables were static. To minimize empirical positivity violations from sparse data [29], we fit models only if the number of outcome events per variable was ≥10 and only fit adjusted models if the number of observations within age, site, and outcome strata was ≥30 [30]. We estimated 95% confidence intervals using robust standard errors [31] to account for clustering at the household level. Additional minor deviations from the plan are noted in Supplement 3, and additional analysis details are in Supplement 4.

We stratified estimates by pre-specified age groups (0-4, 5-12, 13-17, 18-64, and 65+ years). Estimates among children 5-12 years represent “total effects” (including intervention participants and non-participants) and estimates in other age groups represent “indirect effects” among non-participants (Supplement Figure 1) [32]. Per our pre-analysis plan, we also stratified estimates by individual vaccination status.

We performed a sensitivity analysis using alternative influenza seasons with influenza-like-illness thresholds of 2.5% and 3% and the CDC influenza season definition (Week 40 to Week 20 of the following year).

We pre-specified two negative control analyses to detect residual confounding or selection bias [33,34]. First, we repeated our primary analysis with outcomes we did not expect SLIV to affect (medically attended diarrhea and medically attended gastrointestinal illness) [33,34]. We conducted a negative control time period analysis restricting to weeks outside influenza season.

### Ethical statement

This study was approved by the Committee for the Protection of Human Subjects at the University of California, Berkeley (Protocol # 2017-03-9741) and the KPNC Institutional Review Board (Protocol #CN-16-2825).

## Results

### Pre-intervention characteristics

The analyses included 175,628 to 269,266 individuals and 9,436,202 to 11,500,570 person-weeks of observations per calendar year from 2011 to 2017 (Supplement Table 2). During the influenza season, the number of person-weeks ranged from 3,069,633 to 6,801,780 per year.

Pre-intervention characteristics were similar among individuals in the intervention and comparison sites (Table 1). In the intervention site vs. the comparison site, there were fewer Asian (14.2% vs. 19.3%) and Hispanic (30.5% vs. 43.0%) individuals and more Black / African American (28.2% vs. 20.2%) and White individuals (35.2% vs. 29.5%). The proportion of individuals enrolled in MediCAL was lower in the intervention site vs. the comparison site (4.4% vs. 6.7%).

View this table:
[Table 1.](http://medrxiv.org/content/early/2021/05/04/2021.05.03.21256546/T1)

Table 1. Pre-intervention characteristics of the study population in each site

Characteristics were similar overall between the study population, general population (all individuals residing in zip codes overlapping with the study site) and school district population (Supplement Table 3). The percentage of individuals in the study population who were Hispanic was lower than in the general population (31-43% vs. 39-54%). The percentage of the study population whose primary language spoken was not English was also lower (12-14%) than in the general population (40-47%) and school district population (38-43%).

### Influenza vaccination

Among the study population, influenza vaccination coverage was lower in the intervention site than the comparison site across age groups and years. Prior to the intervention, vaccination coverage was also consistently lower in the intervention site than the comparison site (Supplement Table 4); this difference was larger during the intervention period for children 5-12 years (Supplement Figure 3). Including all vaccinations (administered by both KPNC and the SLIV intervention), coverage was 8-11% higher in the intervention site than the comparison site during the intervention period (Figure 1). Across all years, KPNC mostly administered IIV to elementary school aged children in the study (Supplement Figure 4).

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/05/04/2021.05.03.21256546/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2021/05/04/2021.05.03.21256546/F1)

Figure 1. 
**Percentage of elementary school aged study population vaccinated for influenza by Kaiser Permanente Northern California or the Shoo the Flu intervention**

“Vaccination by KPNC – Comparison” includes influenza vaccinations delivered by KPNC in the comparison site. “Vaccination by KPNC – Intervention” includes influenza vaccinations delivered by KPNC in the intervention site. “Vaccination by Shoo the Flu” includes influenza vaccinations delivered by the Shoo the Flu intervention in the intervention site. The percentage vaccinated for influenza does not include vaccinations that were not delivered by KPNC or Shoo the Flu. The denominator is KPNC patients aged 5-12 years.

### Cumulative incidence of influenza-related outcomes

The cumulative incidence of MAARI ranged from 0.14 to 0.58 per season by age and was highest among children 0-4 years (Supplement Figure 5). Very few individuals were tested for influenza (<1% of individuals per season), and the proportion who tested positive varied substantially by age, season, and site. The incidence of influenza hospitalization was low in all age groups, with the highest incidence among adults ≥65 years (range: 0.053 to 0.097 per season). The incidence of filled Oseltamivir prescriptions ranged from 0.0007 to 0.0134 per season.

### Associations among elementary school aged children

Overall, the intervention was not associated with MAARI in elementary school aged children except for in 2015-16, when it was associated with lower MAARI when accounting for pre-season differences between sites (Figure 2, Supplement Figure 6, Supplement Table 5). Among elementary school aged children, in 2016-17 the unadjusted pre-intervention DID in the cumulative incidence of filled Oseltamivir prescriptions per 1,000 was −3.5 (95% CI −5.5, −1.5) in 2015-16 and −4.0 (95% CI −6.5, −1.6) in 2016-17; there was no association in 2014-15. These associations were attenuated towards the null in the analysis accounting for pre-season differences. There was no association with influenza hospitalization elementary school aged children.

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/05/04/2021.05.03.21256546/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2021/05/04/2021.05.03.21256546/F2)

Figure 2. Difference-in-differences accounting for pre-intervention differences
Difference-in-difference in cumulative incidence of each outcome during each influenza season comparing the difference in mean outcome in each district in an intervention year compared to the three pre-intervention years (2011-2013). Difference-in-difference parameters remove any time-invariant differences between groups (measured or unmeasured). Parameters were estimated using a generalized linear model without covariate adjustment due to data sparsity. Standard errors accounted for clustering at the household level. Estimates in children 5-12 years measure total effects and estimates in other age groups measure indirect effects. Analyses were restricted to influenza season defined based the percentage of medical visits for influenza-like illness in California as reported by the California Department of Public Health. Influenza season started when there were at least 2 consecutive weeks in which the percentage of medical visits for influenza-like illness exceeded 2%, and the season ended when there were at least two consecutive weeks in which the percentage was less than or equal to 2%.

### Associations among non-elementary school aged individuals

DIDs indicated no association between the intervention and laboratory-confirmed influenza in most program years and age groups among non-elementary school aged individuals (Supplement Table 5). In intervention vs. comparison sites, MAARI per 1,000 was lower among adults 65 years or older when accounting for pre-intervention differences (2014-15: −13.2, 95% CI −23.2, −3.2; 2015-16: −21.5, 95% CI −31.1, −11.9; 2016-17: −13.0, 95% CI −23.2, −2.9) (Figure 2). When accounting for pre-season differences, DIDs indicated slightly stronger protective associations with MAARI for most age groups and seasons (Supplement Figure 6). DIDs for filled Oseltamivir prescriptions per 1,000 were lower among adults 18-64 years in 2016-17 (−1.6, 95% CI −2.5, −0.7); results were similar when accounting for pre-season differences. DIDs indicated a higher incidence of influenza hospitalization among adults 18 years or older in 2014-15 and adults 18-64 years in 2015-16. Pre-season DIDs displayed a different pattern for adults: associations were null for adults 18 years or older except for adults 65 years or older in 2015-16, when the association was protective.

For all ages, the results of adjusted models were similar to those of unadjusted models (Supplement Table 6). Results stratified by vaccination status were similar overall to the primary analysis (Supplement Tables 7-8).

### Negative control analyses

Analyses using non-influenza outcomes (medically attended diarrhea and medically attended gastrointestinal illness) produced primarily null associations (Supplement Table 9). Analyses restricted to weeks outside of influenza season produced almost exclusively null results (Supplement Table 10).

### Sensitivity analyses using alternative influenza season definitions

Results were similar when using alternative influenza season definitions (Supplement Figure 7). For outcomes with sufficient data to restrict to the peak week of influenza, results suggested no association with the intervention.

## Discussion

In this three-year evaluation of a city-wide SLIV intervention, among elementary school aged children, SLIV was associated with lower Oseltamivir prescriptions but not with MAARI, laboratory-confirmed influenza, or influenza hospitalization. We found some evidence of indirect effects in non-elementary school aged individuals: SLIV was associated with lower MAARI among adults 65 years or older and lower Oseltamivir prescriptions among pre-school children in 2015-16.

Some prior observational studies of SLIV programs have also reported indirect effects of SLIV on MAARI and related outcomes [14,18–20], while others have reported non-significant or null results [7,8,13,16,17]. These studies did not use analytic methods to account for differences between intervention and comparison sites prior to intervention or outside of influenza season. The one prior study that did so found no association between SLIV and hospitalization among the elderly [16]. The present study included three years of pre-intervention data and used a DID approach to adjust for pre-intervention differences; thus, our findings likely have higher internal validity than prior observational SLIV evaluations.

Vaccine effectiveness varied substantially during the study period. In 2014-15 all vaccine formulations had low effectiveness due to a poor strain match [24]. In 2015-16, the LAIV had poor vaccine effectiveness, but the IIV was moderately effective [26]. In 2016-17, only IIV was available and it was moderately effective [35]. These differences and varying vaccine coverage by the SLIV intervention each year [21] contribute to heterogeneous estimates across seasons.

SLIV may increase vaccination coverage among children who would not otherwise be vaccinated and/or shift vaccination location from doctor’s offices to schools. If SLIV merely shifts vaccination location, it would not be expected to reduce influenza. In this study, SLIV appeared to both increase vaccination and shift vaccination location among elementary school aged children. The proportion of 5-12 year-olds vaccinated by their medical provider was lower in the intervention site than the comparison site, but the proportion vaccinated by KPNC or SLIV combined was higher. This finding is consistent with our prior evaluation of Shoo the Flu [21] and evaluations of other SLIV interventions that also increased vaccination coverage [4–13].

Our finding that SLIV was associated with higher influenza hospitalization in adults in some years was unexpected. This finding contrasts with the prior Shoo the Flu evaluation, which found that SLIV was associated with lower influenza hospitalization in non-elementary age groups [21]. Other prior SLIV evaluations have had conflicting findings with regard to indirect effects on hospitalization [7,8,13,16,17]. One cluster-randomized trial reported a higher hospitalization rate in SLIV participants, but when stratifying by child vaccination status, the finding was not significant [4]. In a post-hoc analysis accounting for pre-season differences between sites, which may better account for time-dependent confounding than our primary analysis, associations with hospitalization were null or protective.

This study is subject to several limitations. First, its observational design may be subject to unmeasured confounding. In a small number of negative control outcome and time period analyses, unexpected associations suggest that unmeasured confounding occurred. The DID analysis controlled for time-invariant confounding [28], but unmeasured time-varying confounding may have occurred. Common confounders of influenza vaccine studies are age, calendar time, and health status [36]. Our analysis stratified by age and season, but baseline health status was not available. Second, differences in sociodemographic characteristics between the study population and the general and student populations in the study sites limit the generalizability of our findings. Third, many outcomes were rare, precluding formal analyses for some outcomes and limiting statistical power, which may have contributed to null findings. Finally, we did not have complete vaccination information about each individual. It was not possible to link data from the California Immunization Registry with KPNC member data. This limitation underscores the need for more robust vaccine registries [37]. Notably, the limitations of this study apply to most prior SLIV evaluations, which primarily have been observational and leveraged existing data. We expect that among individuals not targeted by the SLIV intervention, influenza vaccinations outside KPNC were infrequent, so this limitation is unlikely to have meaningfully impacted study findings.

Overall, our findings bolster those of our prior evaluation of a city-wide SLIV intervention. This study supports our prior finding that SLIV both increased vaccination coverage and shifted vaccination location and provides additional evidence that SLIV was associated with lower Oseltamivir prescriptions in school aged children and lower MAARI in older adults.

## Supporting information

Supplement [[supplements/256546_file02.docx]](pending:yes)

## Data Availability

The data from this study is not available due to human subjects protections.

## Funding

This work was supported by a grant from the Flu Lab ([https://theflulab.org/](https://theflulab.org/)) to the University of California, Berkeley (Grant number: 20142281; PI: AR).

## Acknowledgements

The authors would like to acknowledge Kristin Goddard, Karen Nunley, and Berwick Chan at Kaiser Permanente Northern California and Kate Holbrook at Alameda County Public Health Department for their contributions to data collection.

NPK reports research support from Protein Science (Sanofi Pasteur) and Sanofi Pasteur for unrelated influenza vaccine studies and from GlaxoSmithKline, Merck & Co, and Pfizer for unrelated studies. JBC, BFA, KM, CJK, AN, NNP, SD, AS, AEH, AR, JMC report no conflicts.

*   Received May 3, 2021.
*   Revision received May 3, 2021.
*   Accepted May 4, 2021.


*   © 2021, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/)

## References

1.  1.Centers for Disease Control and Prevention. Burden of Influenza. 2020. Available at: [https://www.cdc.gov/flu/about/burden/index.html](https://www.cdc.gov/flu/about/burden/index.html). Accessed 27 July 2020.
    
    

2.  2.Longini IM. A Theoretic Framework to Consider the Effect of Immunizing Schoolchildren Against Influenza: Implications for Research. Pediatrics 2012; 129:S63–S67.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MjA6IjEyOS9TdXBwbGVtZW50XzIvUzYzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjEvMDUvMDQvMjAyMS4wNS4wMy4yMTI1NjU0Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

3.  3.Gaglani MJ. Editorial Commentary: School-Located Influenza Vaccination: Why Worth the Effort? Clin Infect Dis 2014; 59:333–335.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciu344&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24829211&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

4.  4.King JC, Stoddard JJ, Gaglani MJ, et al. Effectiveness of School-Based Influenza Vaccination. N Engl J Med 2006; 355:2523–2532.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa055414&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17167135&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000242778800004&link_type=ISI) 

5.  5.Davis MM, King JC, Moag L, Cummings G, Magder LS. Countywide School-Based Influenza Immunization: Direct and Indirect Impact on Student Absenteeism. Pediatrics 2008; 122:e260–e265.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjEyMi8xL2UyNjAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wNS8wNC8yMDIxLjA1LjAzLjIxMjU2NTQ2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

6.  6.Kjos SA, Irving SA, Meece JK, Belongia EA. Elementary School-Based Influenza Vaccination: Evaluating Impact on Respiratory Illness Absenteeism and Laboratory-Confirmed Influenza. PLoS ONE 2013; 8:e72243.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0072243&link_type=DOI) 

7.  7.Pebody RG, Green HK, Andrews N, et al. Uptake and impact of vaccinating school age children against influenza during a season with circulation of drifted influenza A and B strains, England, 2014/15. Eurosurveillance 2015; 20.
    
    

8.  8.Pebody RG, Green HK, Andrews N, et al. Uptake and impact of a new live attenuated influenza vaccine programme in England: early results of a pilot in primary school-age children, 2013/14 influenza season. Eurosurveillance 2014; 19:20823.
    
    

9.  9.Szilagyi PG, Schaffer S, Rand CM, et al. School-Located Influenza Vaccinations: A Randomized Trial. Pediatrics 2016;:e20161746.
    
    

10. 10.Szilagyi PG, Schaffer S, Rand CM, et al. Impact of elementary school-located influenza vaccinations: A stepped wedge trial across a community. Vaccine 2018; 36:2861–2869.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.vaccine.2018.03.047&link_type=DOI) 

11. 11.Szilagyi PG, Schaffer S, Rand CM, et al. School-Located Influenza Vaccination: Do Vaccine Clinics at School Raise Vaccination Rates? J Sch Health 2019; 89:1004–1012.
    
    

12. 12.Humiston SG, Schaffer SJ, Szilagyi PG, et al. Seasonal Influenza Vaccination at School: A Randomized Controlled Trial. Am J Prev Med 2014; 46:1–9.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.amepre.2013.08.021&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24355665&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

13. 13.Pebody RG, Sinnathamby MA, Warburton F, et al. Uptake and impact of vaccinating primary school-age children against influenza: experiences of a live attenuated influenza vaccine programme, England, 2015/16. Eurosurveillance 2018; 23:1700496.
    
    

14. 14.Glezen WP, Gaglani MJ, Kozinetz CA, Piedra PA. Direct and Indirect Effectiveness of Influenza Vaccination Delivered to Children at School Preceding an Epidemic Caused by 3 New Influenza Virus Variants. J Infect Dis 2010; 202:1626–1633.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1086/657089&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21028955&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000283799900003&link_type=ISI) 

15. 15.Pannaraj PS, Wang H-L, Rivas H, et al. School-Located Influenza Vaccination Decreases Laboratory-Confirmed Influenza and Improves School Attendance. Clin Infect Dis 2014; 59:325–332.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciu340&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24829215&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

16. 16.McBean M, Hull HF, O’Connor H. Possible Herd Immunity in the Elderly Following the Vaccination of School Children with Live, Attenuated Trivalent Influenza Vaccine: A Person-Level Analysis. Procedia Vaccinol 2011; 4:59–70.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.provac.2011.07.009&link_type=DOI) 

17. 17.Hull HF, McBean AM, Caldwell D, O’Connor H. Assessing Herd Immunity in the Elderly Following the Vaccination of School Children with Live Attenuated Trivalent Influenza Vaccine (LAIV): A County-Level Analysis. Procedia Vaccinol 2010; 2:90–98.
    
    

18. 18.Tran CH, Sugimoto JD, Pulliam JRC, et al. School-Located Influenza Vaccination Reduces Community Risk for Influenza and Influenza-Like Illness Emergency Care Visits. PLoS ONE 2014; 9:e114479.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0114479&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25489850&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

19. 19.Piedra PA, Gaglani MJ, Kozinetz CA, et al. Herd immunity in adults against influenza-related illnesses with use of the trivalent-live attenuated influenza vaccine (CAIV-T) in children. Vaccine 2005; 23:1540–1548.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.vaccine.2004.09.025&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15694506&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000227272300002&link_type=ISI) 

20. 20.Piedra PA, Gaglani MJ, Kozinetz CA, et al. Trivalent Live Attenuated Intranasal Influenza Vaccine Administered During the 2003–2004 Influenza Type A (H3N2) Outbreak Provided Immediate, Direct, and Indirect Protection in Children. Pediatrics 2007; 120:e553–e564.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjEyMC8zL2U1NTMiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wNS8wNC8yMDIxLjA1LjAzLjIxMjU2NTQ2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

21. 21.Benjamin-Chung J, Arnold BF, Kennedy CJ, et al. Evaluation of a city-wide school-located influenza vaccination program in Oakland, California with respect to vaccination coverage, school absences, and laboratory-confirmed influenza: a matched cohort study. PLOS Med 2020; 17:e1003238.
    
    

22. 22.Grohskopf LA, Olsen SJ, Sokolow LZ, et al. Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices (ACIP) ╌United States, 2014-15 influenza season. Morb Mortal Wkly Rep 2014; 63:691–697.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/ajt.13065&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25121712&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

23. 23.Grohskopf LA, Sokolow LZ, Olsen SJ, Bresee JS, Broder KR, Karron RA. Prevention and Control of Influenza with Vaccines: Recommendations of the Advisory Committee on Immunization Practices, United States, 2015-16 Influenza Season. Morb Mortal Wkly Rep 2015; 64:818–825.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.15585/mmwr.mm6430a3&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26247435&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

24. 24.Zimmerman RK, Nowalk MP, Chung J, et al. 2014–2015 Influenza Vaccine Effectiveness in the United States by Vaccine Type. Clin Infect Dis 2016; 63:1564–1573.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciw635&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27702768&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

25. 25.Grohskopf LA, Sokolow LZ, Broder KR, et al. Prevention and Control of Seasonal Influenza with Vaccines: Recommendations of the Advisory Committee on Immunization Practices — United States, 2016–17 Influenza Season. Morb Mortal Wkly Rep 2016; 65:1–54.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26766396&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

26. 26.Jackson ML, Chung JR, Jackson LA, et al. Influenza Vaccine Effectiveness in the United States during the 2015–2016 Season. N Engl J Med 2017; 377:534–543.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa1700153&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28792867&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

27. 27.Diamond A, Sekhon JS. Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies. Rev Econ Stat 2013; 95:932–945.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access\_num=10.1162/REST_a_00318&link_type=DOI) 

28. 28.Wing C, Simon K, Bello-Gomez RA. Designing Difference in Difference Studies: Best Practices for Public Health Policy Research. Annu Rev Public Health 2018; 39:453–469.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1146/annurev-publhealth-040617-013507&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29328877&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

29. 29.Petersen ML, Porter KE, Gruber S, Wang Y, van der Laan MJ. Diagnosing and responding to violations in the positivity assumption. Stat Methods Med Res 2010; 21:31–54.
    
    

30. 30.Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49:1373–1379.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0895-4356(96)00236-3&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=8970487&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1996VY92400008&link_type=ISI) 

31. 31.Freedman DA. On The So-Called “Huber Sandwich Estimator” and “Robust Standard Errors”. Am Stat 2006; 60:299–302.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1198/000313006X152207&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000241706900001&link_type=ISI) 

32. 32.Benjamin-Chung J, Arnold BF, Berger D, et al. Spillover effects in epidemiology: parameters, study designs and methodological considerations. Int J Epidemiol 2018; 47:332–347.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/ije/dyx201&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29106568&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F05%2F04%2F2021.05.03.21256546.atom) 

33. 33.Lipsitch M, Tchetgen ET, Cohen T. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiol Camb Mass 2010; 21:383–388.
    
    

34. 34.Arnold BF, Ercumen A, Benjamin-Chung J, Colford JM. Negative Controls to Detect Selection Bias and Measurement Bias in Epidemiologic Studies: Epidemiology 2016; 27:637–641.
    
    

35. 35.Centers for Disease Control. Seasonal Influenza Vaccine Effectiveness, 2016-2017. 2019. Available at: [https://www.cdc.gov/flu/vaccines-work/2016-2017.html](https://www.cdc.gov/flu/vaccines-work/2016-2017.html). Accessed 14 June 2019.
    
    

36. 36.Ainslie KEC, Haber M, Orenstein WA. Challenges in estimating influenza vaccine effectiveness. Expert Rev Vaccines 2019; 18:615–628.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/14760584.2019.1622419&link_type=DOI) 

37. 37.Benjamin-Chung J, Reingold A. Measuring the Success of the US COVID-19 Vaccine Campaign—It’s Time to Invest in and Strengthen Immunization Information Systems. Am J Public Health 2021;:e1–e3.