How can the public health impact of vaccination be estimated?

Impact by calendar year (cross-sectional impact)

The calendar year method calculates the impact accrued over all ages for a specific year. It is generally the most intuitive and frequently used method to calculate the impact of vaccination. In this case, the vaccine impact, D_b−f (a, c, y), for age a, country c in year y is defined as the difference in disease burden between baseline and focal scenarios for a given year, D_{b− f} (a, c, y) = B_b(a, c, y) −B_f (a, c, y). Here, the baseline scenario can have no vaccination or different coverage to the focal scenario. Aggregating the impact over all ages modelled (A_m) means the impact for a country c in year y is given by .This does not account for the future disease burden averted through current vaccine activities.

Impact by birth year (lifetime impact)

The birth year method accounts for the long-term impact accrued over the lifetime of a particular birth cohort k. For country c, this is given by D(c, k) = ∑_y−a=k D_{b− f} (a, c, y), where y ∈Y_m and a ∈A_m. The duration of modelling needs to be appropriate to the pathogen of interest as in some cases, such as HepB, disease occurs later in life. For example, if we model vaccination for birth cohorts born from 2000 to 2030 and model disease burden until 2100, we do not account for the vaccine impact for those born in 2030 once they are over 70 years old. The method also does not specifically account for the impact of vaccinating a cohort outside the cohort vaccinated (e.g. because of herd protection).

Impact by year of vaccination

The YoV methods are vital for determining the long-term impact of vaccination due to activities carried out in a particular year. We obtain the impact ratio, ρ, as the impact attributable per fully vaccinated person (FVP) calculated as the coverage × cohort size. This ratio can be stratified by different characteristics, such as birth cohort in order to catch temporal changes in transmission or healthcare or by activity type to capture the differing effects of routine and campaign vaccination (RV and CV, respectively); see Table 2 for details. The impact ratio allows effects due to a particular year’s worth of vaccination to be attributed to that year. Additionally, the impact ratio can then be projected with the IE method by updating the impact with a new estimate of FVPs; see impact extrapolation section for further details.

Impact by year of vaccination: unstratified impact ratio

The simplest approach to calculate the impact by YoV is with an unstratified impact ratio which assumes that the effect of a vaccine is invariant over time and activity type. As such, the impact ratio, ρ, for a country, c, is the attributable vaccine impact divided by the number of FVPs: where Y_v − A_v refers to cohorts vaccinated in Y_v through RV.

Thus, the impact attributed to a year of vaccination is given by: This method does not take into account any changes in treatment or transmission over time, nor the differing effects of vaccination through RV or CV.

Impact by year of vaccination: impact ratio stratified by activity type

To calculate the impact by YoV using impact ratios stratified by activity type, we assume that RV and CV, which target multiple age groups, have different effects; for example due to dosage clustering. Hence, this method produces multiple, activity-specific impact ratios which can then be multiplied by the number of FVPs to calculate impact.

For RV, the impact ratio is defined as the impact for all cohorts who are vaccinated over time period Y_v per the additional FVPs between the baseline and focal scenarios. The impact for RI, D_R, is given by , where b_R and f_R are the baseline and focal RI scenarios, respectively. Here, the impact ratio for RV is: where Y_v − A_v are cohorts receiving vaccinations in years Y_v. The impact by year of vaccination is then: where FVP_R are FVPs vaccinated through RV. This does not account for the impact generated by indirect effects from and to cohorts not eligible for routine vaccination in Y_v, nor does it account for interactions between RV and CV that may occur during or before years Y − v.

For CV, the impact ratio is averaged evenly over all ages across the entire time period (Y_m). This is because we do not attempt to predict future CV coverage after the final year of credible campaign schedules. Therefore, the only impact due to CV comes from CV years Y_v and all campaign impact for birth cohorts born after this period can be attributed back to these vaccination years. The impact of CV, D_C, is given by , where b_C and f_C are the baseline and focal CV scenarios, respectively. The impact ratio is given by the following: The impact by year of vaccination is then given by the following: where FVP_C are FVPs vaccinated through CV.

Again, this does not account for interactions with the level of routine coverage across or before the same years.

The aggregated impact by YoV for both activities is the sum of the impact from RV and CV, i.e. sum of equations 4 and 6.

Impact by year of vaccination: impact ratio stratified by birth cohort

In this method the impact ratio is invariant to vaccination activity type. However, vaccine effect is assumed to vary over time through birth cohorts. This means that rather than averaging the effect of vaccination over time, we account for the variation in transmission and health of the population. This influences how one year’s vaccination may work compared to another. For example, if therapeutic treatments for a disease improve over time, we may expect the impact of vaccination in 2050 to be less than that now as the population is generally healthier. In this case, the impact ratio is cohort specific, given by the following equations: where k is the cohort defined by k = y − a and a ∈ A_m. In order to find the impact attributed to one year of vaccination, all cohort specific impacts for those cohorts vaccinated in the year of interest must be aggregated. The impact by YoV is then given as follows:

Impact by year of vaccination: impact ratio stratified by both activity type and birth cohort

In the above methods, we illustrate the potential ways the impact ratio could change between birth cohorts or by vaccination activity. However, it is likely that in reality the impact varies with respect to both of these aspects. In the following method, we use the approach of the activity stratification over birth cohort to arrive at an impact ratio per birth cohort and activity. The impact ratios by activity (R for RV; C for CV) are thus the following: Therefore the impact by YoV is provided by the sum of the activity and cohort-specific components: Note that in the above method the numerator may also be affected by vaccination activities experienced by cohorts around them, so not captured in the denominator. Hence the impact ratio could potentially be infinite.

Impact extrapolation

As coverage estimates are frequently updated within the WHO/UNICEF Estimates of National Immunization Coverage (WUENIC) and Gavi’s Operational Forecast (OP)^11–13, one of the vital VIMC tasks is to provide updated impact estimates to inform future RV and CV. Translating these coverage estimates into tangible estimates of vaccine impact in terms of mortality and morbidity averted is crucial. However, given the regular updates of vaccination coverage and due to the effort and time required for full model updates across the VIMC, a simplified process, the IE, has been developed to update past rounds of impact estimates with new coverage data.

The IE is primarily a linear interpolation of current vaccine impact estimates with new coverage estimates assuming constant country- and delivery strategy specific rates of mortality and morbidity averted per dose of vaccine. It can be applied to the YoV methods as these provide us with an impact ratio (ρ in Table 2). Hence, when coverage is updated, the updated number of FVPs can be calculated by multiplying the new coverage with the target population. The updated FVPs (FVP^∗) are then multiplied by the previously calculated ρ to calculate the updated impact (D^∗):

Models and vaccination scenarios

As the burden in each vaccination scenario is calculated using mathematical models of pathogen transmission dynamics and implementation of vaccination activities, the impact of vaccination will vary depending on the underlying model being used. The models within the VIMC vary from being static, whereby the direct effects of vaccination on FVPs are modelled assuming that pathogen transmission intensity is not modified by vaccination coverage, to dynamic, whereby infectious disease transmission dynamics, direct effects of vaccination on FVPs and herd effects are modelled. Herd effects account for any indirect effects of immunity (both natural and/or from vaccination activities) due to a reduction in transmission. For example, an increase in FVPs offers indirect protection as it reduces the risk of a non-vaccinated individual coming into contact with the disease. Herd effects will not arise for all vaccine preventable diseases, for example, Japanese Encephalitis is vector-borne (i.e. not transmitted person-to-person) and the majority of cases are due to spillover events¹⁴. Hence in this case, FVPs will not provide protection to others.

In this analysis, we focus on calculating impact estimates for HepB (dynamic model with long-term outcomes and herd effects), measles (dynamic model with short-term outcomes and herd effects) and YF (static model without herd effects)^15–18. We use anonymised countries for each of the pathogens, denoted by Country A for HepB, Country B for measles and Country C for YF, with a focus on the years 2000–2017. Standardised demographic data (live births per year, death rates) are based on the 2017 United Nations World Population Prospects (UNWPP)¹⁹. Immunisation coverage from 1980 to 2016 corresponds to the WUENIC estimates of National Immunization Coverage as published in July 2017¹², and 2017–2030 coverage corresponds to Gavi’s October 2017 Operational forecast (OP)^{13, 15}. Using each of the vaccine impact methods, we calculate the number of deaths averted due to vaccination over the years 2000–2017. Though we only show this for 2000–2017, we are taking into account the impact of vaccination activities from 2000 to 2030.

We also investigate the performance of the IE under the YoV methods with no stratification, stratification by activity and cohort separately and in combination by applying it to HepB, measles and YF in Countries A, B and C with updated coverage data. More specifically, for each of these pathogens, we update RI coverage from 1980 to 2017 based on WUENIC estimates published in July 2018¹², and 2018–2030 routine and campaign coverage is updated to Gavi’s October 2018 OP¹³. Changing coverage of activities alters the number of FVPs (population demography and models remain the same). We then re-run the models with this updated coverage scenario and compare to the IE projections in order to determine the reliability of the IE.

All of the analyses were carried out in R and the impact methods are available through the R package vimpact on github (https://github.com/vimc/vimpact)²⁰.

Estimates of the impact of vaccination

From 2000 to 2017, HepB, measles and YF have different vaccination activities in each of the countries. We analyse the impact calculation methods for RV alone and in combination with CV. We find that the estimated impact of vaccination varies depending on the method used.

There are only RV for HepB which are given as a birth dose (soon after birth) and as infant doses (<1 year old) (Fig.1A). As HepB-attributable mortality primarily affects those over 40 years of age, we find that the birth year method captures a greater impact as it accrues the long-term benefits of vaccination over the lifetime of birth cohorts. With the calendar year method accruing the impact for all ages over a particular year, this long-term impact is not captured (Fig.1B). Prior to 2013, YF has routine infant dose vaccinations (<1 year old) only with the calendar year method capturing slightly more impact (relative to HepB) as yellow fever largely affects those under 30 due to natural immunity acquired with age in older adults (Fig. 1A–1B). For measles from 2000–2017 in Country B, CV occur alongside RV. RV provides the first dose of a measles vaccine to those aged 1 or under (Fig.1A). As FVPs accumulate, the birth year and calendar year methods show an increasing number of deaths averted over time. In comparison to HepB where it is vital that the long-term benefits are captured, the calendar year method shows a high impact as measles-related mortality is focused in the under-5s (Fig.1B).

• Download figure
• Open in new tab

Figure 1.

Fully vaccinated persons (FVPs) and mean estimates of deaths averted for Hepatitis B (HepB) in Country A, measles in Country B and yellow fever (YF) in Country C from 2000 to 2017. (A) FVPs for HepB birth dose (BD routine) and infant dose (routine) routine vaccination activities. FVPs for first routine dose of a measles containing vaccine and measles campaign activities. FVPs for YF routine and campaign activities. (B) Impact by calendar and birth year methods. (C) Impact by year of vaccination (YoV) with impact ratio stratified by activity type and birth cohort.

The impact of vaccination estimated by the YoV methods varies over time with the activity stratification showing an averaged impact and the birth cohort stratification capturing gradual improvements in population health, particularly when herd effects are modelled. Hence, the activity stratification initially shows a higher number of deaths averted than the birth cohort stratification. Accordingly with RV only for HepB and prior to 2013 for YF, we see that the impact calculated by the activity stratification is greater as it is averaging the impact over all FVPs. The impact for the birth cohort stratification does not surpass the activity stratification within the time frame of this analysis for these two pathogens (Fig.1C). Notably for HepB, with RV occurring in the first year of life, the impact by year of vaccination and impact by birth year methods show a similar impact.

For YF, there is one CV in 2013 which results in a peak in the deaths averted with the largest impact seen with the activity stratification. The birth cohort stratification shows a lower impact as this is the first campaign (Fig.1C). For measles, we see the same pattern for the first campaign but for subsequent campaigns we see a higher impact with the birth cohort stratification because it is capturing the overall improvement in population health. Furthermore, herd effects are modelled for measles which contribute to the birth cohort stratification impact increasing as herd effects builds up reducing transmission in the population (Fig.1C).

Performance of the impact extrapolation method with updated coverage compared to model estimates

We explore the IE under the YoV methods with impact ratios stratified as per Table 2 for scenarios where vaccination coverage is updated from 2017 estimates to those published in 2018 (as described in Methods; change in FVPs shown in Figures 2A and 3A). The accuracy of the IE projections is determined by comparing to model estimates given the same updated coverage. Vaccine impact, i.e. deaths averted, by year of vaccination and by year of birth are visualised in Fig. 2 and Fig. 3, respectively, for Countries A, B and C over the years 2000–2017. The total relative difference in deaths averted over 2000–2017 between the IE and model estimates are shown in Table 4. We find that the accuracy of the IE varies depending on the impact ratio stratification method, underlying model dynamics and pathogen characteristics.

• Download figure
• Open in new tab

Figure 2.

Model estimates and impact extrapolation showing deaths averted by year of vaccination per year for hepatitis B (HepB) in Country A, measles in Country B and yellow fever (YF) in Country C from 2000 to 2017 using 2017 coverage (2017 model), 2018 coverage with corresponding model estimates (2018 model), and 2018 coverage with the impact extrapolation (2017–2018 IE). (A) Fully vaccinated persons (FVPs) in 2017 and 2018. (B) Impact by year of vaccination with unstratified impact ratio. (C) Impact by year of vaccination with impact ratio stratified by activity type. (D) Impact by year of vaccination with impact ratio stratified by birth cohort. (E) Impact by year of vaccination with impact ratio stratified by activity type and birth cohort.

• Download figure
• Open in new tab

Figure 3.

Model estimates and impact extrapolation showing deaths averted by year of vaccination per birth year for hepatitis B (HepB) in Country A, measles in Country B and yellow fever (YF) in Country C from 2000 to 2017 using 2017 coverage (2017 model), 2018 coverage with corresponding model estimates (2018 model), and 2018 coverage with the impact extrapolation (2017–2018 IE). (A) Fully vaccinated persons (FVPs) in 2017 and 2018. (B) Impact by year of birth with unstratified impact ratio. (C) Impact by year of birth with impact ratio stratified by activity type. (D) Impact by year of birth with impact ratio stratified by birth cohort. (E) Impact by year of birth with impact ratio stratified by activity type and birth cohort.

Using impact ratios without stratification, the IE performs relatively well for HepB in comparison to measles and YF (Table 4). As HepB only has RV, the activity type does not need to be taken into account. Furthermore, as this is given as a birth or infant dose, unstratified impact ratios would not capture variation in effect but this change would likely occur subtly over long time periods. Both RV and CV occur for measles and YF for which the IE with unstratified impact ratios underestimates the deaths averted due to the drop in coverage/FVPs in the 2018 coverage scenario prior to 2020 (Figs. 2A,B and 3A,B).

The IE with impact ratios stratified by activity does not perform well for HepB relative to measles and YF as it overestimates the impact associated with a change in coverage (Table 4; Figs. 2C and 3C). In the updated coverage scenario, HepB coverage improves over the years 2017–2022 creating more FVPs. With the activity stratification, this coverage improvement is being averaged over the whole time period. As such, earlier RV are artificially inflated. Additionally, as HepB-attributed disease occurs later in life, vaccinating more people born in later years will produce herd protection for those born over 2000–2017.

For measles, the activity stratification viewed by year of vaccination performs well (Table 4; Figs. 2C and 3C). Similar to HepB, measles coverage for RV improves greatly post-2020 with the 2018 coverage scenario. However, the IE does not overestimate the impact of this in the same way as it did for HepB because measles is focused in the under-5s so creating more FVPs later on does not produce herd effects for the 2000–2017 birth cohorts.

Using the activity stratification for YF, with changes to coverage, the IE works well with only a small underestimation of deaths averted (0.84%) by year of vaccination over 2000–2017 compared to model estimates (Fig. 2C) as it is a static model with no herd protection. However, there is larger underestimation of deaths averted (7.41%) by year of birth (Fig. 3C) as the FVPs vaccinated in the 2000–2017 birth cohorts is lower in the 2018 coverage scenario (Fig. 3A). Furthermore, compared to the unstratified impact ratio, the impact ratio for CV specifically is smaller and more sensitive to a reduction in FVPs. In contrast, the impact ratio for RV is larger than unstratified which accounts for the fact that RV will almost always be given to unvaccinated infants, as opposed to CV which, as dose distribution is assumed to be random, may revaccinate individuals who are already immune.

Using the birth cohort stratification, the IE performs well for HepB but underestimates the number of deaths averted by year of vaccination and birth year under the updated coverage scenarios for measles and YF (Table 4; Figs. 2D and 3D). Although the 2018 coverage scenarios generally have higher coverage post-2020, the coverage drops prior to 2020 for measles and YF but for HepB it improves slightly (Figs. 2A and 3A). As the birth cohort stratification captures greater yearly variation, it does well for the smaller change in HepB coverage and shows a corresponding decline in impact for measles and YF as it is more sensitive to changes in coverage than the activity stratification. This captures a limitation of the birth cohort stratification as it assumes the impact from RV and CV will be similar when they could be distinctively affected by dose wastage i.e. a RV will almost always be given to a susceptible child whereas campaigns may vaccinate individuals who have already been vaccinated, thus non-intuitively having a smaller impact.

Implementing the IE when stratifying by both activity and birth cohort results in the IE being sensitive to a combination of factors that affect the activity and birth cohort stratifications, such as the type of activities occurring, underlying model dynamics and yearly coverage changes. For measles and YF, this stratification captures a decline in coverage prior to 2020 which does not occur for HepB. Additionally it captures any herd protection arising from post-2020 coverage improvements to varying extents (seen more for HepB than measles because of differing age dynamics; Table 4; Figs. 2E and 3E).

When determining the accuracy of the IE, it is useful to look at both the relative total difference in deaths averted over 2000–2017 between the IE and model estimates (Table 4), and the difference across each year (Fig. 2 and Fig. 3). Some of the under- and over-estimation of the IE that is seen across the years can cancel out when looking at the aggregated difference over the whole time period. As such, the IE performs well when looking at the total difference for no stratification viewed by year of birth for measles, the activity stratification viewed by year of birth for HepB and the birth cohort stratification viewed by year of vaccination for measles (Table 4). However, when focusing on the difference in a specific year, the IE is seen to under- and over-estimate in certain years (Fig. 3B, Fig. 3C and Fig. 2D).

Overall, in our examples, we have shown that the IE works well and is an effective tool to update vaccine impacts. Notably, the activity stratification is accurate for static models with dose dependency but may overestimate the impact of coverage improvements when dynamic herd protection comes into play. The birth cohort stratification may underestimate impact if coverage declines before improving in later years.

Caution needs to be taken when deciding the appropriateness of implementing an IE. In cases where coverage is reduced more drastically or an activity is delayed, if the pathogen poses a risk of outbreaks and the model is dynamic, the IE may not be accurate as it may miss capturing the impact of any outbreaks that may occur during this period. With larger delays to activities, the risks of the IE not capturing such outbreaks would rise. Additionally, if a follow-up activity cannot reach the individuals missed, using the IE may not be accurate. In such cases, new model runs would be required to provide updated estimates of vaccine impact.