Cost-effectiveness of routine COVID-19 adult vaccination programmes in England ============================================================================== * Matt J. Keeling * Edward M. Hill * Stavros Petrou * Phuong Bich Tran * May Ee Png * Sophie Staniszewska * Corinna Clark * Katie Hassel * Julia Stowe * Nick Andrews ## Abstract **Background** In England, and many other countries, immunity to SARS-CoV-2 infection and COVID-19 disease is highly heterogeneous. Immunity has been acquired through natural infection, primary and booster vaccination, while protection has been lost through waning immunity and viral mutation. During the height of the pandemic in England, the main aim was to rapidly protect the population and large supplies of vaccine were pre-purchased, eliminating the need for cost-effective calculations. As we move to an era where for the majority of the population SARS-CoV-2 infections cause relatively mild disease, and vaccine stocks need to be re-purchased, it is important we consider the cost-effectiveness and economic value of COVID-19 vaccination programmes. **Methods** Here using data from 2023 and 2024 in England on COVID-19 hospital admissions, ICU admissions and deaths, coupled with bespoke health economic costs, we consider the willingness to pay threshold for COVID-19 vaccines in different age and risk groups. We partition the population into sixteen 5-year age-groups and three risk groups (no-risk, at-risk and immunosuppressed) and calculate the health savings, in terms of costs and quality adjusted life years (QALYs), per vaccine dose. We adopt three different methodological approaches to estimate the level of vaccine protection leading to five different estimates of efficacy. **Findings** Willingness to pay thresholds vary from less than £1 for younger age-groups without any risk factors, to over £100 for older age-groups with comorbidities that place them at risk. This extreme non-linear dependence on age, means that despite the different method of estimating vaccine efficacy, there is considerable qualitative agreement on the willingness to pay threshold, and therefore which ages it is cost-effective to vaccinate. For pre-purchased vaccine, where the only cost is administration (≈£10), a twice-yearly universal booster offer to all those aged 70 and over is cost-effective (the benefits outweigh the administration costs), while for the at-risk group this could be extended to those over 65. When the vaccine cost is included, the cost-effective ages reduce, such that at a total cost of £30 (for vaccine and administration) universal vaccine is only cost-effective for those over 75; while at £60 universal vaccine is only cost-effective for those over 80. **Interpretation** The historic offer of COVID-19 vaccination to those 65 and over for the autumn 2023 programme and those over 75 for the spring 2023 programme, aligns with our cost-effective threshold for pre-purchased vaccine when the only cost was administration. However, for future programmes, when vaccine costs are included, the age-thresholds slowly increase thereby demonstrating the continued importance of protecting the eldest and most vulnerable in the population. ## 1 Introduction The rapid development of COVID-19 specific vaccines was a major scientific triumph, which had profound implications for the control of a pandemic when population-level immunity to SARS-CoV-2 (the causative pathogen of COVID-19) was relatively low. As we move into a scenario in England where the majority of the population have been infected at least once by SARS-CoV-2 [1], and most people have had multiple primary and booster vaccine doses [2, 3], it becomes important to rigorously quantify the impact and cost-effectiveness of future COVID-19 vaccination programmes. Compared to modelling and analysis of COVID-19 vaccination for England in the initial years of the pandemic [4–6], this later problem is confounded by the high variation in immunity within the population to SARS-CoV-2 infection and COVID-19 severity. As a consequence, it is no-longer possible to utilise a simply defined efficacy that relates the protection offered by a COVID-19 vaccine to the risks experienced by unvaccinated individuals [7]. The analysis of the potential impact and cost-effectiveness of future COVID-19 vaccination programmes therefore requires a pragmatic quantitative approach that accounts for this extreme heterogeneity. For example, a cost-effectiveness analysis of the autumn 2023 vaccination programme in the Netherlands used a one-year decision tree-based cost-effectiveness model [8]. To assess the impact and cost-effectiveness of prospective routine COVID-19 vaccination programmes for England, in this study we take the simplifying approach of contrasting individuals who have recently been vaccinated (in the last six months) with those that have not. We use COVID-19 epidemiological data from England for March 2023 to March 2024, that also partitions the population by age and by three risk groups (not at risk, at risk but not immunosuppressed, and immunosuppressed). We present a collection of methodological approaches (each with differing assumptions) to calculate the risk of severe outcomes (hospital admission, ICU admission or death) and the protection offered by COVID-19 vaccination. We combine these estimates with bespoke health economic assessments that capture age and risk dependent heterogeneity in hospital cost and QALY (quality adjusted life years) losses following admission or death. This analysis allows us to calculate the willingness to pay threshold, which determines the price at which costs and benefits are equal. ## 2 Results For the purpose of this study, we partition the data into two equal intervals of 26 weeks associated with protection from the spring and autumn 2023 booster campaigns: spring/summer 2023, from week 11 to week 36 of 2023 (roughly mid-March to mid-September); and winter 2023/24, from week 37 of 2023 to week 10 of 2024 (roughly mid-September to mid-March the subsequent year). Given the lack of seasonal structure in the epidemiological data we acknowledge these partitions are somewhat arbitrary; nonetheless, the results are generally insensitive to the precise temporal divisions. Throughout we stratified the adult population into 16 different five-year age cohorts (*a*) from 15-19 to 90+, and three different risk groups (*r*): not at risk; at-risk (but not immunosuppressed); and immunosuppressed. The immunosuppressed group accounts for only 1.7% of the English population (peaking at 4.9% of those 80-84 years old), and therefore any results are subject to considerable uncertainty; we hence restrict our attention to the other two risk stratifications but return to the immunosuppressed group in the discussion. For each of the sub-groups we had data for the population size, the number of hospital admissions, severe hospital admissions (for brevity we refer to as ICU admissions) and deaths each week partitioned between those vaccinated in the last six months and those unvaccinated in that same time period (Fig. 1). This data allowed us to calculate the rates (and confidence intervals) of different severe health outcomes for each of the sub-groups. If the uptake of vaccine is assumed to be random within a sub-group, then the ratios between the associated rates defines the realised vaccine efficacy. This realised vaccine efficacy should account for the heterogeneous levels of pre-existing partial immunity in the population, including the impacts of past infection and past vaccination. However, it is important to consider some caveats with these data, chiefly that: the severe outcomes data does not separate infection episodes where COVID-19 was the primary or secondary cause; and that we assume outcomes are random within an age-and risk-group (i.e. there is no correlation between the proportion that get vaccinated, take additional precautions or are more severely affected). In the Discussion we elaborate on the potential implications of these data caveats on our findings. ![Fig. 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/11/12/2024.11.08.24316972.1/F1.medium.gif) [Fig. 1:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/F1) Fig. 1: COVID-19 vaccination and severe outcomes data from 2023 to 2024 in England. We display per week and per 100,000 individuals: vaccinations administered (top); number of hospital admissions (middle); number of deaths (bottom). The data is partitioned by risk group (no-risk in darker shades and at-risk in lighter shades) and vaccination status (recently vaccinated in blue and not recently vaccinated in red). The time period is divided into two 26-week seasons (Spring 2023 in orange and Winter 2023/24 in blue), which capture the spring 2023 or autumn 2023 booster campaigns and the major epidemic waves. The data come from the Secondary Uses Service [9], the ONS death records [10] and UKHSA’s Immunisation Information System [11]. We begin by reviewing the temporal pattern of vaccination and severe outcomes (hospital admissions and deaths), summing across all ages, but differentiating by risk-group and vaccination status (Fig. 1). The number of vaccine doses administered (Fig. 1, top panel) has two clear periods of activity (Spring and Autumn); this contrasts with the weekly hospital admissions (Fig. 1, middle panel) and weekly deaths (Fig. 1, bottom panel), which do not show such clear seasonal patterns. We now describe the statistical analysis of these results using the three methods (providing five distinct efficacy estimates) as described mathematically in the methods section. By combining our estimates for the rates of severe outcomes and the level of vaccine protection with the health economic estimates, we arrive at a willingness to pay threshold for the total cost of the vaccine and administration. We follow the guidelines established by the Joint Committee on Vaccination and Immunisation (JCVI - an expert scientific advisory committee which advises the UK government on vaccination and immunisation matters) and calculate two quantities: a central estimate of the willingness to pay threshold (using the maximum likelihood estimates and valuing one Quality Adjusted Life Year or QALY at £20,000) and the threshold for which 90% of scenarios are cost-effective (using £30,000 per QALY and fully accounting for uncertainty). The final willingness to pay threshold is the minimum of these two values, which for the vaccination programmes considered here is generally the central estimate [12]. #### Aggregate approach (Method 1) The simplest method (Method 1) used the aggregate data from one of the 26-week long time periods of interest (the Methods section provides a detailed mathematical description). In principle, this approach can provide estimates of vaccine efficacy for each age and risk group. However, the results are not necessarily reliable when the uptake is biased. For example, during the Autumn 2023 booster programme everyone 65 years of age and above was eligible; for those under 65 but not in a risk group about 1.5 million doses were administered, often due to them being carers or working in frontline health and social care - such individuals may have greater exposure to infection than others of the same age, which would lead to biases in vaccine efficacy estimates. For this reason, we only applied this first method to age-groups where there was a universal offer of vaccination (75 years and above for Spring 2023; 65 years and above for Autumn 2023 booster). As such, Method 1 generates an efficacy that depends on age, season, and risk-group; although the protection does not decline with time-since vaccination. As an exemplar of the calculations conducted for this method, we consider those not at risk and aged 80-84 during the winter 2023/24 season, and calculate the central estimate of the willingness to pay threshold. There were 412,436 individuals in this age and risk group, of whom an average of 335,928 had been vaccinated in the previous six months – leaving 76,508 not recently vaccinated. In the recently vaccinated group there were 186 hospital admissions (a rate of 55.4 per 100,000 over the six-month period), while in the not recently vaccinated group there were 128 hospital admissions (a rate of 167.3 per 100,000). Combining these numbers returned a vaccine efficacy of 66.9% against hospital admission. In addition, those that have been recently vaccinated had a shorter average length of stay in hospital: 12.8 days compared to 15.0 days for those not recently vaccinated. The average rates and lengths of stay mean that for every thousand doses of vaccine administered to this age and risk group, 1.12 hospital admissions and 18.0 days of hospital stay are prevented. (The 18.0 days saving are comprised of both reduced admissions and shorter stays.) These savings can be combined with the health economic estimates that every admission for this age and risk group is associated with an average 0.299 QALY loss, and that each day in hospital costs an average of £1182, to produce a willingness to pay threshold of £27.98 (= [1.12 *×* 0.299 *× £*20, 000 + 18.0 *× £*1182] */*1000) per dose for its protection against hospital admission alone. #### Time since vaccination approach (Method 2) Our second approach (Estimate 2a) used an assumed decline in protection over time (dropping by 74% over six months), generating a population-level average degree of protection (relative to a recently vaccinated individual) for each week. This again allowed us to estimate the basic rates of severe outcomes and the associated vaccine efficacy, and hence to determine the total amount of protection offered by a vaccine over the six-month period. To overcome the issues of biased vaccination in those under 65 year old in the no-risk group (as mentioned above), we extrapolate the vaccine efficacy estimates for all no-risk individuals over 65 to the entire no-risk population. We perform a similar extrapolation for the at-risk population, using the efficacy estimated from at-risk individuals over 65 years of age. As such, Method 2 generates an efficacy that depends on season and risk-group (but not age), where the protection declines with time-since vaccination. Considering hospital admissions, our central estimate of the vaccinate efficacy (in the no-risk group) was 70.0% immediately after vaccination, with an average vaccine efficacy of 47.0% across a six month period. For 80-84 year olds, this adjusts the previous estimates such that for every thousand doses of vaccine, 0.606 hospital admissions and 10.7 days of hospital stay are prevented. Combining these numbers with the health economic costs generated a willingness to pay threshold of £16.27 per dose for its protection against hospital admission. As an extension to this approach, Estimate 2b used a maximum likelihood approach to determine the appropriate decline in protection since vaccination for hospitalisation and death outcomes separately, based on the data for those over 65 years of age. These estimates suggest a slower decline in protection than assumed for Estimate 2a, and are comparable with recently estimates from Scottish data [13]. For hospital admissions in the no-risk group, our revised central estimate of vaccine efficacy was 62.8% immediately after vaccination, with an average vaccine efficacy of 57.7% across a six month period. These revised protection values, change our central estimates for no-risk 80-84 year olds to preventing 0.639 hospital admissions and 11.4 days of hospital stay per thousand doses of vaccine; leading to a willingness to pay threshold of £17.24 per dose for protection against hospital admission. #### Externally predefined vaccine efficacy approach (Method 3) Our third and final method used vaccine efficacy estimates derived from alternative data sets. The approach could be used for any set of vaccine efficacy estimates [15]; here we make use of two UK-based estimates: Estimate 3a uses vaccine efficacy and waning of protection from Public Health Scotland (PHS) [13]; Estimate 3b uses efficacy estimates estimates from UKHSA [14]. The PHS estimates come from individuals aged over 65 that were part of the Evaluation and Enhanced Surveillance of COVID-19 (EAVE-II) cohort, and included a surveillance period of September 2023 to July 2024. The UKHSA estimates used a test-negative case-control study design, where positive PCR tests from hospitalised individuals are cases and comparable negative PCR tests are controls; this approach has been used to produce official vaccine estimates for the UK [16, 17]. Our approach utilises a probability distribution for the efficacy which is then scaled in a deterministic manner over time. We use this formulation to capture the uncertainty in the peak level of protection, and the pattern of change over time follows the mean of the PHS or UKHSA estimates. As such, Method 3 generates an efficacy that is the same for all ages, seasons and risk groups although the protection does decline with time-since vaccination. For hospital admissions, the PHS estimates generated a mean vaccine protection across a six month period of 65.3% (Method 3a), while the UKHSA estimates led to 27.1% protection (Method 3b). These vaccine efficacy estimates correspond to willingness to pay thresholds of £18.74 and £8.86 per dose respectively for protection against hospital admission. ### 2.1 Willingness to pay thresholds Similar calculations can be performed for ICU admissions and deaths, for different age and risk groups and for Spring 2023 and Autumn 2023 boosters; the full calculation also computes the probability distribution associated with the rates and therefore captures the uncertainty in the vaccine protection (see Methods). These statistical results can then be combined with the health economic costs and benefits, which also capture uncertainties to generate a total willingness to pay threshold accounting for all the outcomes and following the JCVI guidelines on dealing with uncertainty. For the spring 2023 and autumn 2023 boosters, and the six-month period of COVID-19 infection, Fig. 3 shows the willingness to pay threshold for different ages, two different risk groups and the five different methods of calculating the rate and efficacy distributions. We did not apply Method 1 (aggregated calculation method) to groups with insufficient data or where we believed there was a notable bias between those receiving a booster dose and those that did not (under 65 for the autumn 2023 booster and under 75 for the spring 2023 booster). For each Method, we independently calculated vaccine efficacy values for each booster season (spring 2023, autumn 2023) and for protection against the three forms of severe disease outcome (hospital admission, ICU admission and death). All methods, risk-groups and seasons show the same basic trend: willingness to pay increases dra-matically with age, capturing the well-recognised observation that older individuals are more at risk of severe outcomes following COVID-19 infection [6, 18, 19]. The results also highlight the generally higher willingness to pay (across all age-groups and methods) for individuals classified as at-risk compared to no-risk, due to the higher chance of more severe outcomes. Finally we observe higher willingness to pay for the autumn 2023 booster compared to the spring 2023 booster, due to the higher number of infections in the associated winter period. We note that the JCVI require both the central estimate (at £20,000 per QALY) and 90% of uncertainty scenarios (at £30,000 per QALY) to be cost effective - this translates to the final willingness to pay threshold being the minimum of the most likely value (dot in Fig. 3) and the 80% credible interval (narrower box in Fig. 3). We consider the implications for the willingness to pay threshold for each age-group (Table 1) in terms of who should be offered the vaccine. Given the uncertainty in the seasonality of COVID-19 cases (we have yet to experience a single distinct winter wave or even a consistent seasonal pattern) we amalgamate results from the spring 2023 and autumn 2023 boosters, and report the average willingness to pay threshold across both boosters. In addition, it is thought that there is substantially better vaccine uptake from a universal programme rather than one that targets at-risk groups. Therefore, we also report the weighted average which is willingness to pay threshold for a universal programme. We again see the general trends that older age-groups have a higher willingness to pay threshold, such that vaccination is more likely to be cost-effective. View this table: [Table 1:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/T1) Table 1: Willingness to pay thresholds for the combined Spring/Autumn 2023 boosters. Thresholds are given for all ages (although ages 15-39 have been amalgamated for simplicity), for no-risk, at-risk and universal vaccine deployment and for the five methods of determining the efficacy. Values are rounded to the nearest pound, and values less than £1 are indicated. Ages, risk-groups and methods where the willingness to pay threshold is determined by the JCVI uncertainty condition (that there is a 90% chance of cost-effectiveness at £30,000 per QALY) are marked with an asterisk. #### Pre-f-lchased vaccine The COVID-19 booster vaccines used during spring and autumn 2023 were pre-purchased during the pandemic; they can therefore be considered to have zero price. The only cost of vaccination is therefore the administration costs, which have generally been between £7.50 and £10.04 (although higher values may apply for administration to house-bound individuals). Using this range as our cost, we observe that vaccination of no-risk individuals aged 75 and older is likely to be cost-effective for all methods of determining efficacy, although for Estimate 3b (UKHSA vaccine efficacy [14]) this will be sensitive to the precise costs of administration. We note that even a vaccine with 100% efficacy (not shown) is not cost effective for no-risk individuals under age 65 as the administration costs are greater than the benefits. For at-risk individuals, the age at which vaccination is cost effective reduces to either 60 or 65 years of age depending on the method used; while for a universal vaccine the age threshold is either 65 or 70 years old. Taking Method 2 (Estimates 2a and 2b) as our benchmark, we would therefore suggest that the ideal strategy for a pre-purchased vaccine would be to offer a universal vaccination to all individuals over 70, and also to offer vaccination to at-risk individuals over 65. While vaccination of those aged 70-74 in the no-risk group is not predicted to be cost-effective (willingness to pay is £6-7), the advantages in terms of higher uptake in the at-risk group of a universal programme is likely to off-set any costs in the no-risk group. If we consider the spring 2023 and autumn 2023 boosters separately then we find the age thresholds for universal vaccination are 75 and 65 respectively (Tables 3 and 4 in the Supplementary Tables). These estimates come with the benefit of hindsight, such that the scale of the outbreak is known, as is the vaccine efficacy; therefore it is remarkable that these are in agreement with the JCVI advice for individuals not at risk for these booster programmes [20, 21]. View this table: [Table 2:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/T2) Table 2: The lower age threshold for cost-effective vaccination using the imputed willingness to pay values. For each combined cost of vaccine and administration, we give the minimum and maximum calculated age thresholds above which vaccination in cost effective, together with the estimates that generate these thresholds. View this table: [Table 3:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/T3) Table 3: Willingness to pay thresholds for the Spring 2023 boosters. Thresholds are given for all ages (although ages 15-39 have been amalgamated for simplicity), for no-risk, at-risk and universal vaccine deployment and for the five estimates of vaccine efficacy. Values are rounded to the nearest pound, and values less than £1 are indicated. Ages, risk-groups and estimates where the willingness to pay threshold is determined by the JCVI uncertainty condition (that there is a 90% chance of cost-effectiveness at £30,000 per QALY) are marked with an asterisk. Shaded cells correspond with ages and risk groups where vaccination was recommended in the Spring 2023 COVID-19 vaccination programme [20]. View this table: [Table 4:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/T4) Table 4: Willingness to pay thresholds for the Autumn 2023 boosters. Thresholds are given for all ages, for no-risk, at-risk and universal vaccine deployment and for the five estimates of vaccine efficacy. Values are rounded to the nearest pound, and values less than £1 are indicated. Ages, risk-groups and estimates where the willingness to pay threshold is determined by the JCVI uncertainty condition (that there is a 90% chance of cost-effectiveness at £30,000 per QALY) are marked with an asterisk. Shaded cells correspond with ages and risk groups where vaccination was recommended in the Autumn 2023 COVID-19 vaccination programme [21]. #### Purchased vaccine In the immediate future, it will become necessary for the UK to purchase vaccines and place vaccination against COVID-19 on a equal cost-benefit assessment as all other vaccines. Which age-groups it is cost-effective to vaccinate will be highly dependent on the price paid for the vaccine, which is currently unknown. To illustrate the patterns, here we consider £25, £50 and £75 as combined costs for the vaccine and administration. At £25 per dose delivered, all Methods broadly agree that universal vaccination of those aged over 75 is likely to be cost effective, assuming that losses from vaccinating no-risk individuals aged 75-79 are compensated for by an increased uptake in at-risk individuals in the same age-group. It is uncertain whether vaccination of 70-74 year olds in the at-risk group will be cost-effective with different Methods producing conflicting results, potentially suggesting a need to refine the co-morbidities that define at-risk. At £50 per dose delivered, the age threshold for universal vaccination increases to 80 (Estimates 1 and 3a) or 85 (Estimates 2a, 2b and 3b). Only Methods 1 and 3a suggest that vaccinating younger individuals (those aged 75-79) in at-risk groups could be cost-effective. Finally, at £75 per administered dose, it is likely that universal vaccination is only cost-effective for those over 85; although Method 3a suggests universal vaccination is not cost-effective at any age, and only vaccination of at-risk individuals over 90 is cost-effective. ### 2.2 Sensitivity Some of the uncertainties in our epidemiological and health-economic model can be quantified. For example, based on the available data, inferring the rate of hospital admissions produces a detailed distribution of parameter uncertainty (Fig. 2). However, for other elements - such as those associated with the precise scale and timing of future waves, or the degree of protection against future variants - the uncertainty cannot be quantified; we cannot determine the probability that the 2024/25 winter wave will be bigger (or smaller) than the 2023/24 winter wave. For these uncertainties we present an uncertainty analysis, showing the willingness to pay threshold as a function of the unknown quantity. ![Fig. 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/11/12/2024.11.08.24316972.1/F2.medium.gif) [Fig. 2:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/F2) Fig. 2: Efficacy estimates from the five approaches, for no-risk individuals following the autumn 2023 booster. Throughout we present efficacy estimates for protection against hospital admissions in green (triangle symbol), protection against ICU admissions in purple (inverted triangle) and protection against mortality in red (square). For the aggregate estimate (Estimate 1), where we assumed vaccine efficacy was fixed across a six-month interval, we estimated the efficacy for each 5-year age group that was universally offered the vaccine. We display the 95% prediction intervals (spanning 2.5th and 97.5th percentile), with symbols denoting the mean values. Estimates 2a and 2b show the inferred efficacy distributions with either an assumed or fitted decline. Estimates 3a and 3b are based on values from PHS [13] and UKHSA [14]; the PHS estimates assumed the same efficacy for hospital and ICU admission, while the UKHSA estimates assumed the same efficacy for all severe outcomes. Throughout we show 95th percentiles of the prediction interval (shaded ribbons) as well as the mean value (solid lines); symbols show the mean level of efficacy over a six-month period. ![Fig. 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/11/12/2024.11.08.24316972.1/F3.medium.gif) [Fig. 3:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/F3) Fig. 3: Using data from the spring and autumn 2023 boosters and the 2023/24 winter wave of COVID-19, vaccine threshold prices stratified by age-group, risk-group and modelling approach. We calculate the vaccine threshold price for the two time periods (top row: winter 2023/24; bottom row: spring 2023), 16 age groups (y-axis), three risk groups (panels) and five methodologies (red, dark-blue, light-blue, grey and black). Dots show the most likely value assuming £20,000 per QALY; when assuming £30,000 per QALY the extended bar-and-whisker plots show the 95%, 80% and 50% credible intervals combining all sources of uncertainty. Points that are above or below the axis are shown as triangles. Note that the y-scale is non-linear to better display lower willingness to pay thresholds. Retaining our focus on universal vaccination, and using Method 2 (Estimates 2a and 2b) throughout, we consider three sensitivities: a shift to the timing of the wave(s); a rescaling of the size of the wave(s); and different levels of vaccine protection (Fig. 4). We also use a combined cost of £25 per administered dose to illustrate the impact of these uncertainties, but the implication of other costs can be assessed from Fig. 4. ![Fig. 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/11/12/2024.11.08.24316972.1/F4.medium.gif) [Fig. 4:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/F4) Fig. 4: Sensitivity of the willingness to pay threshold for universal vaccination to the timing of the epidemic wave, the magnitude of the epidemic waves and the efficacy of the vaccine. For a shift in the timing of waves or waves of different sizes (top row), we consider how the threshold calculated by Estimates 2a (solid line) and 2b (dashed line) for older age-groups vary. When considering the size of the waves, the Spring 2023, Winter 2023-24 and Winter 2022-23 are shown for comparison. For vaccine efficacy, we extrapolate the willingness to pay thresholds from Estimate 2a and consider protection against hospital admissions and mortality separately (bottom row). We consider four age-groups (65-69 to 80-84) and show the contours where the willingness to pay threshold is equal to £7.50-£10.00 (grey shaded region), £25, £50 and £75; we also show the approximate efficacy estimates for the five Methods, weighted so that they apply to universal vaccination over combined spring 2023 and autumn 2023 boosters. Changes to the timing of the waves has the smallest impact on the willingness to pay thresholds (Fig. 4, top-left). Earlier waves consistently reduce the threshold, such that if the waves were 4-7 weeks earlier (depending on the Method) it may no-longer be cost effective to offer universal vaccination to those aged 75-79. There is some evidence that earlier vaccination by around four weeks could increase the benefit of vaccination (assuming the timing of the waves remained the same). The size of the waves of hospital admissions and deaths has a linear impact on the willingness to pay (Fig. 4 top-right). At the £25 cost, very large waves could lead to vaccination of 65-69 year olds being cost effective, although this would require outbreaks more severe than those of Winter 2022-23. Similarly, waves that are consistently 15-25% smaller lead to the vaccination of 75-79 year olds no-longer being cost-effective. We consider uncertainty in vaccine efficacy to capture either improvements in vaccine targeting or the risk of evolution of vaccine escape. For the 65-69 age group, it would require a near perfect efficacy for the vaccination of this age-group (at a cost of £25) to be cost effective. Similarly, a large improvement in efficacy is required for universal vaccine of those aged 70-74 to be cost effective at £25 per administered dose. For older age-groups (for example 80-84 year olds), the efficacy would need to be extremely low to change the cost effectiveness. ## 3 Discussion Here we have quantified the protection offered by recent COVID-19 booster vaccination by using national data sets on COVID-19 associated hospital admissions, ICU admissions and deaths in England linked to individual vaccination records. We used three different approaches leading to five methods to infer the underlying risk of severe outcomes in different age and risk groups (no-risk, at-risk or immunosuppressed, as defined in the Green book chapter 14a [22]) dependent on vaccination status. We combined these inferred risks with bespoke health economic parameters for COVID-19 on hospital costs and QALY losses to define willingness to pay thresholds according to JCVI guidelines. As such, the willingness to pay threshold defines the maximal combined cost of a vaccine dose plus the administration, such that vaccination would be cost effective; the JCVI guidelines ensure that both the most likely scenario and uncertainty are taken into account in this evaluation. In practice the willingness to pay threshold is used to translate the realised total vaccination cost (vaccine plus administrative costs) into a set of age and risk groups who it should be cost effective to vaccinate. For COVID-19, given the increasing vulnerability with age, this equates to a lower age threshold in each risk group, with everyone older being eligible for vaccination. Table 2 shows this lower age threshold for a range of costs and for universal vaccination and at-risk groups; based on the assumption the universal vaccination has the benefit of increased uptake in the at-risk population). This analysis assumes that the same threshold is applied to both spring and autumn boosters, given the absence of a clear seasonal pattern in COVID-19 cases and outcomes to date. For each cost we show the lowest and highest estimated age threshold together with the associated models. Despite the very different assumptions underpinning the five methods, there is a remarkable level of agreement, in part due to the extreme increase in risk with increasing age. We find that Estimate 3a (PHS estimates) is always associated with the lowest age estimates, while Estimate 3b (UKHSA efficacy estimates) is always associated with the highest estimates - providing bounds on who should be considered for cost-effective vaccination. For low total costs (generally associated with the assumption that the vaccine is pre-purchased and therefore zero cost), we find that universal vaccination is cost-effective for those aged over either 65 or 70 depending on the methodology employed; however the majority of the models agree that for those in the at-risk group vaccination should be offered to those 65 and over. For slightly higher combined costs, for example at £25, most methods (Estimates 1, 2a and 2b) agree that the universal threshold should be those over age 75, and this is also the most preferred age threshold for the at-risk group. For much higher costs, for example at £50-60, the majority of the models suggest that universal vaccination should only be offered to those over the age of 85, with weak or no support for vaccination of younger at-risk individuals. Those who are immunosuppressed form a small but highly vulnerable part of the population. Due to the small size of this risk group, and the heterogeneity within it, results from all estimations of efficacy are extremely noisy. However, as a general rule, vaccination of all immunosuppressed individuals over age 35 (or age 45-55 depending of the method) is cost effective at £10 (or £25) per administered dose. Results for younger age-groups, which are only a small proportion of the immunosuppressed population are highly variable, suggesting that uniform vaccination of the entire immunosuppressed population may be the most precautionary option. There are some limitations of our analysis, primarily associated with the properties of the data. The partition of the population into just risk groups (no-risk, at-risk but not immunosuppressed and immunosuppressed) is somewhat simplistic. It is likely, that a finer partitioning of the risk groups could increase the willingness to pay threshold for some risk groups, although this has to be weighed against the practicalities of more complex criteria. For example, recently diagnosed haematological malignancy or severely reduced kidney function would both place an individual in the at-risk (but not immunosuppressed) group; these conditions have been observed to have a substantially greater impact on the risk of severe outcomes than chronic heart disease which also places an individual in the at-risk group [22, 23]. There are also implications if those accepting vaccination are a biased sample of the individual sub-groups (for example, those who perceive themselves at greatest risk may be more likely to be vaccinated). Such biases would reduce the estimated efficacy from Methods 1 and 2. Another potential issue with the data is whether individuals are hospitalised ‘with COVID’ (that is, they have COVID-19 infection at the time of admission but it is not the cause of admission) or ‘for COVID’ (that is, COVID-19 infection is the primary cause of their admission). Our selection criteria for admissions has been designed to minimise the numbers where COVID-19 is not a major contributing factor; inclusion of individuals ‘with-COVID’ will lead to over-estimates the willingness to pay threshold. The static approach we have adopted here provides a worst case scenario as it does not include the indirect protection offered by vaccination. For example, vaccination of 50-54 year olds only accounts for their direct protection against severe COVID-19 disease, and does not account for the associated reduction in the force of infection experienced by more vulnerable individuals, nor any societal impact due to their illness. However, a dynamic approach where SARS-CoV-2 infection is modelled mecha-nistically is impractical due to the extreme heterogeneity in history of infection and vaccination. This is exemplified by those studies that have used dynamic SARS-CoV-2 infection transmission models to perform cost-effectiveness analyses of 2023 COVID-19 booster vaccination programmes [24, 25], where heterogeneity is only dependent on vaccine history not infection history. Furthermore, many unquantifiable uncertainties could influence these findings. We have explored the timing and size of waves (Fig. 4), but cannot associate a probability with any pattern of future outbreaks. These results have shown that while both timing and scale of the epidemic wave(s) have an effect on the vaccine threshold price (with scaling having the largest influence) both are relatively small compared to the impact of age, at most changing eligibility by one 5-year age group for the scale of waves recently experienced. More concerning is the potential impact of novel variants. A novel variant with greater severity or which overcame population-level immunity would likely generate far larger waves of severe cases and therefore merit an expansion of the vaccine programme. Similarly, a new variant with sufficient vaccine escape - such that the level or duration of protection induced by prior COVID-19 vaccination was reduced - may require a re-evaluation of the programme, either reducing its range due to the lower impact per dose, or increasing its range to help combat overall levels of infection and hence protect the most vulnerable. Vaccination against COVID-19 therefore remains an important public health tool for protecting the elderly and most vulnerable in our society. However, wide spread partial immunity in the population, the lower risks of severe illness currently observed, and the need for vaccines to be cost-effective means that offers of population-wide vaccination for all ages are no longer affordable. Instead we need to carefully balance the vaccine costs against the health saving and health benefits; the precise age threshold will depend on the price paid for the vaccine, but only universal programmes that target those above 70 years old are likely to be cost effective. ## 4 Methods Here we provide details on the data used throughout this work (Section 4.1), on the different modelling approaches to infer the health episode outcome rates which are dependent on age, risk and vaccination status (Section 4.2), and the health economic data and assumption used to make cost-effectiveness assessments (Section 4.3). When taken together these generate the willingness to pay thresholds for different target groups being included in a routine COVID-19 vaccination programme for England (Section 4.3). ### 4.1 Data on admissions and deaths by age, risk and vaccination status Our aim was to generate data on the number of individuals each week that were admitted to hospital, admitted to ICU (a term we use for brevity to refer to severe hospitalisation cases) or died with COVID-19. These data are further separated by age (five-year age bins from 15-19 to 90+), risk group (no-risk, at-risk or immunosuppressed, as defined by the Green Book Chapter 14b [22]) and by vaccination status (booster within the last six months or not). This involves the linkage of two data sets on health outcomes (admissions provided by the Secondary Uses Service [9] and deaths provided by the ONS [10]) to individual records on vaccine uptake and demographic status. The Secondary Uses Service [9] is a database of timely completed hospital admissions, including ICU attendance, for all NHS hospitals in England. To determine COVID-19 related hospitalisations for age 15+ we merged COVID-19 testing data with respiratory coded discharge data obtained from the Secondary User Service. The respiratory discharge ICD-10 codes used were: J04* Acute laryngitis and tracheitis ; J09* Influenza due to identified avian influenza virus; J10* Influenza with pneumonia, other influenza virus identified; J11* Influenza with pneumonia, virus not identified; J12* Viral pneumonia, not elsewhere classified; J13* Pneumonia due to Streptococcus pneumoniae; J14* Pneumonia due to Haemophilus influenzae; J15* Bacterial pneumonia, not elsewhere classified; J16* Pneumonia due to other infectious organisms, not elsewhere classified; J17* Pneumonia in diseases classified elsewhere; J18* Pneumonia, organism unspecified; J20* Acute bronchitis; J21* Acute bronchiolitis; J22* Unspecified acute lower respiratory infection; J80* Adult respiratory distress syndrome ; U071* COVID-19, virus identified; U072* COVID-19, Virus not identified; U04* Severe acute respiratory syndrome (SARS). We then identified a COVID-19 related hospital admission either by a hospitalisation (with a respiratory coded discharge) that had a COVID-19 positive test (PCR or lateral flow) within the period two days prior to the day after the date of admission, or a discharge where the primary ICD-10 discharge diagnosis was COVID-19. ICU admissions (corresponding to severe hospitalisation admissions) were those with NHS Office of Population Censuses and Surveys (OPCS) codes for oxygen/ventilation — E85* Ventilation support, E89* Other respiratory support, X58* Artificial support for body system, X52* Oxygen therapy — or Main Specialty/Treatment Function classification associated with an ICU admission (Main Specialty Code: 192 Critical Care Medicine; Treatment Function Code - reporting the specialised service within which the patient was treated: 192 Critical Care Medicine or 242 Paediatric Intensive Care). We collectively refer to these severe hospitalisation admissions as ICU admissions. We collapsed hospitalisations in the same individuals into ‘spells’ where there was more than one admission recorded on the same day or where spells overlapped (based on admission and discharge). We combined spells within individuals if two spells started within 15 days of one another. In these combined spells we counted information on severity if it occurred in any of the collapsed admissions. Formal ONS deaths were identified with COVID-19 mentioned on the death certificate [10]. Both set of individuals (admissions and deaths) were then linked to UKHSA’s national COVID-19 vaccine register, the Immunisation Information System (IIS) [11], that holds COVID-19 vaccine dates, COVID-19 vaccine eligible risk group status [26] and demographic information for all individuals in England. Linkage was performed by matching NHS numbers; matching was perfect for admissions, but failed for around 6% of deaths. The age of these unmatched deaths is known, but vaccination and risk status is randomly assigned based on those with known status. Vaccine coverage data for each week was also derived from the IIS using age on 31st August of each year. ### 4.2 Mathematical outline of the modelling approaches We used three differing modelling methodology leading to five different estimates of the protection offered by COVID-19 vaccination. The first method we call the ‘aggregate approach’, the second method we call the ‘time since vaccination approach’ and the third method we call the ‘externally predefined vaccine efficacy approach’. We base each approach on having a set of weekly observations of a particular public health outcome (admission to hospital, admission to ICU or death due to COVID-19), summed across all ages and risk groups; we label this quantity *Y**w*. We also defined the normalised rate of weekly observations: ![Formula][1] For all that follows we consider a particular age and risk class and a particular observation of a public health outcome; when performing our numerical calculations we need to keep track of these three dimensions, but here for notational brevity we outline how the calculations are performed for a single risk-group, age-class and set of observations. We denote the size of the population as *N*, the number of individuals vaccinated within the last six months as *V**w* (given the current week being *w*) and the number vaccinated exactly *W* weeks ago (that is vaccinated in week *w* − *W*) as ![Graphic][2]. It should be noted that ![Graphic][3] . For simplicity of notation we define those individuals that have not been vaccinated in the last six months by *U**w* = *N* − *V**w*. For a given observation, we assumed the number observed in a given week (*w*) for a particular public health outcome for those not vaccinated in the last six months was ![Graphic][4]. Similarly, we assumed the number observed in a given week for a particular public health outcome for those that had been vaccinated in the last six months was ![Graphic][5]. #### Method 1: Aggregate approach Our simplest calculation was to consider the rate *s* (or *r*) that a random person in the vaccinated (or unvaccinated) group enters our observation class, assuming a Poisson distribution. The probability distribution of a particular rate is proportional to the likelihood: ![Formula][6] with the proportionality constant defined such that the distribution integrates to one. Here ![Graphic][7] and ![Graphic][8] refer to the average number of unvaccinated and vaccinated individuals during the time period, and *π* is the weakly informative prior probability distribution for the per capita rate of observation (*π*(*r*) = *exp*(−*r*)). Given our assumption that the per capita risks for vaccinated and unvaccinated individuals are independent, the joint distribution is given by the product: ![Formula][9] The probability distribution for the vaccine efficacy *ρ* = 1 − *s/r* can then be calculated as: ![Formula][10] where the *r* term in the integral is the natural scaling, and ensures that the probability to integrate to one. This approach effectively treats the entire outbreak as a single observation. #### Method 2: Time since vaccination approach Method 2 requires two combined elements to realise the risks and efficacy distributions: firstly, a method to determine the efficacy assuming that the level of protection wanes after vaccination; and secondly, a method to include the estimated efficacy into the rate estimation approach. This two-step process is needed as we combine the estimated efficacy from older age-groups (which we believe is subject to fewer biases) and use this distribution for all ages. To acknowledge that the level of protection from the vaccine wanes over time, we define the relative level of protection from the vaccine after *W* weeks *X**W*, which peaks at one and is assumed to decay substantially by the time when *W* reaches six months. For Estimate 2a, we assume that *X**W* = exp(−*W* 2*/*500) which produces a decline similar to that estimated against the initial Omicron BA.1 wave [27]. At any time, we can then define the aggregate level of protection across the entire population *x**w* in week *w* as: ![Formula][11] where ![Graphic][12] is the number of individuals vaccinated *W* weeks ago, and *V**w* the number vaccinated within the last six months. Given that the level of protection for vaccinated individuals depends on both *s* (the rate when the vaccine offers maximal protection) and *r* (the rate when the vaccine offers least protection, assumed to be after six months), it is practical to consider the joint distribution: ![Formula][13] Here, the rate of observation for vaccinated individuals is a weighted combination of *r* and *s* depending on the population level amount of protection at any given time, *x**w*. The probability distribution of efficacy can again be defined from this joint distribution: ![Formula][14] As an extension to this approach, Estimate 2b allowed the decline in protection to be estimated. We define *X**W* = exp(−*W* 2*/*(2*σ*2)), and set *σ* to maximise Eq. (4) across all age-groups but independently for each public health outcome. For Method 2 (and hence Estimates 2a and 2b) we combine (by taking the product) the estimated efficacies, *P*eff(*ρ*), for multiple older age-groups, and once normalised, treat this combined probability (*π*Eff(*ρ*)) as the efficacy distribution across all age-groups. The ages chosen to be aggregated, and hence used as a sample for the entire population, correspond to those older age-groups where vaccination was universally offered irrespective of risk status. To calculate the impact of vaccination then requires an approach that can determine the rates consistent with the assumed efficacy distribution *π*Eff(*ρ*). The joint distribution of the rates *r* and the efficacy *ρ* can then be expressed as: ![Formula][15] where *P**J* is defined as in Eq. (4). This allows us to calculate the appropriate joint distribution on the two rates: ![Formula][16] In essence this methodology ensures that the rates for the two groups (vaccinated and unvaccinated) are consistent with both the observations for both classes and the assumed efficacy distribution. However, there are younger individuals in the no-risk group who were vaccinated, and therefore do not represent the average population of that age. (There are reasons why these individuals might have been vaccinated including: being pregnant; working in frontline NHS and social care, or care homes; or being at-risk but incorrectly classified in the data). In such cases it is prudent to exclude these individuals from our analysis, in which case: ![Formula][17] where *P**U* is taken from Eq. (1), and then Eq. (6) still holds. #### Method 3: Externally predefined vaccine efficacy approach For Method 3 (generating Estimates 3a and 3b) the efficacy has been estimated in advance, which requires a slight adjustment to our previous approach. Usually, the available estimates may be of the efficacy distribution at different times since vaccination (e.g. ΠEff(*ρ, W*) in week *W*). Here, in keeping with our previous approaches, we approximate this externally derived distribution with a single distribution (*π*Eff(*ρ*)) coupled with a deterministic scaling in protection *X**W* which is also informed by the external data. Using *π*Eff and *X**W* based on the external source, we can again return to Eqs. (5) and (6) to inform about the rates. ### 4.3 Evaluating vaccine programme cost-effectiveness For the respective modelling approaches outlined above, the health outcome rates and vaccine efficacy estimates, we detail the cost-effectiveness assessments. For each of the different target groups being included in a routine COVID-19 vaccination programme for England, this generated a willingness to pay for each administered vaccine (the combined cost of administration and purchase of the vaccine). We first explain the health economic calculations that determined the price point that a vaccine dose would be deemed cost effective. We then overview the use of the health economic parameters within the cost effectiveness assessment. #### Health economic model calculations Following from the joint probability density calculations outlined above, we computed the probability density for the health economic benefit (*B* = *rCost**U* − *sCost**V*) of a single dose of vaccine as: ![Formula][18] For vaccinated and unvaccinated groups respectively, *Cost**V* and *Cost**U* represent the total cost associated with whichever severe outcome is being considered (hospital admission, ICU admission or death) and include both direct treatment costs and the value placed on QALY losses. In this formulation, the joint distribution (*P**J*) can be as defined by Eq. (2) (for Method 1) or Eq. (6) (for Methods 2 and 3). Following JCVI guidelines we considered two quantities [12]. First, using the mean costs and benefits, and assuming one QALY (Quality Adjusted Life Year) is valued at £20,000; we calculated the benefit per vaccine dose at the maximum likelihood values for the rate a random person in the unvaccinated group has the severe outcome under consideration (*r**) and the rate a random person in the vaccinated group has the severe outcome (*s**): ![Formula][19] Second, using £30,000 per QALY and accounting for variability in cost and rate parameters, we calculated the 10th percentile of the distribution: ![Formula][20] A vaccine is then deemed cost effective if the cost per dose including administration is less than both the calculated benefits: Vaccine price + Admisistration price *<* min(BenefitMLE, Benefit10%) = Willingness to pay threshold. #### Parameters of the health economic model The health economic calculations required monetary costs and a measure of disease severity (captured through the loss of Quality-adjusted life years, QALYs) associated with each level of disease outcome. For infection episodes resulting in hospitalisation and ICU admission, we derived estimates from the PANORAMIC trial (Platform Adaptive trial of NOvel antiviRals for eArly treatMent of COVID-19 in the Community) [28] (Table 5 in the Supplementary Tables). View this table: [Table 5:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/T5) Table 5: Summary of health economic values for hospitalisation, ICU and death events. We provide the following statistics stratified by age-group and risk status, which are displayed in Fig. 5. For infection episodes resulting in hospitalisation and ICU admission, we report the mean estimates (with 95% confidence intervals in parentheses) for the cost per day (£) to the nearest integer and QALY loss to 2 decimal places. Estimates in regular typeface are from the PANORAMIC trial (Platform Adaptive trial of NOvel antiviRals for eArly treatMent of COVID-19 in the Community) [28], with estimates in italic typeface corresponding to age and risk-groups where there were no data and displayed results are extrapolations. Per death we report the QALY loss to 2 decimal places, with estimates produced using the approach of Briggs *et al*. [31] to adjust for the presence of comorbidities that influence both life-expectancy and health-related quality of life. ##### PANORAMIC trial The purpose of the PANORAMIC trial was to find out in which people new antiviral treatments for COVID-19 in the community could reduce the need for hospital admission and improve health outcomes. Participants enrolled in the PANORAMIC trial had tested positive for SARS-CoV-2 infection, had ongoing symptoms consistent with COVID-19, had not been previously hospitalised due to COVID-19 (a criteria resulting from the trial focus being on community-based treatment) and were either aged 50 years or older, or were aged between 18 to 49 years old and considered clinically vulnerable. Although some of these criteria may produce a bias in terms of rates of hospital admission, given we only use to PANORAMIC trial to estimate the cost of hospital episodes any bias should be minimal. We further limited the eligible participants for the calculation of our health economic parameter estimated. This step comprised limiting to participants from England (85.3% of the trial population), participants who had received at least one dose of vaccination, and participants who were treated with either molnupiravir or usual care. We grouped participants into the following health states based on the reporting of any pre-existing medical conditions: Immunocompromised - participant-reported having a weakened immune system; Not immunocompromised but high risk - participant-reported lung disease, heart disease, diabetes, or obesity; Not at risk - participant reported not having any of the aforementioned medical conditions. ##### Cost calculations: Hospital and ICU admissions In our analysis we assumed the raw hospitalisation (or ICU costs) for vaccinated and unvaccinated individuals to be equal, with only the associated length of stay (*LoS**U* and *LoS**V*) varying. Hospital admission costs were a function of the age of the individual and their risk status: ![Formula][21] where *X* ∈ *{U, V }* and *QALY* is the monetary value associated with one QALY loss (either £20,000 for our central estimate or 30,000 when considering uncertainty). We had age and risk dependent per day cost *C**h* and loss of QALYS *Q**h* estimated from the PANORAMIC study (Fig. 5, left-hand column). ![Fig. 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/11/12/2024.11.08.24316972.1/F5.medium.gif) [Fig. 5:](http://medrxiv.org/content/early/2024/11/12/2024.11.08.24316972.1/F5) Fig. 5: Health episode costs and QALY losses (together with uncertainties) used in the health economic assessment. All data is from the PANORAMIC (Platform Adaptive trial of NOvel antiviRals for eArly treatMent of COVID-19 in the Community) trial [28]dataset of 2484 individuals with symptoms consistent with COVID-19. We estimated the healthcare costs per day from the raw data using a generalised linear model (GLM) with log-link. We estimated the hospitalisation and ICU QALY losses using beta regression. We estimated the QALY loss from premature death using risk-group specific increases in mortality and lower quality of life estimates [31] and a discounting of 3.5% per year. We colour code all the estimated values by risk group status: no-risk (green, left of each triplet group); at-risk but not immunosuppressed (purple, centre of each triplet group); and immunosuppressed (orange, right of each triplet group); fainter box-whisker plots are for age/risk-groups where there were no data and displayed results are extrapolations. For costs associated with ICU admission we made a similar calculation: ![Formula][22] with per day cost *C**I* and loss of QALYS *Q**I* again being age and risk dependent (Fig. 5, right-hand column). We took hospital admission unit costs from the NHS 2022/23 National Cost Collection Data Publication [29], then attached the appropriate Healthcare Resource Group (HRG) code to each patient to get their hospitalisation (or ICU) cost per episode. The cost of each patient was then divided by the number of days in the hospital (or ICU) to get the hospitalisation (or ICU) cost per day. We computed costs over six months, adjusted by trial allocation group and sex. We analysed adjusted values using a generalised linear model with log-link. We valued QALYs using EQ-5D-5L and UK utility values, derived using the approach recommended by NICE [30]. This approach applies a validated mapping function onto the UK EQ-5D-3L tariff set that has been developed by the NICE Decision Support Unit [30]. We adjusted QALYs lost by trial allocation group, sex and baseline EQ-5D-3L utility score. We analysed adjusted QALYs using beta-regression. For hospital and ICU admissions we factored in the uncertainty in the monetary cost and QALY-loss estimates. The uncertainty in QALY losses was especially substantial for those admitted to ICU (Fig. 5). ##### Cost calculations: Deaths For infection episodes resulting in mortality (Fig. 5, lower panel), we assumed no monetary costs and assumed that QALY losses (*Q**D*) depended on both age and risk group: ![Formula][23] The quality-adjusted life-year losses associated with COVID-19 deaths (*Q**D*(*a, r*)) were sourced from the literature. These accounted for QALY losses at various ages and adjusted for the presence of co-morbidities that influence both life-expectancy and health-related quality of life [31]. The standard life expectancy estimation approach, which focuses on conditional life expectancy upon reaching a specific age, were modified to incorporate: (i) the impact of comorbidities on life expectancy using the standardized mortality ratio (*SMR*); (ii) the effect of pre-existing comorbidities on health-related quality of life (*qCM*) to estimate QALYs over time; and (iii) discounting (*ε* = 0.035 throughout) [31]. The following parameters were assumed for different risk groups: for the immunocompromised group, defined as participants reporting a weakened immune system (*SMR* = 2; *qCM* = 0.8); for the not immunocompromised but high risk group—e.g., those reporting lung disease, heart disease, diabetes, or obesity (*SMR* = 1.5; *qCM* = 0.9); and for the low-risk group—participants without the mentioned medical conditions (*SMR* = 1; *qCM* = 1). Since these are arbitrary thresholds, we reviewed relevant literature to check their appropriateness and found that our assumptions generally aligned with established findings [32–34]. However, it is important to note that multimorbidity and immunosuppression exist on a spectrum concerning their effects on mortality and morbidity. Specifically, individuals with severe conditions like stroke or multiple chronic diseases may exhibit higher *SMR* and lower *qCM* values. Similarly, immunocompromised individuals often have additional comorbidities, suggesting that a lower *qCM* threshold could also be considered for this group. However, making slight adjustments to these parameters is unlikely to significantly alter the model outcomes. The final QALY losses associated with mortality for each age and risk group are shown in Fig. 5 (lower panel). ### 4.4 Public and Patient Involvement As part of the MEMVIE programme of research, which seeks to embed public and patient voices within vaccination modelling studies [35], we presented the models from this study to a standing group of six public contributors. We presented the model methodology and findings in an accessible narrative form by the modelling team, allowing the PPI group to question the assumptions, parameters and findings. Due to time constraints on the modelling team, in this instance the discussion took place after (rather than during) model development; however, the PPI discussions form part of ongoing work to further refine the MEMVIE Framework for Public Involvement in Mathematical and Economic Modelling, and to facilitate PPI implementation within vaccination modelling. ## Author contributions **Matt J. Keeling:** Conceptualisation, Formal analysis, Funding acquisition, Methodology, Software, Validation, Visualisation, Writing - Original Draft, Writing - Review & Editing. **Edward M. Hill:** Funding acquisition, Methodology, Visualisation, Writing - Original Draft, Writing - Review & Editing. **Stavros Petrou:** Funding acquisition, Methodology, Writing - Review & Editing. **Phuong Bich Tran:** Data curation, Formal analysis, Methodology, Writing - Review & Editing. **May Ee Png:** Data curation, Formal analysis, Methodology, Writing - Review & Editing. **Sophie Staniszewska:** Funding acquisition, Investigation, Writing - Review & Editing. **Corinna Clark:** Funding acquisition, Investigation, Writing - Review & Editing. **Katie Hassel:** Data curation, Investigation, Resources, Writing - Review & Editing. **Julia Stowe:** Data curation, Investigation, Resources, Writing - Review & Editing. **Nick Andrews:** Data curation, Investigation, Resources, Writing - Review & Editing. ## Financial disclosure This research (MJK, EMH, SP, PT, SS and CC) is funded by the National Institute for Health and Care Research (NIHR) Policy Research Programme (MEMVIE 3, NIHR204667). In addition: MJK was supported by the Engineering and Physical Sciences Research Council through the MathSys CDT [grant number EP/S022244/1] and by the Medical Research Council through the JUNIPER partnership award [grant number MR/X018598/1]. SP receives support as an NIHR senior investigator (NF-SI-0616-10103) and from the UK NIHR Applied Research Collaboration Oxford and Thames Valley. The PANORAMIC study was funded by the NIHR (NIHR135366). EMH is affiliated to the National Institute for Health and Care Research Health Protection Research Unit (NIHR HPRU) in Gastrointestinal Infections at University of Liverpool (PB-PG-NIHR-200910), in partnership with the UK Health Security Agency (UKHSA), in collaboration with the University of Warwick. EMH was funded by The Pandemic Institute, formed of seven founding partners: The University of Liverpool, Liverpool School of Tropical Medicine, Liverpool John Moores University, Liverpool City Council, Liverpool City Region Combined Authority, Liverpool University Hospital Foundation Trust, and Knowledge Quarter Liverpool (EMH is based at The University of Liverpool). The views expressed are those of the author(s) and not necessarily those of the NIHR, the Department of Health and Social Care, the UK Health Security Agency or The Pandemic Institute. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission. ## Data availability The raw study data are protected and are not available due to data privacy laws. This work is carried out under Regulation 3 of The Health Service (Control of Patient Information) (Secretary of State for Health, 2002) (3) using patient identification information without individual patient consent. Data cannot be made publicly available for ethical and legal reasons, i.e. public availability would compromise patient confidentiality as data tables list single counts of individuals rather than aggregated data. ## Governance UKHSA has legal permission, provided by Regulation 3 of The Health Service (Control of Patient Information) Regulations 2002, to process patient confidential information for national surveillance of communicable diseases and as such, individual patient consent is not required. ## Ethical approvals The UK Medicines and Healthcare products Regulatory Agency and the South Central - Berkshire Research Ethics Committee of the Health Research Authority approved the PANORAMIC trial (reference: 21/SC/0393). ## Competing interests All authors declare that they have no competing interests. ## Footnotes * Added to the manuscript an ethical approval statement for the PANORAMIC trial. * Received November 8, 2024. * Revision received November 12, 2024. * Accepted November 12, 2024. * © 2024, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## References 1. [1].Office for National Statistics. Coronavirus (COVID-19) latest insights: Infections - Total people infected (2023). URL [https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/articles/coronaviruscovid19latestinsights/infections#total-people-infected](https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/articles/coronaviruscovid19latestinsights/infections#total-people-infected). [Online] (Accessed: 08 November 2024). 2. [2].Office for National Statistics. Coronavirus (COVID-19) latest insights: Vaccines (2023). URL [https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/articles/coronaviruscovid19latestinsights/vaccines](https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/articles/coronaviruscovid19latestinsights/vaccines). [Online] (Accessed: 08 November 2024). 3. [3].NHS England. Vaccinations: COVID-19 (2024). URL [https://www.england.nhs.uk/statistics/statistical-work-areas/covid-19-vaccinations/](https://www.england.nhs.uk/statistics/statistical-work-areas/covid-19-vaccinations/). [Online] (Accessed: 08 November 2024). 4. [4].Moore S, Hill EM, Dyson L, Tildesley MJ, Keeling MJ. Modelling optimal vaccination strategy for SARS-CoV-2 in the UK. PLoS computational biology 17(5):e1008849 (2021). doi:10.1371/journal.pcbi.1008849. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pcbi.1008849&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33956791&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 5. [5].Moore S, Hill EM, Tildesley MJ, Dyson L, Keeling MJ. Vaccination and non-pharmaceutical interventions for COVID-19: a mathematical modelling study. Lancet Infect. Dis. 21(6):793–802 (2021). doi:10.1016/S1473-3099(21)00143-2. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1473-3099(21)00143-2&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33743847&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 6. [6].Hill EM, Keeling MJ. Comparison between one and two dose SARS-CoV-2 vaccine prioritization for a fixed number of vaccine doses. Journal of the Royal Society Interface 18(182):20210214 (2021). doi:10.1098/rsif.2021.0214. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1098/rsif.2021.0214&link_type=DOI) 7. [7].UK Health Security Agency. COVID-19 vaccine surveillance report: 18 July 2024 (week 29) (2024). URL [https://assets.publishing.service.gov.uk/media/669923b20808eaf43b50d1fd/Vaccine\_surveillance\_report\_2024\_week\_29.pdf](https://assets.publishing.service.gov.uk/media/669923b20808eaf43b50d1fd/Vaccine\_surveillance\_report_2024_week_29.pdf). [Online] (Accessed: 08 November 2024). 8. [8].Zeevat F, van der Pol S, Westra T, Beck E, Postma MJ, et al. Cost-effectiveness analysis of COVID-19 mRNA XBB. 1.5 Fall 2023 vaccination in the Netherlands. medRxiv page 2024.09.26.24314420 (2024). doi:10.1101/2024.09.26.24314420. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyNC4wOS4yNi4yNDMxNDQyMHYxIjtzOjQ6ImF0b20iO3M6NTI6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMTEvMTIvMjAyNC4xMS4wOC4yNDMxNjk3Mi4xLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 9. [9].NHS Digital. Secondary Uses Service (2024). URL [https://digital.nhs.uk/services/secondary-uses-service-sus](https://digital.nhs.uk/services/secondary-uses-service-sus). [Online] (Accessed: 08 November 2024). 10. [10].Office for National Satistics. Deaths broken down by age, sex, area and cause of death. (2024). URL [https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths](https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths). [Online] (Accessed: 08 November 2024). 11. [11].Tessier E, Edelstein M, Tsang C, Kirsebom F, Gower C, et al. Monitoring the covid-19 immunisation programme through a national immunisation management system – england’s experience. International Journal of Medical Informatics 170:104974 (2023). doi:10.1016/jijmedinf.2022.104974. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/jijmedinf.2022.104974&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=36577202&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 12. [12].JCVI. Code of Practice: Joint Committee on Vaccination and Immunisation; 2013 (2013). URL [https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/224864/JCVI\_Code\_of\_Practice\_revision\_2013\_-\_final.pdf](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/224864/JCVI\_Code\_of\_Practice\_revision_2013_-_final.pdf). [Online] (Accessed: 08 November 2024). 13. [13].Public Health Scotland. Viral Respiratory Diseases (including Influenza and COVID-19) in Scotland Surveillance Report (2024). URL [https://publichealthscotland.scot/media/29604/week-41-17-10-24-viral-respiratory-diseases-including-influenza-and-covid-19-in-spdf](https://publichealthscotland.scot/media/29604/week-41-17-10-24-viral-respiratory-diseases-including-influenza-and-covid-19-in-spdf). 14. [14].Kirsebom FC, Stowe J, Bernal JL, Allen A, Andrews N. Effectiveness of autumn 2023 COVID-19 vaccination and residual protection of prior doses against hospitalisation in England, estimated using a test-negative case-control study. Journal of Infection 89(1):106177 (2024). [PubMed](http://medrxiv.org/lookup/external-ref?access_num=38719110&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 15. [15].Feikin DR, Higdon MM, Andrews N, Collie S, Knoll MD, et al. Assessing covid-19 vaccine effectiveness against omicron subvariants: Report from a meeting of the world health organization. Vaccine 41(14):2329–2338 (2023). [PubMed](http://medrxiv.org/lookup/external-ref?access_num=36797097&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 16. [16].Kirsebom FCM, Andrews N, Stowe J, Ramsay M, Bernal JL. Duration of protection of ancestralstrain monovalent vaccines and effectiveness of bivalent ba. 1 boosters against covid-19 hospitalisation in england: a test-negative case-control study. The Lancet Infectious Diseases 23(11):1235– 1243 (2023). [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1473-3099(23)00365-1&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=37453440&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 17. [17].Kirsebom FCM, Harman K, Lunt RJ, Andrews N, Groves N, et al. Vaccine effectiveness against hospitalisation estimated using a test-negative case-control study design, and comparative odds of hospital admission and severe outcomes with COVID-19 sub-lineages BQ. 1, CH. 1.1. and XBB. 1.5 in England. The Lancet Regional Health–Europe 35 (2023). 18. [18].Davies NG, Klepac P, Liu Y, Prem K, Jit M, et al. Age-dependent effects in the transmission and control of COVID-19 epidemics. Nat. Med. 26(8):1205–1211 (2020). doi:10.1038/s41591-020-0962-9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41591-020-0962-9&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32546824&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 19. [19].Keeling MJ, Hill EM, Gorsich EE, Penman B, Guyver-Fletcher G, et al. Predictions of COVID-19 dynamics in the UK: Short-term forecasting and analysis of potential exit strategies. PLOS Comput. Biol. 17(1):e1008619 (2021). doi:10.1371/journal.pcbi.1008619. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pcbi.1008619&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33481773&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 20. [20].JCVI. JCVI statement on spring 2023 COVID-19 vaccinations, 22 February 2023 (2023). URL [https://www.gov.uk/government/publications/spring-2023-covid-19-vaccination-programme-jcvi-advice-22-february-2023/jcvi-statement-on-spring-2023-covid-19-vaccinations-22-february-2023](https://www.gov.uk/government/publications/spring-2023-covid-19-vaccination-programme-jcvi-advice-22-february-2023/jcvi-statement-on-spring-2023-covid-19-vaccinations-22-february-2023). [Online] (Accessed: 08 November 2024). 21. [21].JCVI. JCVI statement on the COVID-19 vaccination programme for autumn 2023, 26 May 2023 (2023). URL [https://www.gov.uk/government/publications/covid-19-autumn-2023-vaccination-programme-jcvi-advice-26-may-2023/jcvi-statement-on-the-covid-19-vaccination-programme-for-autumn-2023-26-may-2023](https://www.gov.uk/government/publications/covid-19-autumn-2023-vaccination-programme-jcvi-advice-26-may-2023/jcvi-statement-on-the-covid-19-vaccination-programme-for-autumn-2023-26-may-2023). [Online] (Accessed: 08 November 2024). 22. [22].UK Health Security Agency (Eds Ramsay, Mary). Immunisation against infectious diseases, Chapter 14a. The Stationery Office (2024). 23. [23].Williamson EJ, Walker AJ, Bhaskaran K, Bacon S, Bates C, et al. Factors associated with COVID-19-related death using OpenSAFELY. Nature 584(7821):430–436 (2020). [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41586-020-2521-4&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32640463&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 24. [24].Kohli MA, Maschio M, Joshi K, Lee A, Fust K, et al. The potential clinical impact and costeffectiveness of the updated COVID-19 mRNA fall 2023 vaccines in the United States. Journal of Medical Economics 26(1):1532–1545 (2023). doi:10.1080/13696998.2023.2281083. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/13696998.2023.2281083&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=37961887&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 25. [25].Mahasing C, Suphanchaimat R, Teekasap P, Nittayasoot N, Wongsanuphat S, et al. Cost effectiveness analysis comparing varying booster intervals of vaccination policies to address COVID-19 situation in Thailand, 2023. Plos one 19(9):e0310427 (2024). doi:10.1371/journal.pone.0310427. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0310427&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=39288199&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 26. [26].NHS Digital. Cohorting as a Service. (2024). URL [https://digital.nhs.uk/services/cohorting-as-a-service-caas](https://digital.nhs.uk/services/cohorting-as-a-service-caas). [Online] (Accessed: 08 November 2024). 27. [27].Keeling MJ, Dyson L. A retrospective assessment of forecasting the peak of the sars-cov-2 omicron ba. 1 wave in england. PLOS Computational Biology 20(9):e1012452 (2024). 28. [28].University of Oxford. Homepage - PANORAMIC trial (2022). URL [https://www.panoramictrial.org/homepage](https://www.panoramictrial.org/homepage). [Online] (Accessed: 08 November 2024). 29. [29].NHS England. 2022/23 National Cost Collection Data Publication (2024). URL [https://www.england.nhs.uk/publication/2022-23-national-cost-collection-data-publication/](https://www.england.nhs.uk/publication/2022-23-national-cost-collection-data-publication/). [Online] (Accessed: 08 November 2024). 30. [30].Png ME, Harris V, Grabey J, Hart ND, Jani BD, et al. Cost-utility analysis of molnupiravir for high-risk, community-based adults with covid-19: an economic evaluation of the panoramic trial. British Journal of General Practice 74(745):e570–e579 (2024). doi:10.3399/BJGP.2023.0444. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNzQvNzQ1L2U1NzAiO3M6NDoiYXRvbSI7czo1MjoiL21lZHJ4aXYvZWFybHkvMjAyNC8xMS8xMi8yMDI0LjExLjA4LjI0MzE2OTcyLjEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 31. [31].Briggs AH, Goldstein DA, Kirwin E, Meacock R, Pandya A, et al. Estimating (quality-adjusted) life-year losses associated with deaths: With application to COVID-19. Health Economics 30(3):699–707 (2021). [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/hec.4208&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33368853&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 32. [32].Jia H, Zack MM, Thompson WW. The effects of diabetes, hypertension, asthma, heart disease, and stroke on quality-adjusted life expectancy. Value in Health 16(1):140–147 (2013). [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23337225&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F11%2F12%2F2024.11.08.24316972.1.atom) 33. [33].Turtle L, Thorpe M, Drake TM, Swets M, Palmieri C, et al. Outcome of COVID-19 in hospitalised immunocompromised patients: An analysis of the WHO ISARIC CCP-UK prospective cohort study. PLoS Medicine 20(1):e1004086 (2023). 34. [34].Espanol T, Prevot J, Drabwell J, Sondhi S, Olding L. Improving current immunoglobulin therapy for patients with primary immunodeficiency: quality of life and views on treatment. Patient preference and adherence pages 621–629 (2014). 35. [35].Staniszewska S, Hill EM, Grant R, Grove P, Porter J, et al. Developing a framework for public involvement in mathematical and economic modelling: bringing new dynamism to vaccination policy recommendations. The Patient-Patient-Centered Outcomes Research 14:435–445 (2021). doi:10.1007/s40271-020-00476-x. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s40271-020-00476-x&link_type=DOI) [1]: /embed/graphic-9.gif [2]: /embed/inline-graphic-1.gif [3]: /embed/inline-graphic-2.gif [4]: /embed/inline-graphic-3.gif [5]: /embed/inline-graphic-4.gif [6]: /embed/graphic-10.gif [7]: /embed/inline-graphic-5.gif [8]: /embed/inline-graphic-6.gif [9]: /embed/graphic-11.gif [10]: /embed/graphic-12.gif [11]: /embed/graphic-13.gif [12]: /embed/inline-graphic-7.gif [13]: /embed/graphic-14.gif [14]: /embed/graphic-15.gif [15]: /embed/graphic-16.gif [16]: /embed/graphic-17.gif [17]: /embed/graphic-18.gif [18]: /embed/graphic-19.gif [19]: /embed/graphic-20.gif [20]: /embed/graphic-21.gif [21]: /embed/graphic-23.gif [22]: /embed/graphic-25.gif [23]: /embed/graphic-26.gif