Abstract
Background Phase III trials have estimated COVID-19 vaccine efficacy (VE) against symptomatic and asymptomatic infection. We explore the direction and magnitude of potential biases in these estimates and their implications for vaccine protection against infection and against disease in breakthrough infections.
Methods We developed a mathematical model that accounts for natural and vaccine-induced immunity, changes in serostatus and imperfect sensitivity and specificity of tests for infection and antibodies. We estimated expected biases in VE against symptomatic, asymptomatic and any SARS-CoV-2 infections and against disease following infection for a range of vaccine characteristics and measurement approaches, and the likely overall biases for published trial results that included asymptomatic infections.
Results VE against asymptomatic infection measured by PCR or serology is expected to be low or negative for vaccines that prevent disease but not infection. VE against any infection is overestimated when asymptomatic infections are less likely to be detected than symptomatic infections and the vaccine protects against symptom development. A competing bias towards underestimation arises for estimates based on tests with imperfect specificity, especially when testing is performed frequently. Our model indicates considerable uncertainty in Oxford-AstraZeneca ChAdOx1 and Janssen Ad26.COV2.S VE against any infection, with slightly higher than published, bias-adjusted values of 59.0% (95% uncertainty interval [UI] 38.4 to 77.1) and 70.9% (95% UI 49.8 to 80.7) respectively.
Conclusion Multiple biases are likely to influence COVID-19 VE estimates, potentially explaining the observed difference between ChAdOx1 and Ad26.COV2.S vaccines. These biases should be considered when interpreting both efficacy and effectiveness study results.
Introduction
The coronavirus disease (COVID-19) phase III vaccine trials have demonstrated efficacy against symptomatic infection for multiple vaccine candidates, with estimates ranging from 50% to 95% (1– 3). Yet a vaccine that protects against symptomatic disease may work either by preventing infection, (an infection-blocking vaccine), by preventing the progression to symptoms upon infection (a disease-blocking vaccine), or by a combination of these two mechanisms (Supplementary Figure 1) (4). Understanding the extent to which the COVID-19 vaccines protect against infection is important because the success of each vaccination programme is highly contingent not only on symptomatic cases, but also asymptomatic infection and community transmission (5).
The predominant primary outcome of the COVID-19 vaccine trials is vaccine efficacy against the first case of PCR-confirmed symptomatic disease, VEsym. This is measured by PCR-testing trial participants with COVID-19 symptoms, and is sensitive to the clinical case definition (6). As important secondary outcomes, most trials also measure the incidence of asymptomatic infections, using either i) regular swabbing and PCR-testing or ii) serological testing for anti-nucleocapsid antibodies at pre-specified time intervals (Table 1). The latter allows seroconversion after infection to be identified for trials of vaccines based on the spike, but not inactivated vaccines that include the whole virion (although for inactivated vaccines, it may be possible to infer infection via a rise in SARS-CoV-2 antibodies following a period after vaccination). This allows for vaccine efficacy against asymptomatic infection (VEasym) and vaccine efficacy against any infection (VEin) to be estimated.
VEasym is a complex outcome due to its relationship with the two mechanisms of vaccine efficacy. A vaccine that protects only against infection will reduce the number of symptomatic and asymptomatic infections in equal proportions, leading to a positive VEasym. Yet a vaccine that protects against progression to symptoms will convert symptomatic cases to asymptomatic, potentially giving a negative VEasym. The counterintuitive interpretation of this outcome has been noted (4,7), but the relationship between VEasym, VEin and vaccine efficacy against progression to symptoms (VEpr) has not been quantified. To aid the interpretation of VEasym, it is important that the relationship between this outcome and the two main mechanisms of vaccine efficacy is described.
Estimates of vaccine efficacy are known to be biased by factors such as imperfect test sensitivity and specificity, undetected asymptomatic infections, and the accumulation of immunity over time (8–10). However thus far, there has been little discussion on the accuracy and potential biases of the COVID-19 phase III trial estimates (11,12). We developed a conceptual framework and mathematical model of a vaccine trial to investigate the factors affecting observed values of vaccine efficacy. We first illustrate the factors that influence measured VEasym and derive its relationship with VEin and VEpr. We then quantify the influence of different biases on vaccine efficacy estimates, notably the impact of i) the build-up of immunity from undetected asymptomatic infections, ii) imperfect test sensitivity and specificity for alternative testing strategies, iii) differential detection of asymptomatic and symptomatic infections, and iv) confounding of vaccine efficacy and probability of symptoms by age. We finish by applying our model to published COVID-19 trial results, to estimate the potential bias in their estimates.
Methods
Analytical derivations
Vaccine efficacy (VE) is defined as 1-RR where RR is some measure of the relative risk of the outcome in the vaccine compared with the control arm (13). For most primary and secondary outcomes of the COVID-19 vaccine trials, the relative risk is based on an incidence rate ratio (IRR) such that
where IRv and IRc are the incidence rate in the vaccine and control groups respectively.
For outcomes measured at fixed time points, such as the number of seroconversions to a protein that is not a component of the vaccine, the relative risk can be calculated using the cumulative incidence ratio (CIR) such that
where CIv and CIc are the cumulative incidence in the vaccine and control groups respectively. For a partially protective vaccine, vaccine efficacy based on cumulative incidence approximates that based on the incidence rate for low incidence or short follow-up periods, but biases towards zero as follow-up time and incidence increase (10).
For vaccines that protect against infection and/or progression to symptoms, vaccine efficacy against symptomatic infection is given by
(14). Vaccine efficacy against asymptomatic infection depends on the incidence of asymptomatic infections that are not prevented by the vaccine and on infections that would have become symptomatic, but the vaccine prevents from progressing, such that
where ps is the probability of symptom development in the absence of vaccination. Substituting equation (1) into (2) allows VEin to be derived as a simple function of VEsym and VEasym
Rearranging equation (1) and substituting equation (3) into equation (1) then provides a solution for VEpr
Estimation of confidence intervals for VEin and VEpr is provided in Supplementary Methods. If asymptomatic infections are less likely to be detected than symptomatic infections, and a vaccine is protective against progression to symptoms (VEpr > 0), then observed VEin ≠ true VEin. In this case, the observed VEin depends on the relative incidence of detected infections, and can be related to the true efficacy by
where pd represents the relative probability of asymptomatic to symptomatic infection detection.
Analytical solutions become more complex when incorporating additional biases, such as those resulting from imperfect test sensitivity and specificity and the acquisition of natural immunity, so we developed a stochastic mathematical (cohort) model of a phase III vaccine trial.
Mathematical model
The model follows a susceptible, infected, recovered (SIR) structure, implemented as a Markov model, and allows for asymptomatic and symptomatic infections, natural immunity, changes in measured serostatus and imperfect test sensitivity and specificity. The background infection rate, defined as the rate at which a susceptible unvaccinated person becomes infected, is assumed to be constant over time. We assume the “leaky vaccine” model, so for vaccinated participants the background infection rate is reduced by the VEin. The probability of unvaccinated participants developing symptoms upon infection is defined by ps, while for vaccinated participants ps is reduced by the VEpr. We assume no heterogeneity in population characteristics but perform a sensitivity analysis to assess the effect of variation in ps and vaccine efficacy by age.
We model two testing approaches for asymptomatic infections, i) weekly PCR testing and ii) serological testing at 1, 2, 6, 12 and 24 months after baseline. We assume that responsive PCR testing detects all symptomatic infections. Observed vaccine efficacy is calculated from the simulated incidence of detected infections in each trial arm. Efficacy is estimated as 1-IRR for all outcomes except those estimated using serology, for which efficacy is estimated as 1-CIR, using the cumulative number of seroconversions detected in each serology assessment up to the present time interval. Point estimates and confidence intervals are given by the mean and 2.5 and 97.5 percentiles of 1000 simulated estimates.
Application to COVID-19
Applying the model to COVID-19, we assumed a natural probability of developing symptoms upon infection of 0.67 (15), a serology test specificity of 99.84% (16) and sensitivity of 95% and 80% to symptomatic and asymptomatic infections, respectively (17). We used data on the probability of PCR detection over time since infection for individuals without symptoms (18) to estimate the probability of detecting an asymptomatic infection with weekly PCR swabbing (Supplementary Table 1), and we assumed a PCR test specificity of 99.945% (19). We used the model to estimate bias-adjusted vaccine efficacy estimates for two adenovirus vector vaccines that have published trial data on asymptomatic infection, ChAdOx1 (Oxford-AstraZeneca) and Ad26.COV2.S (Janssen), and generated 95% uncertainty intervals (UI) using Latin Hypercube Sampling. We then used rank regression to evaluate the contribution of individual parameters to the biases (Supplementary Methods).
The model is described further in Supplementary Methods and illustrated in Supplementary Figures 2 and 3. The model parameters are provided in Supplementary Table 2.
Results
Interpretation of vaccine efficacy against asymptomatic infection
Observed VEasym was dependent on VEin, VEpr and the proportion of infections that were symptomatic (Figure 1). For vaccines that worked only by blocking infection, VEasym was equal to VEsym. For vaccines with efficacy predominantly mediated by prevention of symptom development, VEasym was low or negative, particularly when a large proportion of infections were naturally symptomatic. VEin and VEpr could be estimated from reported values of VEsym and VEasym using Equations 3 and 4 (Supplementary Figure 4).
Red to blue gradient represents the transition from a disease-blocking vaccine (VEin = 0% and VEpr = VEsym) to an infection-blocking vaccine (VEin = VEsym and VEpr = 0%). Each successive line represents a 10% increase in VEin, and a corresponding decrease in VEpr (such that VEsym remains the same). Lines show predicted values from Equation 2, points and error bars show the observed mean and 2.5 and 97.5 percentiles from 1000 simulations, with efficacy calculated as 1-IRR, censoring after the first infection. A) Vaccine with 70% efficacy against symptomatic infection (e.g., Oxford-AstraZeneca/Janssen vaccines). B) Vaccine with 90% efficacy against symptomatic infection (e.g., Pfizer/Moderna vaccines). VEin= vaccine efficacy against infection, VEpr= vaccine efficacy against progression to symptoms, VEsym= vaccine efficacy against symptomatic infection, IRR = incidence rate ratio.
Possible biases in the COVID-19 vaccine trials
The build-up of immunity from undetected asymptomatic infections caused VEsym to bias in opposite directions for infection-blocking and disease-blocking vaccines (Figure 2). For an infection-blocking vaccine, estimated VEsym decreased over time, with greater decreases observed for higher forces of infection and lower probabilities of symptoms. For a disease-blocking vaccine, a downward bias was only observed when the probability of symptoms was low. Instead, for most combinations of parameters, estimated VEsym increased slightly over time. Further sensitivity analysis showed that the biases were sensitive to the vaccine efficacy calculation (Supplementary Figure 5).
Points and error bars show the observed mean and 2.5 and 97.5 percentiles from 1000 simulations, with efficacy calculated as 1-IRR, censoring after the first symptomatic infection. Time = 0 months represents 2 weeks post second dose. Sensitivity and specificity = 100%. True vaccine efficacies: vaccine efficacy against symptomatic infection (VEsym) = 70%, A) vaccine efficacy against infection (VEin) = 70%, vaccine efficacy against progression to symptoms (VEpr) = 0%, B) VEin = 50%, VEpr = 40%, C) VEin = 0%, VEpr = 70%. IRR = incidence rate ratio.
Imperfect test sensitivity and specificity biased efficacy estimates towards zero. Factors increasing the magnitude of the bias were: reduced specificity, reduced sensitivity, increased testing frequency, and calculation with the CIR instead of the IRR. Although the serology estimated VEin was based on the CIR (as person-time at risk is unknown), the bias was usually lower than the weekly-PCR estimate, for a given sensitivity and specificity, due to the lower frequency of testing (Figure 3).
A) Low force of infection (5% per year), B) High force of infection (30% per year). 6-month follow-up visit: Serology tests taken at month 1, 2 and 6 (cumulative seroconversions up to 6-month visit); PCR tests taken weekly. Serology efficacy calculated using 1-CIR; PCR efficacy calculated using 1-IRR. Sensitivity assumed to be equal for symptomatic and asymptomatic infections. Points and error bars represent the mean and 2.5 and 97.5 percentiles from 1000 simulations. VEin= vaccine efficacy against infection, CIR = cumulative incidence ratio, IRR = incidence rate ratio.
For a vaccine that was protective against symptom development, VEin was overestimated when asymptomatic infections were less likely to be detected than symptomatic infections (Figure 4). The greater the difference in the probability of detection and the greater the vaccine’s protection against symptoms, the greater the overestimation.
Lines show values estimated with Equation 4, points and error bars show the observed mean and 2.5 and 97.5 percentiles from 1000 simulations, with efficacy calculated as 1-IRR, censoring after the first infection. Specificity = 100%, yearly force of infection = 5%, follow up = 12 months from 2 weeks post 2nd dose. VEin= vaccine efficacy against infection, VEpr= vaccine efficacy against progression to symptoms, IRR = incidence rate ratio.
These results were insensitive to adding age stratification to the probability of symptoms. However, also adding age-stratification to vaccine efficacy led to biased estimates of VEsym and VEasym, when not adjusted for age (e.g. in Poisson regression; Figure 5). When vaccine efficacy decreased with age and the probability of symptoms increased, VEasym was overestimated and VEsym underestimated. The magnitude of the difference was greater with an increased association between age and the probability of symptoms, and between age and efficacy.
Points and error bars represent the mean and 2.5 and 97.5 percentiles of 1000 simulations. True vaccine efficacy against asymptomatic infection (VEasym) = true vaccine efficacy against infection (VEin) = true vaccine efficacy against symptomatic infection (VEsym) = 70%. Specificity = 100%, yearly force of infection = 20%, follow up = 12 months from 2 weeks post 2nd dose.
Estimating VEin, VEpr, and the likely bias from the published trial results
Applying Equations 3 and 4 to the reported trial results gave an estimated VEpr for ChAdOx1 of 43.6% (95% CI 20.6 to 59.9) (Table 2). For Ad26.COV2.S, VEin was estimated at 66.2% (95% CI 55.9 to 74.1) and VEpr just 0.9% (95% CI -46.8 to 33.2).
Estimated vaccine efficacy against infection (VEin) and vaccine efficacy against progression to symptoms (VEpr) using trial-reported vaccine efficacy against symptomatic infection (VEsym) and vaccine efficacy against asymptomatic infection (VEasym). Trials that have not yet reported an estimate are left blank. VEin and VEpr calculated using Equations 3 and 4, respectively.
Incorporating the aforementioned biases, the model gave best estimates for ChAdOx1 VEin, VEasym and VEpr of 59.0% (95% UI 38.4 to 77.1), 27.4% (95% UI -32.1 to 87.4) and 31.5% (95% UI -20.7 to 55.0) respectively. While for Ad26.COV2.S, the corresponding bias-adjusted estimates were 70.9% (95% CI 49.8 to 80.7), 79.2% (95% CI 14.6 to 99.0) and -15.2% (95% CI -73.3 to 33.2). The rank regression analyses showed that the biases were most sensitive to the test specificity, baseline hazard of infection and adherence to weekly PCR testing (Supplementary Tables 3-5).
Discussion
Accurately estimating COVID-19 vaccine efficacy against infection and disease is important to understand vaccine benefits, their likely impact on transmission and the long-term prospects for disease control. Estimating the magnitude and direction of biases in a simulated COVID-19 vaccine trial helps to understand their likely influence, and may help to explain differences seen between vaccines, trials, and populations.
We firstly derived the relationship between VEasym with efficacy against infection and against disease in breakthrough infections. While increasing VEin increased VEasym, increasing VEpr had the opposite effect because more infections were prevented from becoming symptomatic. This influence of VEpr was stronger when a greater proportion of infections were symptomatic in the absence of vaccination. Therefore, although counterintuitive, for COVID-19 where a minority of infections present asymptomatically and the vaccines have high efficacies against symptomatic infection, protection against infection can be expected even when VEasym is low or negative.
Secondly, we estimated that the ChAdOx1 weekly PCR-measured VEin was underestimated by 8.1% (Trial 50.9%, Model 59.0%) and VEasym by 12.8% (Trial 14.6%, Model 27.4%). The VEpr calculated from the trial reported VEin and VEsym would therefore be an overestimation (Calculation 43.6%, Model 31.5%). However, a wide range of values are compatible with the reported trial results when considering stochastic variation and parameter uncertainty. The true VEin may range between 38.4% and 77.1%, and VEpr between -20.7% and 55.0%. Given the strong bias towards zero that can be caused by reduced test specificity and a high frequency of testing, it would not be unreasonable for the true VEin to be closer to our upper uncertainty interval. For Ad26.COV2.S, our model suggests that the true VEin lies between 49.8% and 80.7%, with a best estimate of 70.9%. Although this indicates a negative VEpr, we believe this is unlikely and rather explained by the small sample size informing the trial reported VEasym estimate.
We explain these overall expected differences by four biases that are likely to act in the COVID-19 trials.
A lower probability of detecting asymptomatic infections relative to symptomatic infections leads to overestimation of VEin if the vaccine protects against progression to symptoms. For these vaccines (with VEpr > 0) some infections will be prevented from causing symptoms, so will be less likely to be detected. VEpr would be mistaken for VEin, so VEin would be overestimated. Both conditions for this bias are likely to be satisfied in the COVID-19 trials, as multiple studies have indicated that virological and serological testing approaches are less sensitive to asymptomatic infections (20,21). This bias is likely to have influenced the ChAdOx1 VEin estimate, however we expect it was overridden by a competing downwards bias.
Imperfect test sensitivity and specificity bias estimates towards zero, with greater bias with higher frequency of testing, lower force of infection and when vaccine efficacy is calculated using cumulative incidence rather than incidence rates. This is caused by a build-up of false positives in both trial arms, and is greater when the ratio of false positives to true positives is greater (8). Regression analysis showed that this was the predominant factor leading to underestimation for both ChAdOx1 and Ad26.COV2.S in our model. As the bias is greater when testing is frequent, even a test with high specificity could bias the estimated ChAdOx1 VEin and VEasym noticeably towards zero. This could therefore explain such contrasting trial reported VEasym estimates between ChAdOx1 and Ad26.COV2.S, despite their similar platforms and neutralising antibody responses (22,23).
A build-up of natural immunity from undetected asymptomatic infections contributes a small downwards bias in VEsym for infection-blocking vaccines and a small upwards bias for disease-blocking vaccines. For an infection-blocking vaccine, the proportion of infections that are asymptomatic is unaltered by the vaccine. Therefore, the rate at which immunity from asymptomatic infections accumulates is equivalent across trial arms, leading to an underestimation of VEsym (10,24– 26). Yet for disease-blocking vaccines, a greater proportion of infections in the vaccine arm will be asymptomatic, accelerating the acquisition of immunity from undetected infections and introducing a conflicting upward bias. Our model and real-world effectiveness studies suggest the COVID-19 vaccines protect against both infection and symptoms, to varying degrees (27,28). Therefore we expect the overall direction of this bias to be towards zero, and for its magnitude to be greater for vaccines with higher VEin (e.g., Pfizer-BioNTech BNT162b2 (29)).
Decreasing vaccine efficacy by age will bias estimated VEsym downwards and VEasym upwards, unless adjusted for age. This is due to older participants contributing more to VEsym estimates than younger participants, who contribute more to VEasym estimates. This bias is dependent on the probability of symptoms increasing with age, for which there is mixed evidence (30–32). However, it should be considered when interpreting estimates based on different subgroups, such as if VEasym is estimated from a subgroup with serological data while VEsym is based on the total population.
These biases also apply to effectiveness studies, based on cohort or case-control designs. Most notably, the bias arising from differential detection of asymptomatic and symptomatic infections will likely be greater in real-world studies, where asymptomatic testing is less rigorous. This should be considered when comparing real-world and trial reported estimates, as it could lead to greater bias towards overestimation of VEin in effectiveness studies, if the vaccine prevents symptom development.
There are limitations to our analysis, notably uncertainties over parameter estimates. There is limited evidence on both serology and PCR test sensitivities for asymptomatic infections, and how these change over time since infection. As we show, differences in test sensitivity by symptom status can lead to overestimation of VEin, so further studies are needed to clarify the potential role of this bias. We also did not consider the vaccines’ effects on viral load and how this alters virological and serological test sensitivity. Multiple COVID-19 vaccines have been shown to reduce SARS-CoV-2 viral load (33,34), and lower load infections are less likely to lead to seroconversion (35). Therefore serology-based efficacy estimates may be more representative of high viral load infections than all infections. They may be comparable to estimates based on DNA sequenced swabs, as these samples must exceed a threshold viral load to be sequenced. Finally, we do not consider the use of point prevalence estimates from single time point PCR swabs, however this has been explored elsewhere (11,12).
In conclusion, multiple biases have the potential to influence the COVID-19 vaccine efficacy estimates, with their direction and magnitude dependent on the vaccine properties and testing strategies. These biases may explain differences between the ChAdOx1 and Ad26.COV2.S trial reported estimates despite similar vaccine platform technologies, and should be considered when interpreting both efficacy and effectiveness study results as these begin to be reported for these and other vaccines.
References
