“Leaky” vaccines are those for which vaccine-induced protection reduces infection rates on a per-exposure basis, as opposed to “all-or-none” vaccines, which reduce infection rates to zero for some fraction of subjects, independent of the number of exposures. Leaky vaccines therefore protect subjects with fewer exposures at a higher effective rate than subjects with more exposures. This simple observation has serious implications for analysis methodologies that rely on the assumption that the vaccine effect is homogeneous across subjects. We argue and show through examples that this heterogeneous vaccine effect leads to a violation of the proportional hazards assumption, to incomparability of infected cases across treatment groups, and to nonindependence of the distributions of the competing failure processes in a competing risks setting. We discuss implications for vaccine efficacy estimation, correlates of protection analysis, and mark-specific efficacy analysis (also known as sieve analysis).

Public health vaccines have reduced the global burden of disease considerably over the past century. Statistical design and analysis of vaccine efficacy trials are well-studied and critical components of the development of these interventions. As discussed in [

Vaccine efficacy has multiple definitions (see [

If each subject experiences exactly one exposure during the trial, then a leaky vaccine reducing susceptibility by 50% has the same attack-rate efficacy as an all-or-none vaccine that fully protects 50% of the subjects. Here we focus on examples such as HIV-1 vaccine trials, in which multiple exposures are possible and in which some (or many) participants will experience no exposures at all. In such settings, the effect of a partially efficacious leaky vaccine is to reduce attack rates for subjects who experience one exposure more than for subjects who experience multiple exposures, since each exposure has an independent opportunity to infect. Although in this setting reinfection is possible, we assume that the endpoint of interest is initial infection only, so that infected subjects are removed from the at-risk population.

In this paper, we consider the analysis of leaky vaccines when there is heterogeneity in subjects’ infecting exposure distributions (defined as either heterogeneity in exposure or in per-exposure infection susceptibility or both). Through arguments and simulation, [

In this section, we introduce the notation and examples that we will use to demonstrate the implications of risk heterogeneity for evaluating the efficacy of leaky vaccines. We assume a setting of a well-conducted placebo-controlled randomized clinical trial to evaluate a vaccine intervention, where the effect of the intervention is to reduce the per-exposure infection probability by a (multiplicative) factor

As shown in [

We assume that we do not observe the exposure processes at all; we are given data of the form of per-subject pairs

We define three distinct notions of vaccine efficacy, based on different quantities. First we define the attack rate for treatment group

We assume for simplicity of presentation that there are two risk groups. We allow that some fraction

For illustration, we suppose arbitrarily that the baseline hazard function is constant, as in the exponential model. Under this model, an infection event occurs in a low-risk placebo recipient at a time-constant rate

The fraction

A leaky vaccine with multiplicative vaccine effect

This differential attack-rate efficacy by risk group results in a different proportion of high-risk participants among infected subjects at the end of the trial across the two treatment groups. To see this, consider that if the beginning-of-trial probability of being high risk is

For partially efficacious vaccines with

This correspondingly results in a different proportion of high-risk participants among subjects remaining at-risk at the end of the trial across the two treatment groups. The posterior odds of being high risk among those remaining uninfected are

If

We may still have

In this section we have shown that, for leaky vaccines, subject heterogeneity in risk results in time variation of

Next, we turn to implications of these observations. First, we show, as has been shown previously, that

The differential efficacy for high-risk and low-risk subjects has the effect of inducing a violation of the proportional hazards assumption for the marginal hazards, even if it holds separately for the low-risk hazards and the high-risk hazards. Each marginal hazard function is a mixture of the two risk-group hazards, and the mixing proportion changes over time differently for placebo recipients than for vaccine recipients as the at-risk frequencies diverge due to different rates of infection in the two risk groups.

The marginal hazard rate of infection is a mixture over high- and low-risk subjects. At the beginning of the trial the marginal hazard for placebo recipients is

For vaccine recipients, there is also a change in the marginal hazard over the course of the trial, but the change is different than for placebo recipients. At the beginning of the trial the marginal hazard for vaccine recipients is

If the study enrolls

Unless the end-of-trial rates of high-risk subjects among the uninfected are the same for both treatment groups (i.e., unless

We demonstrate the situation with a simple example of a leaky vaccine with about

If the study enrolls 100 vaccine recipients, 5 of whom are high risk, then a 2% infection rate among low-risk vaccine recipients (and a corresponding 20% among the high-risk vaccinees) over the course of the trial yields a difference in the mixture of high : low risk hazards from 5 : 95 (

If that study also enrolls 100 placebo recipients, 5 of whom are high risk, then a 4% infection rate among low-risk placebo recipients and a corresponding 36% among the high-risk placebos over the course of the trial yields a difference in the mixture of high : low risk hazards from 5 : 95 at the beginning to 3 : 91 (about

For the conditions of our example, the hazard ratio at the beginning of the trial (vaccine/placebo) is

The hazard proportion also changes over the duration of the trial; Figure

Effects of differential enrichment for high-risk subjects in the at-risk population across treatment groups. (a) The marginal hazards as a function of time for placebo recipients (dashed blue line) and vaccine recipients (solid red line) for the conditions of our example trial in which a 1 : 1 randomization allocates subjects to receive a placebo or a leaky vaccine with per-exposure efficacy

The differential enrichment of high-risk subjects among those infected across treatment groups implies that even if a vaccine has an equal per-exposure effect on every subject, its effects on overall attack rates are expected to differ by risk group. When evaluating a vaccine candidate to determine if its partial efficacy can be attributed to unequal vaccine effects across subjects (by for instance identifying preexisting subject traits or immune responses to vaccination that differentiate subjects for whom the vaccine worked best), care must be taken to differentiate between these expected attack-rate effects (which do not reflect differential per-exposure efficacy by subject trait) from effects that truly modify the per-exposure efficacy by subject trait.

Several authors have noted that the analysis of vaccine trials to identify subject correlates of

Here we point out that leaky vaccines with heterogeneous subject risk constitute a concrete example of this difficulty. Since we expect differential enrichment of high-risk subjects even when the vaccine has an equal per-exposure effect, then any correlate of infection risk in the placebo group will necessarily correlate with

This suggests a test for any putative candidate correlate of

This reasoning also warrants caution about so-called “case-only” methods, which evaluate only the infected cases. Such methods can be cost saving because correlates need not be measured in uninfected subjects. However if there is enrichment of different risk groups among infected subjects in the two treatment groups (which should be expected for any leaky vaccine), then covariate differences across treatment groups among infected subjects may simply reflect differential baseline risk. Since correlate information is unavailable for uninfected subjects, in case-only analyses the test of placebo-recipient risk correlation is not possible. Below we examine a special case of case-only analysis in the setting of competing risks, known as “sieve analysis.”

In addition to evaluating vaccine efficacy as a function of subject-specific covariates, it is often of interest to evaluate the extent to which a vaccine’s efficacy differs by type of infection. In a series of papers on what has variously been called “mark-specific intervention efficacy” or “sieve effects,” Gilbert et al. defined sufficient conditions under which estimates are unbiased for quantities relevant to the identification of these effects [

A sieve effect is defined as any violation of equivalence of

Thus, a sieve effect is defined as a lack of equivalence across all types

In Appendix

In Appendix

In Appendix

From these arguments we conclude that existing methods for evaluating hypotheses of sieve effects of leaky vaccines are expected to be biased if there is any subject heterogeneity in risk (or in response to the treatment), or if replacement failures are imbalanced. Gilbert has evaluated bias under violations of Assumption 2 in simulation studies [

Particular caution is warranted when using case-only sieve analysis methods as introduced in [

It remains likely that these methods, though not proven unbiased, retain their power to detect sieve effects under the heterogeneous risk conditions that we have considered. Although the conditions of those proofs may not hold under heterogeneity, we have not proven the contrary assertion; other proofs that establish conditions under which unbiased estimation is robust to subject variation in risk may yet be devised. Also, in practice absolute unbiasedness may not be required; with further work evaluating the practical implications of these insights, we expect that these methods will be approximately unbiased for many or most applications to leaky vaccines with heterogeneous risk. It remains to future work to conduct a thorough evaluation of the loss of power or the potential anticonservatism of analyses that assume risk homogeneity when the assumption is not justified.

In this paper we have restated the argument that when conducting statistical analysis of vaccine efficacy trials with heterogeneous exposure or susceptibility risk, care should be taken to account for the putative mechanism of the vaccine. Two extremes of the spectrum of vaccine mechanisms are considered. At one extreme (all-or-none), a vaccine protects some fraction of subjects completely and the remaining fraction are unaffected by it. At the opposite extreme (leaky), a vaccine reduces the per-exposure transmission rate for all recipients equally. We have shown that leaky vaccines induce a violation of the proportional hazards condition that is often assumed in survival analysis, due to a changing fraction of at-risk subjects over time in both vaccinated and unvaccinated individuals. Since these fractions change over time differently in the two treatment groups, even if the proportional hazards condition holds for each risk group individually, the marginal hazard ratio changes over time.

Another effect of subject risk heterogeneity in leaky vaccine trials is that the relative proportions of the risk groups among infected subjects changes over time. We showed that associations between subject covariates and vaccine efficacy will be biased unless those covariates are distributed equivalently in all risk groups. A simple diagnostic analysis of the risk of infection among placebo recipients as a function of the covariate could be used to reject the hypothesis of independence that is required for interpreting correlations with vaccine efficacy as indicative of differential efficacy rather than differential baseline risk, but this is not possible in a “case-only” analysis (which evaluates the association only among infected subjects). This argument cautions against case-only analysis of correlates of the partial efficacy of leaky vaccines when there is subject heterogeneity in risk.

We also addressed the context of competing risks and showed that leaky vaccines with risk heterogeneity will induce time variation in the relative proportion of marks (types of the competing risks) of infections and that since this time variation occurs at different rates in the vaccine and placebo groups, this induces a violation of the equivalence between observable relative attack rates and unobservable per-exposure relative risks that is required for unbiased analysis of mark-specific vaccine efficacies (called “sieve effects” when they differ across types) [

Longini and Halloran [

Recent work has introduced sieve analysis methods for nonleaky vaccines, which have all-or-none style protection but perhaps against only a subset of risk mark categories (in which case they are called “some-or-none” vaccines) [

The arguments in this paper together imply that it is generally not possible to differentiate between mark-specific efficacy and subject-covariate-specific efficacy using failure time and failure type data alone unless subject risk is homogeneous. The only exception is when risk groups (though heterogeneous in overall failure rate) have homogeneous relative rates of the marks of infecting exposures across competing risk mark types. Future work is needed to develop statistical analysis methods that account for both subject heterogeneity (as in a frailty model) and competing risks such that the effects of each can be differentiated in an analysis of a partially efficacious vaccine. Such approaches would likely require parameterization not just of the frailty model but also of the exposure processes, requiring considerable modeling effort and sensitivity analysis.

In practice we are usually unable to observe exposure events (as noted in [

This is distinct from the “prospective” (or “joint attack rate”) mark type distribution

Both the retrospective and prospective probabilities are distinct from the per-exposure probabilities that we intend to estimate. Gilbert et al. showed in [

We note that the first term on the right hand side should include the condition

Since the per-exposure failure probability is assumed to be independent of exposure history, we can directly define the marginal failure hazard by dropping the condition from the right-hand side:

We define the first term of this corrected failure hazard as the “exposure pseudohazard”:

Conceptually, if exposures are occurring that do not result in failures (we call these “avoided failures”), then the subject may nevertheless fail, but later than she would have otherwise. Since lower per-exposure failure probabilities result in more avoided failures, the time-to-event distribution among those who fail in the treated group will be right-shifted compared to what it would have been in the untreated group. We note that it may not be right-shifted for a particular failure type, but aggregating over all types, lower per-exposure failure rates will result in later expected failure times. Mathematically, this dependence between the probability of nonfailure by time

By Bayes’ theorem,

In Appendix

Here we repeat the proof, given in [

Then the prospective probabilities can be written in terms of the exposure pseudohazards: as

Then if we define the integrated type

This proof relies on the proportional pseudohazards condition to enable the factorization that separates the integrated exposure pseudohazard

Finally, this result guarantees equivalence of retrospective, prospective, and per-exposure odds ratios, since

The result of Appendix

Proportional exposure pseudohazards could also be accomplished if we assume that each subject experiences at most one failure during

Gilbert noted that conditions of the proof require a “leaky” vaccine in which all subjects experience the same intervention effect, and he explored violations of this assumption through simulations [

For the example with two risk groups, the exposure pseudohazard for vaccine recipients becomes

Since

Gilbert argued in [

We now show that any amount of subject heterogeneity in the intervention effect will lead to a violation of the noninformative censoring assumption, except under the null hypothesis of no sieve effect. We have shown that infection time

So it remains to show that failure mark type

The noninformative censoring assumption will never hold if there are sieve effects, but even under the null hypothesis of no sieve effects the assumption will only hold in the extreme case in which the (placebo-recipient) infecting exposure processes for all marks are equidistributed within each risk group. As long as some mark exposures occur at higher rates than others, then the attack-rate vaccine effect will differ against the different types, leading to a violation of the noninformative censoring condition.

The author declares that there is no conflict of interests regarding the publication of this paper.

The author thanks Betz Halloran, James Dai, Peter Gilbert, and Jason Shao for inspiration and feedback. The author retains all blame for errors or inaccuracies. This research was supported by NIH NIAID Grant 2 R37 AI054165-11. The opinions expressed in this paper are those of the author and do not represent the official views of the NIAID.