Vaccine Clinical Trials: Efficacy and Immunogenicity Endpoints with Statistical Analysis Using SAS

How vaccine efficacy and immunogenicity are analysed in SAS: exact confidence intervals for vaccine efficacy, Poisson regression, geometric mean titres, seroconversion and non-inferiority, with dummy data, annotated code, output and interpretation.

Vaccine trials differ from most drug trials in one basic way: the participants are healthy, and the aim is to prevent a disease rather than treat it. This leads to endpoints and statistical methods that are specific to vaccines. Two questions dominate: does the vaccine prevent disease (efficacy), and does it produce an adequate immune response (immunogenicity)?

This article explains both types of endpoint and works through each on a dummy dataset: the statistical concepts, the SAS code, the output, and the interpretation.

Endpoints in vaccine trials

EndpointWhat it measuresTypical use
Vaccine efficacy (VE)Relative reduction in disease incidence in vaccinated compared with unvaccinated participantsPrimary endpoint of large Phase III efficacy trials
Geometric mean titre (GMT)Average antibody level after vaccinationPhase I/II, lot consistency, immunobridging
Seroconversion or seroresponse rateProportion of participants with a pre-defined rise in antibody, commonly four-fold or moreCo-primary endpoint with GMT in immunogenicity trials
Reactogenicity and safetySolicited local and systemic reactions, unsolicited adverse eventsAll phases

An efficacy trial is needed when no accepted immune marker of protection exists. Once efficacy has been shown, later studies (new age groups, new lots, updated strains) often rely on immunobridging: showing that the immune response is non-inferior to that seen in the population where efficacy was demonstrated.

Statistical concepts behind the analysis

1. Vaccine efficacy

Vaccine efficacy is defined as VE = (1 − RR) × 100%, where RR is the ratio of the disease incidence rate in the vaccine arm to that in the placebo arm. A VE of 85% means vaccinated participants had 85% fewer cases than they would have had without the vaccine.

2. Person-time and incidence rates

Participants are followed for different lengths of time, so the incidence rate is calculated as cases divided by person-years at risk. Follow-up for the primary analysis usually starts at a pre-defined point, for example 14 days after the final dose, when the vaccine is expected to be protective.

3. Case-driven design

The precision of VE depends on the total number of cases, not on the number of participants. Efficacy trials are therefore designed to continue until a target number of confirmed cases has accrued, and thousands of participants are enrolled because only a small proportion become cases.

4. The exact conditional method

Given the total number of cases, the number that fall in the vaccine arm follows a binomial distribution. If θ is the proportion of all cases that occur in the vaccine arm, then RR = θ / (1 − θ) × (person-years placebo / person-years vaccine). An exact Clopper-Pearson confidence interval for θ is converted directly into a confidence interval for VE. This method needs no large-sample approximation, which matters because the number of cases in the vaccine arm is often very small.

5. Success criteria for efficacy

A statistically significant result (lower confidence limit above 0%) is not enough. Regulators expect the lower limit to exceed a pre-specified threshold that rules out a weakly effective vaccine. For example, FDA guidance for COVID-19 vaccines asked for a point estimate of at least 50% and a lower limit above 30%.

6. Poisson regression

A Poisson model with the log of person-years as an offset estimates the log rate ratio and allows adjustment for covariates such as age group or region. The exponentiated coefficient is the rate ratio, and VE is one minus that value.

7. Antibody titres are log-normal

Antibody titres are strongly right-skewed and span several orders of magnitude. They are analysed after log transformation (base 10 or base 2). The mean of the log values, back-transformed, is the geometric mean titre. The difference between two groups on the log scale, back-transformed, is the GMT ratio.

8. Values below the limit of quantitation

Assays cannot measure very low titres. Values below the lower limit of quantitation (LLOQ) are commonly replaced with half the LLOQ before analysis. The rule must be pre-specified because it affects baseline GMTs and fold-rise calculations.

9. Seroconversion and fold rise

Seroconversion is usually defined as at least a four-fold rise in titre from baseline. The geometric mean fold rise (GMFR) summarises the average rise. Seroconversion rates are reported with exact (Clopper-Pearson) confidence intervals because they are often close to 100%, where approximate intervals perform poorly.

10. Non-inferiority for immunobridging

Immunobridging compares a test group with a reference group using pre-specified non-inferiority margins. Commonly used criteria are: the lower limit of the 95% confidence interval for the GMT ratio (test/reference) is above 0.67, and the lower limit of the 95% confidence interval for the difference in seroconversion rates (test minus reference) is above −10 percentage points. The Miettinen-Nurminen score method is the usual choice for the confidence interval of the difference.

Part A: Vaccine efficacy

Dummy data

A randomised, placebo-controlled trial enrolled 6,000 participants (1:1). Cases are confirmed symptomatic disease with onset at least 14 days after the final dose. The values are simulated and are for illustration only.

data ve;
   length trt $8;
   input trt $ trtn n cases py;
   logpy = log(py);                  /* offset for Poisson model */
   datalines;
Vaccine 1 3000  7 1482.6
Placebo 0 3000 48 1469.3
;
run;

Step A1: Vaccine efficacy with an exact conditional confidence interval

The BETAINV function returns the Clopper-Pearson limits for θ, which are then converted to limits for VE. Note that the upper limit of θ gives the lower limit of VE.

proc sql noprint;
   select cases, py into :xv, :tv from ve where trtn = 1;
   select cases, py into :xp, :tp from ve where trtn = 0;
quit;

data ve_exact;
   xv = &xv;   tv = &tv;            /* vaccine: cases, person-years */
   xp = &xp;   tp = &tp;            /* placebo: cases, person-years */
   alpha = 0.05;

   n     = xv + xp;                  /* total cases                  */
   r     = tv / tp;                  /* person-time ratio            */
   theta = xv / n;                   /* share of cases on vaccine    */

   /* Clopper-Pearson limits for theta */
   if xv > 0 then theta_l = betainv(alpha/2, xv, n - xv + 1);
   else           theta_l = 0;
   if xv < n then theta_u = betainv(1 - alpha/2, xv + 1, n - xv);
   else           theta_u = 1;

   inc_v  = 100 * xv / tv;           /* incidence per 100 person-years */
   inc_p  = 100 * xp / tp;

   ve     = 100 * (1 - theta   / (r * (1 - theta)));
   ve_lcl = 100 * (1 - theta_u / (r * (1 - theta_u)));
   ve_ucl = 100 * (1 - theta_l / (r * (1 - theta_l)));

   length success $3;
   if ve ge 50 and ve_lcl gt 30 then success = 'Yes';
   else                              success = 'No';
run;

proc print data=ve_exact noobs;
   var xv xp inc_v inc_p ve ve_lcl ve_ucl success;
   format inc_v inc_p ve ve_lcl ve_ucl 8.2;
run;

Step A2: Poisson regression

proc genmod data=ve;
   class trt (ref='Placebo') / param=ref;
   model cases = trt / dist=poisson link=log offset=logpy;
   estimate 'Vaccine vs Placebo' trt 1 / exp;
run;

Output: vaccine efficacy

TreatmentNCasesPerson-yearsIncidence per 100 person-years
Vaccine300071482.60.47
Placebo3000481469.33.27
MethodRate ratioVaccine efficacy (%)95% CI for VE (%)
Exact conditional (primary)0.144585.5567.88 to 94.48
Poisson regression (Wald)0.144585.5568.06 to 93.46

Exact method: θ = 7/55 = 0.1273 with Clopper-Pearson limits 0.0527 to 0.2448. Poisson model: estimate = −1.9343, standard error = 0.4046, Wald chi-square = 22.86, p < 0.0001; rate ratio 0.1445 (95% CI 0.0654 to 0.3194).

Interpretation: vaccine efficacy

  1. Incidence. Disease occurred at 0.47 per 100 person-years in the vaccine arm and 3.27 per 100 person-years in the placebo arm.
  2. Vaccine efficacy. VE is 85.55%: the vaccine reduced the rate of disease by about 86%.
  3. Confidence interval. The exact 95% confidence interval is 67.88% to 94.48%. The lower limit is well above 30% and the point estimate is above 50%, so the pre-specified success criterion is met.
  4. Agreement between methods. The Poisson model gives the same point estimate and a similar interval. The exact interval is slightly wider and is preferred as the primary analysis because only 7 cases occurred in the vaccine arm.
  5. Precision comes from cases. Although 6,000 participants were enrolled, the result rests on 55 cases. This is why efficacy trials are event-driven.

Part B: Immunogenicity and immunobridging

Dummy data

An immunobridging study compares neutralising antibody titres 28 days after the final dose in a test group (for example adolescents) with a reference group (for example young adults in whom efficacy was shown), 30 participants each. The assay LLOQ is 10, and values below it have been set to 5. The values are simulated.

VariableMeaning
USUBJIDUnique subject identifier
GRPNGroup: 1 = Reference, 2 = Test
BASETitre at baseline
AVALTitre at Day 29
data adis;
   length usubjid $8 grp $10 scfl $1;
   input usubjid $ grpn base aval;

   if grpn = 1 then grp = 'Reference';
   else             grp = 'Test';

   log10aval = log10(aval);                /* log10 titre            */
   foldrise  = aval / base;                /* fold rise from baseline */
   log10fr   = log10(foldrise);

   if foldrise ge 4 then scfl = 'Y';       /* seroconversion          */
   else                  scfl = 'N';

   /* Sort keys so PROC FREQ compares Test - Reference for 'Y' */
   grpord = 3 - grpn;                      /* 1 = Test, 2 = Reference */
   scord  = ifn(scfl = 'Y', 1, 2);         /* 1 = Yes,  2 = No        */

   datalines;
IM-001 1   5   906
IM-002 2  15   564
IM-003 1  10   416
IM-004 2  12    35
IM-005 1  12  1350
IM-006 2  16   820
IM-007 1  14   527
IM-008 2   5    95
IM-009 1  17   933
IM-010 2   5   688
IM-011 1  17   273
IM-012 2  18   672
IM-013 1  11   430
IM-014 2   5   479
IM-015 1   5   434
IM-016 2   5   917
IM-017 1   5   593
IM-018 2  11   517
IM-019 1   5   795
IM-020 2  17   876
IM-021 1   5   802
IM-022 2   5  1198
IM-023 1  12   472
IM-024 2   5   889
IM-025 1   5  1289
IM-026 2  18   543
IM-027 1  18  1111
IM-028 2   5  2138
IM-029 1  21   371
IM-030 2  23  1302
IM-031 1  12   182
IM-032 2  20  1037
IM-033 1   5  1298
IM-034 2   5  1498
IM-035 1  15   348
IM-036 2   5   623
IM-037 1   5  1506
IM-038 2  17   357
IM-039 1   5  1116
IM-040 2  11   575
IM-041 1   5  1044
IM-042 2   5   816
IM-043 1   5   381
IM-044 2   5   494
IM-045 1   5   591
IM-046 2   5   948
IM-047 1   5   246
IM-048 2   5   735
IM-049 1  10   766
IM-050 2  26  1026
IM-051 1   5     9
IM-052 2  11  1464
IM-053 1  14   274
IM-054 2   5   407
IM-055 1  22    67
IM-056 2   5  1061
IM-057 1  11   936
IM-058 2  13   804
IM-059 1  12  1024
IM-060 2  17  1125
;
run;

Step B1: Geometric mean titre and fold rise by group

proc means data=adis noprint nway;
   class grpn grp;
   var log10aval log10fr;
   output out=gm n(log10aval)=n
                 mean(log10aval)=mean lclm(log10aval)=lcl uclm(log10aval)=ucl
                 mean(log10fr)=mean_fr;
run;

data gmt;
   set gm;
   gmt     = 10**mean;            /* back-transform to titre scale */
   gmt_lcl = 10**lcl;
   gmt_ucl = 10**ucl;
   gmfr    = 10**mean_fr;
   keep grp n gmt gmt_lcl gmt_ucl gmfr;
run;

proc print data=gmt noobs;
   format gmt gmt_lcl gmt_ucl gmfr 8.2;
run;

Step B2: GMT ratio and non-inferiority

ods output Estimates=gmr_est;

proc glm data=adis;
   class grpn;
   model log10aval = grpn / clparm alpha=0.05;
   estimate 'Test vs Reference' grpn -1 1;
run;
quit;

data gmr;
   set gmr_est;
   gmr     = 10**estimate;        /* GMT ratio, Test / Reference   */
   gmr_lcl = 10**lowercl;
   gmr_ucl = 10**uppercl;

   length ni $8;
   if gmr_lcl gt 0.67 then ni = 'Met';
   else                    ni = 'Not met';
run;

proc print data=gmr noobs;
   var gmr gmr_lcl gmr_ucl ni;
   format gmr gmr_lcl gmr_ucl 8.3;
run;

Step B3: Seroconversion rates and their difference

/* Exact (Clopper-Pearson) CI for each group */
proc sort data=adis;  by grpn grp;  run;

proc freq data=adis;
   by grpn grp;
   tables scfl / binomial(level='Y' cl=exact);
run;

/* Difference Test - Reference with Miettinen-Nurminen CI */
proc freq data=adis;
   tables grpord*scord / riskdiff(cl=mn column=1) nocol nopercent;
run;

Output: immunogenicity

GroupNBaseline GMTDay 29 GMT95% CI for GMTGMFR
Reference308.60503.84342.54 to 741.0858.59
Test308.89674.65501.57 to 907.4575.92
ComparisonDifference in log10 titreStandard errorGMT ratio95% CINon-inferiority (lower limit above 0.67)
Test vs Reference0.12680.10331.3390.832 to 2.156Met
GroupSeroconverted, n/NRate (%)Exact 95% CI (%)
Reference28/3093.377.9 to 99.2
Test29/3096.782.8 to 99.9

Difference in seroconversion rates (Test − Reference) = 3.3 percentage points; Miettinen-Nurminen 95% CI −11.1 to 18.7.

Interpretation: immunogenicity

  1. Immune response. Both groups started with very low titres (baseline GMT about 9) and responded strongly. The Day 29 GMT was 674.65 in the test group and 503.84 in the reference group, a rise of roughly 76-fold and 59-fold.
  2. GMT ratio. The ratio is 1.339 with a 95% confidence interval of 0.832 to 2.156. The lower limit is above 0.67, so non-inferiority of the test group is shown for GMT. The interval includes 1, so there is no evidence that the test group responds better; the claim is only that it is not meaningfully worse.
  3. Seroconversion. 96.7% of the test group and 93.3% of the reference group seroconverted, a difference of 3.3 percentage points.
  4. Seroconversion non-inferiority. The lower confidence limit for the difference is −11.1 percentage points, which is below the −10 margin. Non-inferiority for seroconversion is not formally shown, even though the observed rate is higher in the test group.
  5. Overall conclusion. In this dummy example the GMT criterion is met and the seroconversion criterion is narrowly missed. If both were co-primary, the study would not have met its objective.

Why the seroconversion criterion fails with a better observed rate. With 30 participants per group, a single participant changes the rate by 3.3 percentage points, so the confidence interval for the difference is almost 30 points wide. The margin cannot be met reliably at this size. Real immunobridging studies typically enrol several hundred participants per group for exactly this reason, and sample size must be calculated for both co-primary endpoints.

Practical points

  • Analysis populations. Efficacy and immunogenicity are usually analysed in a per-protocol population (participants who received all doses as planned with no major deviations), with a supportive analysis in all randomised participants.
  • Case definition. Define confirmed cases, the surveillance period and adjudication before unblinding. VE is very sensitive to which cases are counted.
  • Baseline serostatus. Participants with evidence of prior infection at baseline are commonly excluded from the primary analysis or analysed separately.
  • Interim analyses. Event-driven trials often include interim analyses. The confidence level of the final analysis must be adjusted to preserve the overall type I error.
  • Adjusting GMT comparisons. When baseline titres differ between groups, use ANCOVA with the log baseline titre as a covariate.
  • Multiple strains or serotypes. Multivalent vaccines require the criteria to be met for each component, which increases the sample size needed.

References

  • US FDA. Development and Licensure of Vaccines to Prevent COVID-19, Guidance for Industry. 2020.
  • US FDA. Clinical Data Needed to Support the Licensure of Seasonal Inactivated Influenza Vaccines, Guidance for Industry. 2007.
  • World Health Organization. Guidelines on clinical evaluation of vaccines: regulatory expectations. WHO Technical Report Series No. 1004, Annex 9. 2017.
  • Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 1934;26(4):404–413.
  • Miettinen O, Nurminen M. Comparative analysis of two rates. Statistics in Medicine. 1985;4(2):213–226.

The datasets in this article are simulated and are provided for training purposes only.

Found this useful?Share it with your team or network.

Need help applying this to your study?

Our statisticians, programmers and quality specialists can review it with you.