Economic evaluation. Methods for adjusting survival estimates in the presence of treatment crossover: a simulation study.

Methods for adjusting survival estimates in the presence of treatment crossover: a simulation studyNicholas Latimer, University of SheffieldCollaborators: Paul Lambert, Keith Abrams, Michael Crowther, Allan Wailoo and James Morden

Contents

What is the treatment crossover problemPotential solutionsSimulation studyResultsConclusions

Treatment switching – the problem

In RCTs often patients are allowed to switch from the control treatment to the new intervention after a certain timepoint (eg disease progression)

OS estimates will be confounded

OS is a key input into the QALY calculation

Cost effectiveness results will be inaccurate an ITT analysis is likely to underestimate the treatment benefit

Inconsistent and inappropriate treatment recommendations could be made

Treatment switching – the problem

Survival time

Control Treatment

Intervention

Control Intervention

PFS

PFS

PFS

PPS

PPS

PPS

True OS difference

RCT OS differenc

e

Switching is likely to result in an underestimate of the treatment effect

What is usually done to adjust?No clear consensus

Numerous ‘naive’ approaches have been taken in NICE appraisals, eg:

Take no action at allExclude or censor all patients who crossover

Occasionally more complex statistical methods have been used, eg:Rank Preserving Structural Failure Time Models (RPSFTM)Inverse Probability of Censoring Weights (IPCW)

And others are available from the literature, eg:Structural Nested Models (SNM)

Very prone to selection bias – crossover isn’t random

What are the consequences?

TA 215, Pazopanib for RCC [51% of control switched]

ITT: OS HR (vs IFN) = 1.26 ICER = Dominated

Censor patients: HR = 0.80 ICER = £71,648

Exclude patients: HR = 0.48 ICER = £26,293

IPCW: HR = 0.80 ICER = £72,274

RPSFTM: HR = 0.63 ICER = £38,925

Potential solutions

RPSFTM (and IPE algorithm)Randomisation-based approach, estimates counterfactual survival times

Key assumption: common treatment effect

IPCWObservational-based approach, censors xo patients, weights remaining patients

Key assumptions: “no unmeasured confounders”; must model OS and crossover

SNMObservational version of RPSFTM

Key assumptions: “no unmeasured confounders”; must model OS and crossover

Simulation study (1)None of these methods are perfectBut we need to know which are likely to produce least bias in different scenarios

Simulation study

Simulate survival data for two treatment groups, applying crossover that is linked to patient characteristics/prognosisIn some scenarios simulate a treatment effect that changes over timeIn some scenarios simulate a treatment effect that remains constant over timeTest different %s of crossover, and different treatment effect sizes

How does the bias and coverage associated with each method compare?

Simulation study (2)Methods assessed

Naive methodsITTExclude crossover patientsCensor crossover patientsTreatment as a time-dependent covariate

Complex methodsRPSFTMIPE algorithmIPCWSNM

Results: common treatment effectRPSFTM / IPE worked very well

IPCW and SNM performed ok when crossover % was lower

IPCW and SNM performed poorly when crossover % was very high

Naive methods performed poorly (generally led to higher bias than ITT)

-10.00

0.00

10.00

20.00

30.00

40.00

50.00

3 4 7 8 15 16 19 20 27 28 31 32 39 40 43 44 51 52 55 56 63 64 67 68

AUC

mea

n bi

as (%

)

ITT IPCW RPSFTM IPE SNM

Results: effect 15% in xo patientsRPSFTM / IPE produced higher bias than previous scenarios

IPCW and SNM performed similarly to RPSFTM / IPE providing crossover < 90%

IPCW and SNM performed poorly when crossover % was very high

Bias not always lower than that associated with the ITT analysis

-15.00

-5.00

5.00

15.00

25.00

35.00

45.00

1 2 5 6 13 14 17 18 25 26 29 30 37 38 41 42 49 50 53 54 61 62 65 66

AUC

mea

n bi

as (%

)


RPSFTM / IPE produced even more bias

IPCW and SNM produce less bias than RPSFTM / IPE providing crossover < 90%

No ‘good’ options when crossover % is very high

Often ITT analysis likely to result in least bias (esp. when trt effect low)

-20.00

-10.00

0.00

10.00

20.00

30.00

40.00

50.00

9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48 57 58 59 60 69 70 71 72

AUC

mea

n bi

as (%

)


Results: effect 25% in xo patients

ConclusionsTreatment crossover is an important issue that has come to the fore in HE arena

Current methods for dealing with treatment crossover are imperfect

Our study offers evidence on bias in different scenarios (subject to limitations)

RPSFTM / IPE produce low bias when treatment effect is common But are very sensitive to this

IPCW / SNM are not affected by changes in treatment effect between groups, but in (relatively) small trial datasets observational methods are volatile Especially when crossover % is very high (leaving low n in control group)

Very important to assess trial data, crossover mechanism, treatment effect to determine which method likely to be most appropriate

Don’t just pick one!!

Back-up Slides

Data Generation (1)Used a two-stage Weibull model to generate underlying survival times and a time-dependent covariate (called ‘CEA’)Longitudinal model for CEA (for ith patient at time t):

where is the random intercept is the slope for a patient in the control arm is the slope for a patient in the treatment arm (all is the change in the intercept for a patient with bad prognosis

compared to a patient without bad prognosis Picked parameter values such that CEA increased over time, more slowly in the experimental group, and was higher in the badprog group

Data Generation (2)The survival hazard function was based upon a Weibull (see Bender et al 2005):

In our case,

where is the log hazard ratio (the baseline treatment effect) is the time-dependent change in the treatment effect

is the impact of a bad prognosis baseline covariate on survival

is the coefficient of CEA, indicating its effect on survivalWe used this to generate our survival times

So, CEA has an effect on survival over time, and there is a separate time-dependent treatment effect

) (cea(t)+badprog+trt*) log(t)*(+trt*= i2ii1 X1

2

)exp()( 1 Xtth

Data Generation (3)We applied a reduced treatment effect to crossover patients – this was equivalent to the baseline treatment effect multiplied by a factor to ensure that this effect was less than (or equal to) the average effect received in the experimental group

Data Generation (4)

We then selected parameter values in order that ‘realistic’ datasets were created:

0.00

0.20

0.40

0.60

0.80

1.00

Sur

viva

l pro

babi

lity

249 186 126 71 42 21 0trtrand = 1251 154 86 52 25 9 0trtrand = 0

Number at risk

0 200 400 600 800 1000 1200analysis time

trtrand = 0 trtrand = 1

0

0.5

1

1.5

2

2.5

0 200 400 600 800 1000 1200

Acc

eler

ation

Fac

tor

Time (days)

AF actual

AF predicted

Data Generation (5)We made several assumptions about the ‘crossover mechanism’:

1. Crossover could only occur after disease progression (disease progression was approximately half of OS, calculated for each patient using a beta(5,5) distribution)

2. Crossover could only occur at 3 ‘consultations’ following disease progression

These were set at 21 day intervalsProbability of crossover highest at initial consultation, then falls in second and third

3. Crossover probability depended on time-dependent covariates:CEA value at progression (high value reduced chance of crossover)Time to disease progression (high value increased chance of crossover)This was altered in scenarios to test a simpler mechanism where probability only depended on CEA

Given all this, CEA was a time-dependent confounder

ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group

Log hazard ratio = 0.5

Survival time distribution Alter parameters to test two levels of disease severity

Initially assumed treatment effect Alter to test two levels of treatment effect Time-dependence of treatment effect Treatment effect received is equal in all crossover

patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios

Probability of switching treatment over time

Test two levels of treatment crossover proportions

Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over




(2)









(2)

Initially assumed treatment effect Alter to test two levels of treatment effect (4)Time-dependence of treatment effect Treatment effect received is equal in all crossover








(2)



(8)

(12)







(2)



(8)

(12)


Test two levels of treatment crossover proportions (24)





(2)



(8)

(12)


Test two levels of treatment crossover proportions (24)


(72)

This combined to 72 scenarios









Estimating AUC

1. ‘Survivor function’ approachApply treatment effect to survivor function (or hazard function) estimated for experimental group calculate AUC

2. ‘Extrapolation’ approachExtrapolate counterfactual dataset to required time-point (only relevant for RPSFTM/IPE approaches) calculate AUC

3. ‘Shrinkage’ approachUse estimated acceleration factor to ‘shrink’ survival times in crossover patients in order to obtain an adjusted dataset calculate AUC (only relevent for AF-based approaches)

28A topical analogy...

VsEngland

(control group)Spain

(intervention group)

Kick-off...


Halftime: England 0 – 3 Spain

Spain are statistically significantly better than England For ethical reasons, the Spanish team are cloned and the English team are sent home. So the second half is Spain Vs Spain


Switching is likely to result in an underestimate of the true supremacy of the intervention

Full time: England 2 – 5 Spain

Spain still win, but not as comfortably as they would have

09/04/2023© The University of Sheffield

31

Results (4)While the SNM and IPCW methods appeared to produce similar levels of bias, the SNM approach was more volatile:

Relatively often failed to converge (in up to 90% of sims in one scenario)Typically when disease severity was high and treatment effect was low

Only produced lower bias than the ITT analysis in 38% of scenarios (compared to 60% for the IPCW method)

Was more sensitive to increasing crossover %


32

Results (5)Relationship between bias and treatment crossover %

-40

-20

0

20

40

60

80

100

120

140

60% 70% 80% 90% 100%

Mea

n %

bia

s

Crossover proportion (at-risk patients)

ITT


33


-40

-20

0

20

40

60

80

100

120

140

60% 70% 80% 90% 100%

Mea

n %

bia

s


IPE

RPSFTM

ITT


34


-40

-20

0

20

40

60

80

100

120

140

60% 70% 80% 90% 100%

Mea

n %

bia

s


IPCW

IPE

RPSFTM

ITT


35


-40

-20

0

20

40

60

80

100

120

140

60% 70% 80% 90% 100%

Mea

n %

bia

s


IPCW

IPE

RPSFTM

SNM

ITT

36ConclusionsTreatment crossover is an important issue that has come to the fore in HE arena

Current methods for dealing with treatment crossover are imperfect

Our study offers evidence on bias in different scenarios (subject to limitations)

1. RPSFTM / IPE produce low bias when treatment effect is common

2. When treatment effect ≈ 15% lower in crossover patients RPSFTM / IPE and IPCW / SNM methods produce similar levels of bias (5-10%) [provided suitable data available for obs methods and <90% crossover]

3. When treatment effect ≈ 25% lower IPCW/SNM produce less bias than RPSFTM/IPE[provided suitable data available for obs methods and <90% crossover][and significant bias likely to remain ITT analysis may offer least bias]

4. Very important to assess trial data, crossover mechanism, treatment effect to determine which method likely to be most appropriate

Don’t just pick one!!

Economic evaluation. Methods for adjusting survival estimates in the presence of treatment crossover: a simulation study.

Health & Medicine