Are we really including all relevant evidence

Are we really including all relevant evidence? Considerations for

network meta-analysis

Beth Woods

Centre for Health Economics, University of York

with thanks to: David Scott, Neil Hawkins

What do we need for economic evaluation?

• A set of consistent relative treatment effects on the endpoint(s) of interest for all comparators of interest

• Network meta-analysis (NMA) or indirect comparison is typically used to estimate these effects

Including all relevant evidence

• To improve precision and avoid bias

• Demarcation of relevance i.e. what forms the relevant evidence space?

– Study set (comparators only)

– Endpoint space

Include Decision Comparators

• Three decision comparators (DC1, DC2, DC3)

• Direct comparisons only

Expand to Synthesis Comparators

• Second-order comparisons

• Third-order comparisons

All synthesis comparators here are secondary comparators

How far should we go?

• Connected network

• Increased precision

• Increased heterogeneity and inconsistency

– Scope for addressing these issues may increase

• Ability of decision makers to scrutinise NMA

Iterative searching

• At design stage knowledge of network may be limited – Value of a wider network unclear

– Can’t search for third-order comparisons unless you know the secondary comparators

• Iterative searching (Hawkins et al 2009) – Searching sequentially for primary comparators,

then secondary etc.

– Each iteration uncovers all direct, second-order, etc. comparisons

Endpoint space

Endpoint of interest

Statistic 1

Time point 1 Time point 2 Time point n

Statistic 2

Time point 1 Time point 2 Time point n

Statistic n

Time point 1 Time point 2 Time point n

Endpoint space

Endpoint space

Endpoint of interest

Statistic 1

Time point 1 Time point 2 Time point n

Statistic 2

Time point 1 Time point 2 Time point n

Statistic n

Time point 1 Time point 2 Time point n

Example (Woods et al 2010)

• Treatments for chronic obstructive pulmonary disease (COPD)

• Trial endpoints historically focused on measures of lung function and exacerbations

• Interest in the potential treatment effect on mortality

Available statistics

• Majority of trials reported mortality data only in adverse event reporting – Binary data

• New trials started analysing mortality endpoint using standard survival analysis methods – hazard ratios from Cox proportional hazards model

– includes additional information on time to event and censoring

• Challenge – incorporate all evidence, selecting preferred statistic where reported

BUGS shared parameter model

• Separate “loops” required to incorporate hazard ratio and binary data

• Treatment effect estimates (β’s) are the “shared parameters”

• Fixed effects model shown

Model for hazard ratio data (two arm trials)

• Normal likelihood 𝑥 𝑠,𝑘,𝑏 ~ 𝑁(ln (ℎ𝑟𝑠,𝑘,𝑏), 𝑠𝑒𝑠,𝑘,𝑏)

• Treatment effect model ln ℎ𝑟𝑠,𝑘,𝑏 = β𝑘 − β𝑏

β1 = 0

Model for hazard ratio data (multi-arm trials)

• Correlation in contrast data ln ℎ𝑟𝑠,𝑘,𝑏 = ln (ℎ𝑠,𝑘)−ln (ℎ𝑠,𝑏)

• Could model data as multivariate normal

• Or convert contrast data to arm level data

Converting contrast to arm-level data

• Set ℎ𝑠,𝑏 = 0

• Compute 𝑠𝑒(ℎ𝑠,𝑏) – Use fact that 𝑣𝑎𝑟(ℎ𝑟𝑠,𝑘,𝑏) = 𝑣𝑎𝑟 ℎ𝑠,𝑘 + 𝑣𝑎𝑟(ℎ𝑠,𝑏) – If have variance for hazard ratios comparing 2 vs. 1, 3

vs. 1 and 2 vs. 3 then can solve for 𝑣𝑎𝑟(ℎ𝑠,𝑏)

– If not assume common 𝑠𝑑(ℎ𝑠,𝑖)

• Normal likelihood 𝑥 𝑠,𝑘 ~ (ln (ℎ𝑠,𝑘), 𝑠𝑒𝑠,𝑘)

• Treatment effect model ln ℎ𝑠,𝑘 = 𝛼𝑠 + β𝑘 − β𝑏

Model for binary data

• Binomial likelihood 𝑟𝑠,𝑘~𝐵𝑖𝑛(𝐹𝑠,𝑘 , 𝑛𝑠,𝑘)

• Derive arm-specific log cumulative hazard

ln 𝐻𝑠,𝑘 = ln (− ln 1 − 𝐹𝑠,𝑘 )

• If we assume proportional hazards the ratio of cumulative hazards must equal the ratio of instantaneous hazards

• Treatment effect model ln 𝐻𝑠,𝑘 = 𝛼𝑠 + 𝛽𝑘 − 𝛽𝑏

Data # Data set descriptors

list(LnObs = 5, BnObs = 2, nTx = 4, nStudies = 3)

# Log hazard ratio data

# Binary data

Lstudy[] Ltx[] Lbase[] Lmean[] Lse[] multi[]

1 1 1 0 0.066 1

1 2 1 0.055 0.063 1

1 3 1 -0.154 0.070 1

1 4 1 -0.209 0.072 1

2 2 1 -0.276 0.203 0

Bstudy[] Btx[] Bbase[] Br[] Bn[]

3 3 1 1 229

3 1 1 1 227

BUGS code - model

#For hazard ratio reporting studies

for(ii in 1:LnObs ){

Lmean[ii] ~ dnorm(Lmu[ii],Lprec[ii])

Lprec[ii] < - 1/pow(Lse[ii],2)

Lmu[ii] < - alpha[Lstudy[ii]]*multi[ii] + beta[Ltx[ii]] - beta[Lbase[ii]] }

#For binary data reporting studies

for(ss in 1:BnObs){

Br[ss] ~ dbin(cumFail[ss], Bn[ss])

cumFail[ss] < - 1-exp(-1*exp(logCumHaz[ss]))

logCumHaz[ss] < - alpha[Bstudy[ss]] + beta[Btx[ss]] - beta[Bbase[ss]] }

Shared parameters

Shared parameter models

• Important when different statistics are reported for a specific endpoint – Median, mean and % event free at 21/28 days

(Welton et al 2008)

– Mean events per week, patients achieving ≥1 event by week 10

– Mean values of continuous outcome X, proportion achieving ≥X*

• Additional assumptions required

• Bayesian approach required?

Endpoint space

Endpoint of interest

Statistic 1

Time point 1 Time point 2 Time point n

Statistic 2

Time point 1 Time point 2 Time point n

Statistic n

Time point 1 Time point 2 Time point n

Why include repeated measures?

• Relative treatment effects for a study s and comparison k,b can be informed by data at more than one time point

• Precision and bias

• Explore temporal changes in treatment effects

– Within trial

– To inform extrapolation

Existing work

• Within-subject correlation generally ignored (e.g. Dakin et al 2011, Ding & Fu 2013)

• Models for time-varying treatment effects

– Treatment-specific time-by-treatment interactions

– Unrelated piecewise (e.g. Dakin et al 2011) or explicit functional form (e.g. Ding & Fu 2013; Jansen 2011)

Models for decision analysis?

• Sparse data and varying follow-up across treatments – Treatment specific time interactions may not be

estimable or may be very uncertain

• Could impose more structure (assumptions): – Exchangeable and related interactions

– Single common interaction

– Likely to make these assumptions in the decision model…

• Could use external data (observational, expert) – Inform functional form or priors on parameters

Bias in repeated measures NMA

• Changing potential for bias over time

– Long term follow-up may depend on study success

– Trial continuation may depend on efficacy / futility

• Changes in confounders over time may not reflect clinical practice

– Cross-over, adherence, comorbidities

Endpoint space

Multiple outcome (N)MA

• Multivariate meta-analysis • Jointly modelling two or more endpoints

– Incorporate within-study correlation between treatment effects

– Incorporate between-study correlation between treatment effects

• Models typically assume simple correlation (i.e. linear relationship)

• Increased precision – Particularly when correlation is high and reporting

incomplete

• Reduced bias if outcome reporting is selective

Additional benefits for economic evaluation

• Quantify surrogate relationships – E.g. across trials does a treatment effect on PFS lead to a

treatment effect on OS?

– Makes better use of evidence base than common decision modelling approaches • Using trial data from an individual study

• Using independent MTCs for PFS and OS

• Using PFS data and making assumptions about PPS | treatment

• More accurate representation of parameter uncertainty – Provides joint distribution of treatment effects on different

endpoints

Could we incorporate a little more structure?

• Models reflect simple linear relationships between treatment effects

– Unlikely to correspond to any underlying biological process

• Could impose model on surrogate-final endpoint relationship

– Test treatment independence of this relationship

• Examples of this exist (Welton et al 2008)

Potential benefits of structure

• Test modelling assumptions/estimate additional model parameters

• Reduced risk of over-parameterisation

– More appropriate estimation of parameter uncertainty

• Easier to incorporate external data

• Careful consideration needs to be given to the study space

• Extensions to synthesis models can help us include additional relevant endpoint data – They may also provide information about

important aspects of model structure

• Avoiding structure in the synthesis model may lead to stronger structural assumptions in decision models

Conclusion

References Hawkins, N., Scott, D. a & Woods, B. How far do you go? Efficient searching for indirect evidence. Med. Decis. Making 29, 273–81 (2009).

Woods, B. S., Hawkins, N. & Scott, D. a. Network meta-analysis on the log-hazard scale, combining count and hazard ratio statistics accounting for multi-arm trials: a tutorial. BMC Med. Res. Methodol. 10, 54 (2010).

Welton, N. J., Cooper, N. J., Ades, A. E., Lu, G. & Sutton, A. J. Mixed treatment comparison with multiple outcomes reported inconsistently across trials : Evaluation of antivirals for treatment of influenza A and B. Stat. Med. 27, 5620–5639 (2008).

Dakin, H. a et al. Mixed treatment comparison of repeated measurements of a continuous endpoint: an example using topical treatments for primary open-angle glaucoma and ocular hypertension. Stat. Med. 30, 2511–2535 (2011).

Ding, Y. & Fu, H. Bayesian indirect and mixed treatment comparisons across longitudinal time points. Stat. Med. 32, 2613–28 (2013).

Jansen, J. P. Network meta-analysis of survival data with fractional polynomials. BMC Med. Res. Methodol. 11, 61 (2011).