Page 1
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
1
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Alpha Recyclingin
Confirmatory Clinical TrialsUnifies, Simplifies, Extends
many common MTPs
Olivier GuilbaudSenior Principal Scientist
AstraZeneca R&D, Södertälje, Sweden
Page 2
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
2
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Outline
• Background
• Splitting, Recycling, Adding, (parts of alpha)
• Graphs and Default Graphs for MTPs
• Rejection Algorithm
• Holm’s MTP for groups of Hs, and a problem to discuss
• Improvements through extensions of the Default Graph
• Summary, and further results:
• (Simultaneous Confidence regions, CTP-formulation of MTPs)
Page 3
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
3
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Background: Regulatory environment
Citation from EMEA/CPMP’s (2002) Points to Consider on Multiplicity Issues in Clinical Trials :
“ … multiplicity can have a substantial influence on the rate of false positive conclusions …”
”As a general rule it can be stated that control of the family-wise type-I error rate in the strong sense … is a minimal prerequisite for confirmatory claims”
Page 4
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
4
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Background: Multiple Confirmatory Comparisons
• Several Efficacy Variables • Several Tolerability Variables • Several Comparators (e.g. Placebo, Active 1, Active 2,
…)• Several Doses
• Delta-Noninferiority / Superiority / Delta-Superiority • Several Subgroups/Kinds of Subjects
• Several kinds of Administration (e.g. once daily, twice daily, …)
Confirmatory:Pre-specified family of multiple comparisons (i.e. statistical tests) .Risk of getting any ”false positive” result/conclusion is (no matter how many or which H0s are true – Strong Control)
Page 5
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
5
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Background: Abbreviations/Terminology
• MTP : Multiple Testing Procedure
• MCP : Multiple Comparison Procedure
• FWER : Family-Wise Error Rate (type-I errors) =Pr[at least 1 true Hi in the Family is rejected by the
MTP](to be controlled to be in the strong sense)
, no matter how many, or which, His in the family are true
In contrast to the weak sense (old) : , if all His in the family are true
Page 6
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
6
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Background: brief refresher about three basic MTPs
Three basic MTPs for Family {H1, H2, H3} based on raw p-values p1, p2, p3
• Holm MTP (1979)p(1) /3, p(2) /2, p(3) /1
• Fixed-Sequence MTP (old) p1 , p2 , p3
• Fallback MTP (2003/2005)weights w1, w2, w3 p1 w1, p2 (w2+), p3 (w3+)
Now, to the Alpha-Recycling framework
Page 7
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
7
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Two articles in Statistics in Medicine, February 2009
Only this now
Page 8
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
8
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Idea: Splitting and Recycling test-mass (= part of )
Bonferroni:
split of Fixed-Sequence: recycling of Combination of
split & recycling of
Throughout: Raw p-value available for each null hypothesis H
Page 9
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
9
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Idea: Adding test-mass from different paths
Parallel-gatekeeping MTP: 2 graphs for same MTP
Default graph
Important for rejection AlgorithmIts sequences reflect the possible rejection paths
Page 10
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
10
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Idea: Adding test-mass from different paths
Fallback MTP for 3 Hs: 2 graphs for same MTP
Default graph
Important for rejection AlgorithmIts sequences reflect the possible rejection paths
Page 11
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
11
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Idea: Adding test-mass from different paths
Holm’s MTP for 3 Hs: 2 graphs for same MTP
Default graph
Important for rejection AlgorithmIts sequences reflect the possible rejection paths
Page 12
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
12
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Rejection Algorithm - based on Default-Graph Seqs
This MTP controls (in the strong sense) the FWER to be !
Step 1. For each ”first” H in Sequences,
(a) add-upp test mass from ”its” Sequences(b) test H at this added-up level
If nothing is rejected, then Stop; otherwise:
Reduce Sequences by deleting all rejected H from Sequences, and go to Step 2 with remaining reduced Sequences
Step r = 2, 3, …. For each ”first” H in remaining reduced Sequences,
(a) add-upp test mass from ”its” Sequences(b) test H at this added-up level
If nothing is rejected, then Stop; otherwise:
Reduce Sequences by deleting all rejected H from Sequences, and go to Step r + 1 with remaining reduced Sequences
Page 13
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
13
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Comments
• number of sequences can be increased(no restriction)
• length of sequences can be increased(no restriction)
• if not all sequences contain all Hs, thenimprovements are possible (by increasing length of relevant sequences)
• interpretability of MTP is important(also when improvements are considered)
Page 14
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
14
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
A refresher – Bonferroni & Holm for groups of Hs
Step1: Bonferroni-version as above
Step2: If one group is entirely rejected, but not the other, then try again
in the other at increased level
1 2
3 4
Bonferroni for 2 groups of 2 Hs, with
Fixed-Seq testing within group
/2 /2
Holm for 2 groups of 2 Hs, with Fixed-Seq testing within group:
1 2
3 4
/2 /2
1
1 2
3 4
/2
1
Page 15
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
15
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Problem to discuss with your neighbour
1 2
3 4
Bonferroni for 2 groups of 2 Hs, with
Fixed-Seq testing within group
/2 /2
(is already in default-graph form):
Extend sequences of default graph to get recycling improvement equivalent toHolm for 2 groups of 2 Hs, with Fixed-Seq testing within group
1 2
3 4
/2 /2
Page 16
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
16
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Recent Example discussed by FDA statisticians
• Confirmatory family with 4 null hypotheses :
H1 = (Null hypothesis concerning non-inferiority for primary variable) H2 = (Null hypothesis concerning superiority for primary variable) H3 = (Null hypothesis concerning non-inferiority for secondary variable) H4 = (Null hypothesis concerning superiority for secondary variable)
H1
H2 H3
H4
w (1-w)
Page 17
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
17
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Recent Example discussed by FDA statisticians
Alpha-recycling applied to show Non-inf & Superiority for Primary & Secondary Variables
H1
H2 H3
H4
w (1-w)
Basic version
H1
H2 H3
w (1-w)
H1
H4
Reformulation in Recycling terms
Very easy to test with rejection algorithm !
But why not recycle also after the last H in each sequence ?
Page 18
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
18
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Recent Example discussed by FDA statisticians
Alpha-recycling applied to show Non-inf & Superiority for Primary & Secondary Variables - Simple improvement
Improvement Reformulation in Recycling terms
H1
H2 H3
w (1-w)
H1
H4H3
H2H4
H1
H2 H3
H4
w (1-w)
Very easy to test with rejection algorithm !This improved MTP is ”-exhaustive” !
May be questionedby some (not me)!
May be questionedby some (not me)!
Page 19
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
19
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Sequences can be extended/added for more rejections
Improved Fallback MTP for 3 Hs: 2 graphs for same MTP
Default graph
This improved MTP is ”-exhaustive” !
Page 20
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
20
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Sequences can be extended/added for more rejections
Improved Parallel-gatekeeping : 2 graphs for same MTP
Default graph
This improved MTP is ”-exhaustive” !
Page 21
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
21
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
XXX example: 2 primary & 2 second vars, 2 age stata
1
2 3
4
5 6
8
9 11
7
10 12
Original proposal
Simple improvement possible ( Holm instead of Bonf after H8 ):
Add sub-seq (11, 12) to Default-graph sequences ending with (9, 10) Add sub-seq (9, 10) to Default-graph sequences ending with (11, 12)
1
2
7
8
9
10
/81
3
7
8
9
10
/84
5
7
8
9
10
/84
6
7
8
9
10
/81
2
7
8
11
12
/81
3
7
8
11
12
/84
5
7
8
11
12
/84
6
7
8
11
12
/8
Default graphReformulation in Recycling terms
to show strong control of FWER
Page 22
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
22
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Compact notation for MTP in XXX example
1
2 3
4
5 6
8
9 11
7
10 12
Original proposal
Compact notation for this recycling MTP (B here is Bonf-operator):
B(H1 B(H2, H3), H4 B(H5, H6)) H7 H8 B(H9 H10, H11 H12)
Can be expanded (using certain rules) to 8 sequences of Hs above
1
2
7
8
9
10
/81
3
7
8
9
10
/84
5
7
8
9
10
/84
6
7
8
9
10
/81
2
7
8
11
12
/81
3
7
8
11
12
/84
5
7
8
11
12
/84
6
7
8
11
12
/8
Default graphReformulation in Recycling terms
Page 23
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
23
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
YYY example: 3 doses vs. 0, 1 primary & 1 second var
1
3
1
5
2
5
2
4
6 6
/4 /4 /4 /4
Reformulation in Recycling termsDefault graph
1 2
3 45
6
Basic version
/2 /2
Idea: To give lowest dose a chance if at least 1 of the 2 larger doses works for the primary variable
Page 24
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
24
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
YYY example – Improvement 1
1
3
1
5
2
5
2
4
6 6
/4 /4 /4 /4
Reformulation in Recycling termsDefault graph
5
6
5
6
1 2
3 45
6
Improvement 1
/2 /2
Increases level for H5 and H6 if H3 and/or H4 are rejected
Page 25
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
25
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
YYY example – Improvement 2
1 2
3 45
6
Improvement 2
1/2 1/2
Gives H1 or H2 a (Holm-type) second opportunity to be rejected if not rejected initially
1
3
1
5
2
5
2
4
6 6
/4 /8 /8 /4
Reformulation in Recycling termsDefault graph
5
6
5
6
1
5
6
/8
2
5
6
/8
2
4
2 2
4
1 1
3 1
3
3
4 3
4
3 4
Page 26
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
26
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Summary, and further results
• Easy to formulate (cf. Closed-Testing Pocedures) • Easy to perform (even manually)
• Easy to calculate multiplicity-adjusted p-values (algorithm similar to that for rejections)
• Easy to obtain weights of Bonferroni test for any intersection hypothesis HI of corresponding CTP(useful for contruction of Conf regions)
• Easy to construct new MTPs (for new problems)
• Unifies many common MTPs
• Compact algebraic notation available(to describe and derive default graphs)
Page 27
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
27
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Corresponding Simultaneous Confidence Regions ???
• Holm MTP (1979)p(1) /3, p(2) /2, p(3) /1
• Fixed-Sequence MTP (old) p1 , p2 , p3
• Fallback MTP (2003/2005)weights w1, w2, w3 p1 w1, p2 (w2+), p3 (w3+)
Solution inJASA 1999
Open until2007
Three basic MTPs for Family {H1, H2, H3} based on raw p-values p1, p2, p3
Page 28
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
28
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Corresponding Simultaneous Confidence Regions !!!
Amazing:
Simultanouslypresented at MCP 2007in Vienna
28 years after Holm (1979)
ExtensionsandRelations
Confidence regions require weights that can be obtained directly from rejection algorithm of recycling approach
Page 29
Proprietary and Confidential © AstraZeneca 2009FOR INTERNAL USE ONLY
34
O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29
Outline
• Background
• Splitting, Recycling, Adding, (parts of alpha)
• Graphs and Default Graphs for MTPs
• Rejection Algorithm
• Holm’s MTP for groups of Hs, and a problem to discuss
• Improvements through extensions of the Default Graph
• Summary, and further results:
• (Simultaneous Confidence regions, CTP-formulation of MTPs)