Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC FDR Controlling Step-up-down Tests Related to the Asymptotically Optimal Rejection Curve By Helmut Finner Institute of Biometrics & Epidemiology German Diabetes Center Leibniz-Institute at the Heinrich-Heine-University, Duesseldorf, Germany μTOSS 2010 Berlin - Multiple comparisons from theory to practice February 15-16, 2010
56
Embed
FDR Controlling Step-up-down Tests Related to the Asymptotically ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
FDR Controlling Step-up-down Tests Relatedto the Asymptotically Optimal Rejection Curve
By Helmut Finner
Institute of Biometrics & EpidemiologyGerman Diabetes Center
Leibniz-Institute at the Heinrich-Heine-University, Duesseldorf, Germany
µTOSS 2010 Berlin - Multiple comparisons fromtheory to practice
February 15-16, 2010
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Co-Workers
Main Co-Workers:
Thorsten Dickhaus (Berlin Institute of Technology)Veronika Gontscharuk (DDZ)Markus Roters (Omnicare Clinical Research)
Further:
Marsel Scheer (DDZ)Klaus Strassburger (DDZ)
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
References
• Finner, H., Dickhaus, T. & Roters, M. (2009). On the falsediscovery rate and an asymptotically optimal rejectioncurve. Ann. Statist. 37, 596-618.
• Finner, H. & Gontscharuk, V. (2009). FDR controllingstep-up-down tests related to the asymptotically optimalrejection curve. Submitted for publication.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Personal view on FDR
• 1995: Very sceptical, FDR allows cheating; Proof in B&H‘incomplete’; no reference to Eklund (1961-1963), Eklund& Seeger (1965)
• 1996: First talk on FDR, Title: Controlling the falsediscovery rate: a doubtful concept in multiple comparisonsBut statement: There will be hundreds of papers on FDRin the near future
• 1997-1999: Draft with the same title• 2001: Publication (Biom. J.) with a more relaxed title• 2002: Theoretical paper (Ann. Statist.) on expected errors
(FWER and FDR), Asymptotics• 1998-2003: Work on the Partitioning Principle (FWER)• 2005-2007: FDR, dependency, asymptotics, optimal
rejection curve
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Acknowledgment
We should all be very grateful to
Prof. Yoav Benjamini and Prof. Yosef Hochberg
for their 1995 paper which can be viewed as the initial point fora new era in multiple testing !
Benjamini, Y. and Hochberg, Y. (1995). Controlling the falsediscovery rate: A practical and powerful approach to multipletesting. J. R. Stat. Soc., Ser. B, 57, 289-300.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
OverviewIntroduction
Warm-up: Linear step-up test (LSU)
Critical value functions and rejection curves
FDR bounds for step-up procedures
Asymptotically optimal rejection curve
Finite FDR control
Recursion for the computation of critical values: Exact solving
Simple adjustments for the AORC
Iterative method
A general results on FDR bounding curves and rejection curves
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
RemarkIt is sometimes convenient to write
Vn(t) = {i ∈ In,0 : pi ≤ t},Rn(t) = {i ∈ In : pi ≤ t},
where t may be replaced by a random threshold τ .
Many multiple testing procedures can be defined in terms of asingle (random) threshold τ resulting in the decision rule
Reject Hi if and only if pi ≤ τ.
Here we give a proof for the LSU-FDR-Theorem based onOptional Stopping.
(Idea from Storey, Taylor, Siegmund, (2004). J. R. Statist. Soc.B 66, 1, 187-205. Sketch of proof faulty.)
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Re-Definition of LSU Threshold
t∗ in the definition of LSU can be replaced by
τ = max{t : Fn(t) ∨1n
=tα}
= max{t : Rn(t) ∨ 1 =nα
t}
= max{αi:n : Rn(αi:n) ∨ 1 = i, i ∈ In},
hence,
FDRϑ(ϕLSU) = Eϑ
[Vn(τ)
Rn(τ) ∨ 1
]=
α
nEϑ
[Vn(τ)
τ
].
It remains to show that Eϑ
[Vn(τ)
τ
]= n0.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Martingale and Stopping Time
Fact 1: τ is a stopping time with respect to the backwardfiltration Ft = σ(I{pi≤s}, t ≤ s ≤ 1, i ∈ In), 0 < t ≤ 1.
Proof. It suffices to show that {τ ≤ αi:n} ∈ Fαi:n . Obviously,
{τ ≤ αi:n} =n⋂
j=i+1
{pj:n > αj:n} ∈ Fαi+1:n ⊆ Fαi:n for i = 1, . . . , n−1.
For i = n we have αn:n = α and {τ ≤ α} ∈ Ft for all t ∈ (0, 1].
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Martingale and Stopping TimeFact 2: V(t)/t for 0 < t ≤ 1 is a martingale with time runningbackwards with respect to the filtration Ft, 0 < t ≤ 1.
Proof. Easy to check:
(1)V(t)
tis Ft −measurable for 0 < t ≤ 1,
(2) Eϑ
[V(s)
s|Ft
]=
V(t)t
for 0 < s ≤ t ≤ 1,
(3) Eϑ
[V(t)
t
]= n0 < ∞.
Finally, the Optional Stopping Theorem yields
Eϑ
[Vn(τ)
Rn(τ) ∨ 1
]=
α
nEϑ
[Vn(τ)
τ
]=
α
nEϑ
[Vn(1)
1
]=
n0
nα.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Can we improve LSU ?
Since
FDRϑ(ϕLSU) =n0
nα < α for n0 < n,
an obvious question is whether LSU can be improved.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Do there exit better rejection curves ?
What happens with the FDR if we use
non-linear critical values / a non-linear rejection curve
for an SU- or more generally a step-up-down procedure ?
Does there exist a rejection curve such that
Eϑ
[Vn(τ)
Rn(τ) ∨ 1
]= α
for some ϑ’s with n0 < n ?
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Critical value functions
Let ρ : [0, 1] → [0, 1] be non-decreasing and continuous with
ρ(0) = 0 and positive values on (0, 1].
Define critical values
αi:n = ρ(i/n) ∈ (0, 1], i = 1, . . . , n.
We call ρ a critical value function.
Define r by
r(x) = inf{u ∈ [0, 1] : ρ(u) = x} for x ∈ [0, 1].
We call r a rejection curve.
Often: r(x) = ρ−1(x).
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Least favorable configurations for SU based on ρ
From Benjamini and Yekutieli (2001) we get the following.
Assume BIA and consider an SU-procedure based on ρ.
If ρ(x)/x is non-decreasing in x ∈ (0, 1] then the FDR is largest
if pi = 0 a.s. for all alternative p-values pi, i ∈ In,1,
i.e., Dirac-uniform configurations are least favorable (LFC) forthe FDR.
Note: ρ(x)/x non-decreasing in x iff αi:n/i non-decreasing in i
We refer to this monotonicity property as (M).
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
FDR bound for SULet ϑ ∈ Θ such that n0 = |In,0| = number of true nulls,
let ρ satisfy (M) and let I′n,0 = In,0 \ {i0} for some i0 ∈ In,0.
Then the FDR of the SU-test ϕSU,ρ(n) based on ρ is bounded by
FDRϑ(ϕSU,ρ(n) ) ≤ n0
nEI′n,0
[ρ(Rn/n)
Rn/n
],
with ” = ” for the Dirac-uniform configuration DU(n, n0).
Moreover, if
limn→∞
n0
n= ζ and lim
n→∞
Rn
n= γζ a.s. (w.r.t. DU(n, n0)),
then the FDR is asymptotically bounded by
ζρ(γζ)γζ
.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Idea: Optimize ρ for an SU procedure
Try to find a ρ such that
ζρ(γζ)γζ
≡ min{ζ, α}
for as many ζ ’s as possible !
In other words, try to find a critical value function (or rejectioncurve) such that the FDR-level α is (asymptotically) exhaustedunder DU-configurations.
Solution: ρ(t) =αt
1− (1− α)t
r(t) = fα(t) =t
α + (1− α)t
fα: Asymptotical Optimal Rejection Curve (AORC)
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Simes’ line and AORC for α = 0.1
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
A further heuristic for the AORCConsider DU(n, n0)-models with limn→∞ n0/n = ζ.
Reject all Hi with pi ≤ t (t ∈ (0, 1)). Then the asymptotic FDR(depending on ζ and t) is given by
FDRζ(t) =tζ
(1− ζ) + tζ.
Aim:FDR ≡ α for possibly all ζ ∈ (0, 1).
We have:
FDRζ(tζ) = α ⇐⇒ tζ =α(1− ζ)ζ(1− α)
(=⇒ ζ ∈ (α, 1))
⇐⇒ ζ =α
tζ(1− α) + α.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
A further heuristic for the AORCAnsatz: tζ crossing point between rejection curve r and thelimiting cdf Gζ(t) = 1− ζ + ζt,
i.e., r(tζ) = Gζ(tζ).
Plugging in ζ = αtζ(1−α)+α yields
r(tζ) = Gζ(tζ)= (1− ζ) + ζtζ
=tζ
tζ(1− α) + α.
Hence, r = fα defined by
fα(x) =x
x(1− α) + α, x ∈ [0, 1],
is the curve we are looking for !
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Problem: SU does not work with fα
Critical values induced by AORC:
αi:n = f−1α (i/n) =
inα
1− in(1− α)
=iα
n− i(1− α), i = 1, . . . , n,
are not valid for SU because of αn:n = 1.
Ways out of this dilemma:
• Adjust the AORC slightly• Step-up-down procedures
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Step-down (SD), step-up (SU), step-up-down (SUD)
(1) SUD(λ) test ϕλ with parameter λ ∈ {1, . . . , n}:
pλ:n ≤ αλ:n ⇒ SD-part:m = max{j ∈ {λ, . . . , n} : pi:n ≤ αi:n for all i ∈ {λ, . . . , j}} ,
converge to α or a well defined FDR bounding curve.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Critical values with most influence on FDR under DU
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Fix Point Ansatz with respect to critical values
Suppose (αi:n)ni=1 satisfies (M).
Idea: Re-write critical values (αi:n)ni=1 as critical values
of an AORC fα, that is, write
αi:n =ici
n− i(1− ci)= f−1
ci(i/n), i ∈ In.
c = (c1, . . . , cn): Vector of ”‘local FDR levels”’.
Mapping: ci → αci
FDRn,n0(i)(c), i = 1, . . . , i∗,
ci → ci, i = i∗ + 1, . . . , n.
n0(i) integer closest to n− i(1− α),i∗ = i∗(n, k, α) = b(n− k)/(1− α)c,only FDRn0,n-values for n0 = k, . . . , n are iterated.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Iterative method with β-adjustment: FDR curves
SU test with n = 100, α = 0.05Start with β100 = 1.76-adjusted critical values.Choose k = 15, i∗ = 89 and J = 50 iterations.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Iterative Method
Iterative Method yields FDRs close to α.
Advantage:
Critical values always admissible.
Disadvantage:
FDRs sometimes slightly larger than α.
General problem:
Convergence properties of the algorithm not clear.
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Summary of methods
(M1) Critical values based on alternative FDR curves and exactsolving;
(M2) Adjusted critical values based on fα with βn or β∗n ;
(M3) Critical values based on iterative method.
Recommendation for large n:For example, for α = 0.05 and n ≥ 2000:
SUD tests with λn = 0.7n and β ∈ [β2000, 2] = [1.58, 2]or SU tests with β∗n ∈ [β2000, 2] = [1.45, 2] for k ≈ n(1− 2α).
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
FDR bounding curves and rejection curves
Remember the heuristic for AORC.
Let g denote an FDR bounding curve and consider
DU-models with n0/n → ζ. Try to find a rejection curve such that
FDRζ(t) = g(ζ).
This leads to an implicit definition of r and ρ = r−1:
r(
g(ζ)(1− ζ)ζ(1− g(ζ))
)=
1− ζ
1− g(ζ), ζ ∈ (0, 1) (4)
ρ
(1− ζ
1− g(ζ)
)=
g(ζ)(1− ζ)ζ(1− g(ζ))
, ζ ∈ (0, 1). (5)
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
FDR bounding curves and rejection curves
The following lemma shows that r and ρ are well defined forsuitable FDR bounding curves g.
Lemma 1. Let g be a continuous FDR bounding curve suchthat g(ζ)/ζ is non-increasing in ζ ∈ (0, 1] and letb = limζ→0 g(ζ)/ζ. Then r : [0, b] → [0, 1] and ρ : [0, 1] → [0, b]are well defined via (4) and (5), respectively, and by settingρ(0) = r(0) = 0 and r(b) = 1, ρ(1) = b. Moreover, ρ fulfills (M).
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
FDR bounding curves and rejection curves
Theorem. Let g be an FDR bounding curve with the sameproperties as in Lemma 1. Consider SUD(λn) tests ϕn based onr defined in (4) with λn/n → κ. Then we obtain for the limitingFDR in DU(n, n0) models with n0/n → ζ that
limn→∞
FDRn,n0 = g(ζ)
for (i) κ ∈ (0, 1] and ζ ∈ [0, 1] if b < 1, (ii) κ ∈ (0, 1) and ζ ∈ [0, 1]if b = 1 and (iii) κ = 0 and ζ ∈ [0, 1).
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Open Problems
• Adjusted AORC: βn bounded for SUD ?• We know that DU is LFC for SU.• Is DU LFC for SUD(λ) for λ < n ?• No counterexample is known to the speaker.• Bounds are not sharp for SUD.• Do there exist FDR bounding curves working for all n
(others than g(ζ) = αζ) ?• FDR bounds for plug-in methods (e.g. Storey’s procedure)
are not sharp.• Is DU LFC for plug-in methods ?• Do there exist better formulas for the computations of SUD
tests ?
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
References• Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery
rate: A practical and powerful approach to multiple testing.J. R. Stat. Soc., Ser. B, 57, 289-300.
• Benjamini, Y. and Yekutieli, D. (2001). The control of the false discoveryrate in multiple testing under dependency. Ann. Stat., 29, 4, 1165-1188.
• Finner, H. , Dickhaus, T. , and Roters, M. (2009). On the false discoveryrate and an asymptotically optimal rejection curve. Ann. Stat., 37, 2,596-618.
• Finner, H. and Gontscharuk, V. (2009). FDR controlling step-up-downtests related to the asymptotically optimal rejection curve. Submitted.
• Finner, H. and Gontscharuk, V. (2009). Controlling the familywise errorrate with plug-in estimator for the proportion of true null hypotheses.J. R. Stat. Soc. B, 71, 1031-1048.
• Gavrilov, Y., Benjamini, Y. and Sarkar, S. K. (2009). An adaptivestep-down procedure with proven FDR control. Ann. Stat., 37, 2,619-629.
´
Start LSU CVF and RC FDR bounds AORC Finite control Exact solving Adjusted AORC Iterative method FDR BC
Thanks for your patience !
Helau !
Altweiber 2010: Jecke feiern auf dem Rathausplatzin Dusseldorf