A low variance consistent test of relative dependency A low variance consistent test of relative dependency Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus, Matthew Blaschko 32nd International Conference on Machine Learning 2015 CVN – L2S Gatsby Unit Galen Team
20
Embed
Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus ... · Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus, Matthew Blaschko 32nd International Conference on Machine Learning
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A low variance consistent test of relative dependency
A low variance consistent test of relative dependency
Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus, Matthew Blaschko
32nd International Conference on Machine Learning 2015
CVN – L2S Gatsby Unit Galen Team
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Motivation questions
Tests of dependence : Spearman’s ρ, Kendall’s τ , kernel measure of covariance andcorrelation, distance covariance ...
However, there may be multiple dependencies: Is the dependency betweenEnglish and Dutch stronger than the dependency between English and Spanish ?
Embedding of probability measures into Reproducing Kernel Hilbert Space- In particular, we can look at the set of distributions and take each distribution P as a
point that we can embed through the mean-embedding µP :
P 7→ µP = EX∼Px k(.,X ) =∫
Ω φ(x) dP(x) ∈ F- Each distribution can thus be uniquely represented in the F .- Inner product easily compute 〈µP , µQ〉F = EX ,Y k(x , y)
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Maximum Mean Discrepancy
feature spaceProbability in
−→ DiscrepancyMaximum Mean
−→ MeasureKernel Dependence
Maximum Mean Discrepancy (MMD): [Gretton et al, 2007]
MMD2(P,Q) = ‖µP − µQ‖2F
= 〈µP , µP〉+ 〈µQ , µQ〉 − 2〈µP , µQ〉
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Kernel dependence measure
feature spaceProbability in
−→ DiscrepancyMaximum Mean
−→ MeasureKernel Dependence
Dependence Measure using the Hilbert-Schmidt Independence Criterion (HSIC):[Gretton et al, 2005, 2008]
HSIC2(Px ,Py ) = ‖µPxy − µPxPy ‖2F
HSIC2(Px ,Py ) = 0⇐⇒ Pxy = PxPy when kernels K and L are characteristic on theirrespective marginal domains.
Empirical HSIC2(Px ,Py ) : HSICXYm =
1
m2Tr(K L), O(m2) computation time
HSICXYm can be rewritten in terms of a U-statistic, which produces minimum-variance
unbiased estimators.
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
The Problem of relative dependency
Is the dependency between English and Dutch stronger than the dependencybetween English and Spanish ?
- Naively: compute the value of the two independent statistics HSICX ′Y ′m/2 and
HSICX ′′Z ′′m/2 on sample subsets;
- Efficiently: compute the value of the two dependent statistics HSICXYm and HSICXZ
m
and if empirical HSICXYm −HSICXZ
m is :
- ”less or equal than 0”: reject H0
- otherwise: do not reject H0
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
A simple consistent test via independent HSICs
Construction of two independent statistics HSICX ′Y ′m/2 and HSICX ′′Z ′′
m/2 by subsampling
K L M
Joint asymptotic distribution of independent HSIC: [Serfling, 2009]
√m
((HSICX ′Y ′
m/2
HSICX ′′Z ′′m/2
)−(
HSIC(Px ,Py )HSIC(Px ,Pz )
))d−→ N
((00
),
(σ2X ′Y ′ 00 σ2
X ′′Z ′′
))Relative dependency test with independent HSIC statistic: p-value
√m[HSICX ′Y ′
m/2 − HSICX ′′Z ′′m/2 ]
d−→
N(√
2
2(HSIC(Px ,Py )− HSIC(Px ,Pz ),
1
2(σ2
X ′Y ′ + σ2X ′′Z ′′
)
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Joint asymptotic distribution of two dependent HSIC
Joint asymptotic distribution of HSIC and test statistic
√m
((HSICXY
m
HSICXZm
)−(HSIC(Px ,Py )HSIC(Px ,Pz)
))d−→ N
((00
),
(σ2XY σXYXZ
σXYXZ σ2XZ
))
σXYXZ =16
m
1
m
m∑i=1
( (m − 1)!
(m − 4)!
)2 ∑(j,q,r)∈im3 \i
hijqrgijqr
− HSICXYm HSICXZ
m
σXYXZ =16
m
((4m)−1(m − 1)−2
3 hXYThXZ − HSICXY
m HSICXZm
)
hXY = (m − 2)2(K L
)1−m(K1) (L1)
+ (m − 2)(
(Tr(KL))1− K(L1)− L(K1))
+ (1T L1)K1 + (1T K1)L1− ((1T K)(L1))1
We have a O(m2) computation for all terms.
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Joint asymptotic distribution of two dependent HSIC
Joint asymptotic distribution of HSIC and test statistic
√m
((HSICXY
m
HSICXZm
)−(HSIC(Px ,Py )HSIC(Px ,Pz)
))d−→ N
((00
),
(σ2XY σXYXZ
σXYXZ σ2XZ
))
σXYXZ =16
m
1
m
m∑i=1
( (m − 1)!
(m − 4)!
)2 ∑(j,q,r)∈im3 \i
hijqrgijqr
− HSICXYm HSICXZ
m
σXYXZ =
16
m
((4m)−1(m − 1)−2
3 hXYThXZ − HSICXY
m HSICXZm
)
hXY = (m − 2)2(K L
)1−m(K1) (L1)
+ (m − 2)(
(Tr(KL))1− K(L1)− L(K1))
+ (1T L1)K1 + (1T K1)L1− ((1T K)(L1))1
We have a O(m2) computation for all terms.
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Properties of the test of relative dependency
Relative dependency statistical test: p-value
√m[HSICXY
m −HSICXZm ]
d−→
N(√
2
2(HSIC(Px ,Py )− HSIC(Px ,Pz),
1
2(σ2
XY + σ2XZ − 2σXYXZ
)
The dependent test is more powerful than the independent test
Theorem
The asymptotic relative efficiency of the independent approach relative to the dependentapproach is always greater to 1.
1
2(σ2
XY + σ2XZ − 2σXYXZ ) <
1
2(2σ2
XY + 2σ2XZ ) (1)
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Experiments on Synthetic Data
We control the relative degree of functional dependency between variates.
Dependency (X,Y) > Dependency (X,Z) ?si
n(t
)+γ
1N
(0,
1)
−1 0 1 2 3 4 5 6 7−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
tsi
n(t
)+γ
2N
(0,
1)
−10 −5 0 5 10 15−15
−10
−5
0
5
10
tsi
n(t
)+γ
3N
(0,
1)
−15 −10 −5 0 5 10 15−15
−10
−5
0
5
10
t + γ1N (0, 1) t cos(t) + γ2N (0, 1) t cos(t) + γ3N (0, 1)
(X) γ1 = 0.3 (Y) γ2 = 0.3 (Z) γ3 = 0.6
Pow
ero
fth
ete
sts
0.5 1 1.5 2 2.5 3 3.50
0.2
0.4
0.6
0.8
1
dependent testsindependent tests
HS
ICXZ
mvs
HS
ICX′′Z′′
m/
2
0.01 0.015 0.02 0.025 0.03 0.0350.01
0.015
0.02
0.025
0.03
0.035independent testsdependent tests
γ3 HSICXYm vs HSICX
′Y ′m/2
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Experiments on Multilingual Data
Uralic: Finnish (fi), Romance: Italian (it), French (fr), Spanish (es), Portuguese (pt),Germanic: English (en), Dutch (nl), German (de), Danish (da), Swedish (sv).
H0 : Dep(Sc., Tg.1) ≤ Dep(Sc., Tg.2)
Source Target 1 Target 2 p-valuefr es it 0.0157fr pt it 0.1882es fr it 0.2147es pt it < 10−4
es pt fr < 10−4
pt fr it 0.7649pt es it 0.0011pt es fr < 10−8
Relative dependency tests between Romance
languages.
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Pediatric high-grade gliomas (pHGG)
Brain tumors localisation pHGG have different genetics origins depending on thelocation of the tumor in the brain. The goal is to identify the mechanismsresponsible for the tumor.
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments
Conclusion
A novel non-parametric statistical test that determines whether a source variable ismore stronger dependent on one target variable or another.
The test is low variance, consistent and unbiased.
Computation requirement is quadratic time.
Bibliography:- Gretton, A., Fukumizu, K., Teo, C. H., Song, L., Scholkopf, B., Smola, A. J. (2008).
A kernel statistical test of independence. In Advances in Neural Information ProcessingSystems.
- Gretton, A., Herbrich, R., Smola, A., Bousquet, O., et Schoelkopf, B., (2005). KernelMethods for Measuring Independence, Journal of Machine Learning Research, 6 ,2075-2129,
- Hoeffding, W. (1948). A class of statistics with asymptotically normal distribution.The annals of mathematical statistics, 293-325.
- Serfling, R. J. (2009). Approximation theorems of mathematical statistics, 162. JohnWiley & Sons.
A low variance consistent test of relative dependencyIntroductionTest of relative dependencyExperiments