Page 1
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal Experiment Design (OED)for kernel ridge regression
and
the Minimum Volume Covering Ellipsoid (MVCE)
Tijl De Bie(K.U.Leuven)
Joint work with: Alexander Dolia John Shawe-TaylorMichael Titterington
Chris Harris
Page 2
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 2
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
The next hour…
Page 3
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 3
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design?OED?
Notation
Examples
Deliverables…
Page 4
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 4
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design?OED?
Notation
Examples
Deliverables…
Page 5
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 5
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design?OED?
Notation
Examples
Deliverables…
Page 6
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 6
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
NotationOED?
Notation
Examples
Deliverables…
Page 7
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 7
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
ExamplesOED?
Notation
Examples
Deliverables…
Page 8
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 8
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
ExamplesOED?
Notation
Examples
Deliverables…
Page 9
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 9
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Deliverables in this talk…OED?
Notation
Examples
Deliverables…
Page 10
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 10
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Least squares regression (LS)
Least squares
Ridge regression
Kernel RR
Page 11
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 11
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Least squares regression (LS)
Least squares
Ridge regression
Kernel RR
Page 12
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 12
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design for LS
Least squares
Ridge regression
Kernel RR
Page 13
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 13
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design for LS
Least squares
Ridge regression
Kernel RR
Page 14
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 14
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design for LS
Least squares
Ridge regression
Kernel RR
Page 15
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 15
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design for LS
Least squares
Ridge regression
Kernel RR
Page 16
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 16
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Ridge regression (RR)
Least squares
Ridge regression
Kernel RR
Page 17
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 17
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Ridge regression (RR)
Least squares
Ridge regression
Kernel RR
Page 18
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 18
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Ridge regression (RR)
Least squares
Ridge regression
Kernel RR
Page 19
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 19
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design for RR
Least squares
Ridge regression
Kernel RR
Page 20
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 20
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design for RR
Least squares
Ridge regression
Kernel RR
Page 21
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 21
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Optimal experiment design for RR
Least squares
Ridge regression
Kernel RR
Page 22
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 22
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel ridge regression (KRR)
Least squares
Ridge regression
Kernel RR
Page 23
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 23
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel ridge regression (KRR)
Least squares
Ridge regression
Kernel RR
Page 24
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 24
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel ridge regression (KRR)
Least squares
Ridge regression
Kernel RR
Page 25
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 25
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel D-OED
Least squares
Ridge regression
Kernel RR
Page 26
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 26
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel D-OED
Least squares
Ridge regression
Kernel RR
Page 27
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 27
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
OED: summaryD-OED MVCE
stan
dard
regu
lariz
edke
rnel
Least squares
Ridge regression
Kernel RR
Page 28
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 28
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Now over to novelty detection!
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 29
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 29
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Now over to novelty detection!
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 30
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 30
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Regularized MVCE
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 31
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 31
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel MVCE
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 32
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 32
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel MVCE
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 33
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 33
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Kernel MVCE
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 34
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 34
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
MVCE: summary
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 35
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 35
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
MVCE: summaryD-OED MVCE
stan
dard
regu
lariz
edke
rnel
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 36
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 36
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
MVCE: dealing with outliers
Novelty detection
MVCE and duality
Regularized MVCE
Kernel MVCE
Page 37
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 37
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Experiments – MVCE
Linear
Page 38
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 38
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Experiments – MVCE
Linear, centered
Page 39
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 39
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Experiments – MVCE
RBF-kernel
Page 40
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 40
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Experiments – MVCE
RBF-kernel
Soft-margin
Page 41
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 41
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Experiments – D-OED
Page 42
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 42
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Experiments – D-OED
2-norm
infinity norm
1-norm
Costs:
Random vs
Uniform vs
OED (blue)
Page 43
Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 43
OED for KRR, and the MVCE
Introduction
OED
MVCE
Experiments
Conclusions