Component Analysis for CV & PRcvpr11.cecs.anu.edu.au/files/australia_summer_school_4_Fernando.pdf · array processing (e.g. cocktail problem, eco cancellation), blind source Component

1

Component Analysis Methodsfor Computer Vision and

Pattern Recognition

Fernando De la TorreFernando De la Torre

Component Analysis for CV & PR F. De la Torre CVPR Easter School-2011 1

Computer Vision and Pattern Recognition Easter School Computer Vision and Pattern Recognition Easter School March 2011March 2011

Component Analysis for CV & PR • Computer Vision & Image Processing

– Structure from motion.– Spectral graph methods for segmentation.– Appearance and shape models.– Fundamental matrix estimation and calibration.– Compression.– Classification.– Dimensionality reduction and visualization.

• Signal Processing– Spectral estimation, system identification (e.g. Kalman filter), sensor

array processing (e.g. cocktail problem, eco cancellation), blind source


array processing (e.g. cocktail problem, eco cancellation), blind source separation, …

• Computer Graphics– Compression (BRDF), synthesis,…

• Speech, bioinformatics, combinatorial problems.

• Computer Vision & Image Processing– Structure from motion.– Spectral graph methods for segmentation.– Appearance and shape models.

Structure from motion

Component Analysis for CV & PR

– Fundamental matrix estimation and calibration.– Compression.– Classification.– Dimensionality reduction and visualization.








Spectral graph methods for segmentation.









2

• Computer Vision & Image Processing– Structure from motion.– Spectral graph methods for segmentation.– Appearance and shape models.Appearance and shape models





pp p










Dimensionality reduction and visualization









array processing (e.g. cocktail problem, eco cancellation), blind sourcecocktail problem





cocktail problem

Independent Component Analysis (ICA)Sound

Source 1Mixture 1

Sound Source 2

Mixture 2

Output 1

Output 2ICA


Sound Source 3

Mixture 3

Output 3

3










Why CA for CV & PR?• Learn from high dimensional data and few samples.

– Useful for dimensionality reduction.

(Everitt,1984)

• Easy to incorporate – Robustness to noise, missing data, outliers (de la Torre & Black, 2003a)– Invariance to geometric transformations (Frey et al. 99, de la Torre & Black,

2003b;, Cox et al. 2008)

– Non-linearities (Kernel methods) (Scholkopf & Smola,2002; Shawe-Taylor & Cristianini,2004)

– Probabilistic (latent variable models)M lti f t i l (t ) ( & O’ &


features samples

• Efficient methods O( d n<

4

Generative Models

• Principal Component Analysis/Singular Value Decomposition

BCD

Decomposition1) Robust PCA/SVD, PCA with uncertainty and missing data.2) Parameterized PCA3) Filtered Component Analysis4) Subspace regression5) Kernel PCA

• K-means and spectral clustering


6) Aligned Cluster Analysis (ACA)• Non-Negative Matrix Factorization• Independent Component Analysis.

Principal Component Analysis (PCA)(Pearson, 1901; Hotelling, 1933;Mardia et al., 1979; Jolliffe, 1986; Diamantaras, 1996)


• PCA finds the directions of maximum variation of thedata based on linear correlation.

• PCA decorrelates the original variables.

PCA

Tnn μ1BCdddD ...21

d=d=pixelspixels

nn= images= images

kdnd BD

kccc ......21

kbbb 21

1 dnk μC

•Assuming 0 mean data the basis B that preserve the maximum


Assuming 0 mean data, the basis B that preserve the maximumvariation of the signal is given by the eigenvectors of DDT.

BΛBDD Td d

Snap-shot Method & SVD• If d>>n (e.g. images 100*100 vs. 300 samples) no DDT.• DDT and DTD have the same eigenvalues (energy) and

related eigenvectors (by D). • B is a linear combination of the data! (Sirovich 1987)• B is a linear combination of the data!

• [α,L]=eig(DTD) B=D α(diag(diag(L))) -0.5ΛDαDDαDDDDαBBΛBDD TTTT

TVUΣD

• SVD factorizes the data matrix D as:

BCD

TT UUΛDD

TT VVΛDD

(Beltrami, 1873; Schmidt, 1907; Golub & Loan, 1989)

(Sirovich, 1987)


SVDPCA

diagonal

nnnkkd

T

ΣIVVIUUVΣU

VUΣD

TT

TT CCIBBCB

BCDnkkd

5

Error Function for PCA

(Eckardt & Young, 1936; Gabriel & Zamir, 1979; Baldi & Hornik, 1989; Shum et al., 1995; de la Torre & Black, 2003a)

n

E BCDBdCB 2)(

• PCA minimizes the following function:

• Not unique solution:• To obtain same PCA solution R has to satisfy:

TT CCIBB

CRCBRBˆˆˆˆ

ˆˆ 1

kk RBCCBRR 1

Fi

iiE BCDBcdCB, 1

21)(


• R is computed as a generalized k×k eigenvalue problem.

CCIBB

11 BRBRCC TT (de la Torre, 2006)

PCA/SVD in Computer Vision• PCA/SVD has been applied to:

– Recognition (eigenfaces:Turk & Pentland, 1991; Sirovich & Kirby, 1987; Leonardis & Bischof, 2000; Gong et al., 2000; McKenna et al., 1997a)

– Parameterized motion models (Yacoob & Black, 1999; Black et al., 2000; Black, 1999; Black & Jepson, 1998)

– Appearance/shape models (Cootes & Taylor, 2001; Cootes et al., 1998; Pentland t l 1994 J & P i 1998 C i & S l ff 1999 Bl k & J 1998 Bl &et al., 1994; Jones & Poggio, 1998; Casia & Sclaroff, 1999; Black & Jepson, 1998; Blanz &

Vetter, 1999; Cootes et al., 1995; McKenna et al., 1997; de la Torre et al., 1998b; de la Torre et al., 1998b)

– Dynamic appearance models (Soatto et al., 2001; Rao, 1997; Orriols & Binefa, 2001; Gong et al., 2000)

– Structure from Motion (Tomasi & Kanade, 1992; Bregler et al., 2000; Sturm & Triggs, 1996; Brand, 2001)

– Illumination based reconstruction (Hayakawa, 1994)– Visual servoing (Murase & Nayar, 1995; Murase & Nayar, 1994)– Visual correspondence (Zhang et al., 1995; Jones & Malik, 1992)

C i i i


– Camera motion estimation (Hartley, 1992; Hartley & Zisserman, 2000)– Image watermarking (Liu & Tan, 2000)– Signal processing (Moonen & de Moor, 1995)– Neural approaches (Oja, 1982; Sanger, 1989; Xu, 1993)– Bilinear models (Tenenbaum & Freeman, 2000; Marimont & Wandell, 1992)– Direct extensions (Welling et al., 2003; Penev & Atick, 1996)

1-Robust PCA•Two types of outliers:

Sample outliers Intra-sample outliers(Xu & Yuille., 1995) (de la Torre & Black, 2001b; Skocaj & Leonardis, 2003)

•Standard PCA solution (noisy data):


Robust PCA• Using robust statistics:

Pixel residual(de la Torre & Black, 2001b; de la Torre & Black, 2003a)

n

i

d

pp

k

jjipjppirpca cbdE

1 1 1

),(),,( μCB

quadraticoutlieroutlier


meanBasis (B) &Coefficients(c)

robustrobust

6

Numerical Problems• No closed form solution in terms of an eigen-equation.• Deflation approaches do not hold.

First eigenvector with

T

T11

uuAA

uuAA

222

1

'''

'

First eigenvector with

highest eigenvalue.

Second eigenvector with highest eigenvalue.


• In the robust case all the basis have to be computed simultaneously (including the mean).

How to Optimize it?

n

i

d

pp

k

jjipjppirpca cbdE

1 1 1

),(),,( μCB

C

HCC

BHBB

rpcac

nn

rpcab

nn

E

E

11

11

)(max

)(max

2

2

Tii

rpca

Tii

rpca

Ediag

Ediag

ccH

bbH

c

b

• Normalized Gradient descent


(Blake & Zisserman, 1987)• Deterministic annealing methods to avoid local minima.

Example

Statistical outlier

• Small region• Short amount of time


Robust PCA

Original PCA RPCA Outliers


7

Structure from Motion

More work on Robust PCA


• Robust estimation of coefficients (Black & Jepson, 1998; Leonardis & Bischof, 2000; Ke & Kanade, 2004)

• Robust estimation of basis and coefficients (Gabriel & Odoro, 1984; Croux & Filzmoser., 1981; Skocaj et al., 2002;Skocaj & Leonardis, 2003; de la Torre & Black, 2001b; de la Torre & Black, 2003a)

• Other Robust PCA techniques (sample outliers) (Campbell, 1980; Ruymagaart, 1981; Xu & Yuille., 1995)

More work on Robust PCA

1- PCA with Uncertainty and Missing Data

• If weights are separable closed-form solutionTwwW

d

i

n

j

k

ssjisijijF

cbdwE1 1

2

12 )()()( BCDWCB, • Adding uncertainty

If weights are separable closed form solution.

productHadamard

wij

nd

0 W

D

n

n

dd

dd

221

111

r

r

r w

w

...2

1

w

cnccc www 21w ……

r cwwW


• Generalized SVD

productHadamard

dnd dd 1

rdw

(Greenacre, 1984; Irani & Anandan, 2000;)

General Case• For arbitrary weights no closed-form solution.

dpTppTpTp

n

iiii

TiiF

diag

diagE1

2

))(()(

))(()()()(

bCdwbCd

BcdwBcdBCDWCB,

(Wiberg, 1976 , Torre & Black, 2003a)

• Alternated least squares algorithms– Slow convergence, easy implementation.

• Damped Newton Algorithm– Fast convergence.

B

CBBCDWCB,

12

212

][)(

||||||||)()(

EEvec

E FFF

p 1

I

repeat

EErepeat

)(10

1

22

22

gHxy

vg

vH

(Buchanan & Fitzgibbon., 2005)


– H definite positive:

vvvv

CB

v

22

22

)1( ][)()( EE

vecvec nn

Iv

H

22

2E econvergencuntil

FFuntilI

10;

)()()(

yx

xygHxy

Related work

• Iterative (Wiberg, 1976; Shum et al., 1995; Morris & Kanade, 1998; Aans et al.,2002; Guerreiro & Aguiar, 2002)

• Closed-form (Aguiar & Moura, 1999; Irani & Anandan, 2000)P f t i ti• Power factorization (Hartley & Schaalitzky, 2003)

• Bayesian estimation (L.Torresani & Bregler, 2004)


8

• Learn a subspace invariant to geometric transformations?

2- Parameterized Component Analysis (PaCA) (de la Torre & Black, 2003b)

. . .

• Data has to be geometrically normalized– Tedious manual cropping.


Tedious manual cropping.

– Inaccuracies due to matching ambiguities.

– Hard to achieve sub-pixel accuracy.

Error function for PaCA

)()()(),,( 211

2

1caBc)af(x,daCB

WtppE

T

ttt

Basis ((BB) &) &Motion Regularization


coefficients ((cc))(warping)Regularization

2

3121 1

2

211 WWaΓacΓc

tat

T

t

L

ltct

EigenEye Learning


More examples• UPS dataset.

Random selection of 100 images (16×16 pixels). Incrementally update until preserve 80% of the energy.

PaK PCAOriginal CongealingPaK-PCAOriginal Congealing

Component Analysis for CV & PR F. De la Torre CVPR Easter School-2011

9

Improving facial landmark labeling•Hand label (red dots), PaK-PCA label (yellow)


More on Parameterized CA• Probabilistic model

– Search scales exponentially with the number of motion parameters (Frey & Jojic, 1999a; Frey & Jojic, 1999b; Williams & Titsias, 2004)

• Other continuous approaches.

• Invariant clustering

• Non-rigid motion

(Schewitzer, 1999; Rao, 1999; Shashua et al., 2002, Cox et al. 2008)

(Fitzgibbon & Zisserman, 2003)

(Baker et al., 2004)


• Invariant recognition

• Invariant support vector machines• Parameterized Kernel Component Analysis (De la Torre, 2008)

(Black & Jepson, 1998)

(Avidan, 2001)

3- Filtered Component Analysis(de la Torre et al.,2007b)

1) No local minimum in the expected place.

2) Many local minima


2) Many local minima.

Multi-band representation

• Texture classification (Nunes et al. ‘03, Freeman, Zalesny & Van Gool, Leung & Malik ‘01, Cula & Dana ‘01, Varma & Zisserman ‘02, De Bonet ’97, Heeger & Bergen ’95, Portilla & Simoncelli ’00, Zhu et al. ‘98)

• Face recognition (Wang et al ’03 Hie et al ’04 Wiskott et al ’97 Lades et alFace recognition (Wang et al. 03, Hie et al. 04, Wiskott et al. 97, Lades et al. ’93, Wechler et al. ’02, Zhao et al. ‘98)

• Filters (Gabor, Wavelets, Volterra, Fourier transform, …)Convolution


10

Multi-band representation

1) Global minimum in the


1) Global minimum in the expected place.

2) Distance between global and other minima is larger.

Filtered Component Analysis (FCA)

22

11

22

11 ||)(||||)(||),...,(

2

fbackgroundj

n

j

F

ff

n

iFE FμdFμdFF

Filters

Images

ConvolutionConvolution

jivecvecvecvec

jTi

iTi

0)()(1)()(

FFFF

No overlap between filters

No trivial solution (0)

F n

T 2


f i

TfPCAE

1

22

1

||)(|| μdF

Robustness of FCA Training: 100 images Testing: 120 images

Correct global minimum

Gray FCA (4) Gabor(4)

41 % 74 % 62%


Correct global minimum 41 % 74 % 62%14.59 26 19.683.28 1.4 1.92

Correct to 2nd minimum distance

Average number of local minima

Other work

• Incremental PCA (de la Torre et al., 1998b; Ross et al., 2004; Brand, 2002; Skocaj & Leonardis, 2003; Champagne & Liu., 1998; A. Levy, 2000)

Mixture of subspaces• Mixture of subspaces (Vidal et al., 2003; Leonardis et al., 2002)• Changing the margin in SVM (Ashraf and Lucey 2010)• Exponential family PCA (Collins et al. 01)


11

4- Subspace Regression: From a Single Image to a Subspace

• Traditional subspace methods

• Subspace Regression (Kim et al. 2010)


4- Subspace Regression (II)

frontal(s=0) Subject Subspace

subj=1

subj=2

subj=i

… … … … ……

… … … … ……

……


TestImage(s=0)

?Predict a subspace from a single image

Subspace Regression (II)

b1 b2 b3 b4 b5

• Generated samples for each pose

Optimi ation problem

1 2 3 4 5


• Optimization problem

Experiment I


ErrorMeasure

Baseline I(img -> img)

Baseline II(img -> subsp)

SubspaceRegression

Matlab®’ssubspace()

1.3507(1.2312)

1.4088(1.1645)

1.0860(1.0651)

12

Experiment II

• Predicting a Subspace for Illumination– CMU PIE data set– 60 aligned subjects– 19 different illuminations


Subspace tracking


(Template Matching: 42.99)

IVT-SS: 38.41

Subspace Regression: 37.98

5-Kernel PCA

),,(),,(),( 32122

212121 zzzxxxxxx

• The kernel defines an implicit mapping (usually high dimensional andnon-linear) from input to feature space so the data becomes linearly

Feature spaceInput space


non linear) from input to feature space, so the data becomes linearlyseparable.

• Computation in the feature space can be costly because it is(usually) high dimensional– The feature space is typically infinite-dimensional!

Kernel Methods• Suppose (.) is given as follows

• An inner product in the feature space is

• So, if we define the kernel function as follows, there is no need to carry out (.) explicitly

• This use of kernel function to avoid carrying out ( )


• This use of kernel function to avoid carrying out (.) explicitly is known as the kernel trick. In any linear algorithm that can be expressed by inner products can be made nonlinear by going to the feature space

13

Kernel PCA(Scholkopf et al., 1998)


Generative Models


BCD

Decomposition1) Robust PCA/SVD, PCA with uncertainty and missing data.2) Parameterized PCA3) Filtered Component Analysis4) Subspace regression5) Kernel PCA

• K-means and spectral clustering


6) Aligned Cluster Analysis (ACA)• Non-Negative Matrix Factorization• Independent Component Analysis.

The Clustering Problem• Partition the data set in c-disjoint “clusters” of data points.

• Number of possible partitions


12

1

10421

)1(1),(

cn

iic

ccnS n

c

i

c

• NP-hard and approximate algorithms (k-means, hierarchical clustering, mog, …)

K-means

FTE ||)(||),(0 MGDGM

(Ding et al., ‘02, Torre et al ‘06)

xyD

TMG xy

57

y

TG

yD

M xy


1

2

3

45

6

7

8

9

10

x

14

Spectral ClusteringAffinity Matrix

(Dhillon et al., ‘04, Zass & Shashua, 2005; Ding et al., 2005, De la Torre et al ‘06)

FcTE ||)(||),(0 WMCΓCM


Fc0

)(DΓ )](...)()([ 21 ndddΓ Normalized Cuts (Shi & Malik ’00)Ratio-cuts(Hagen & Kahng ’02)

6- Aligned Cluster Analysis (ACA)• Mining facial expression


• Mining facial expression for one subject• Mining facial expression for one subject

Problem

• Summarization

• Visualization


• Indexing

• Mining facial expression for one subjectLooking up Sleeping SmilingLooking forwardWaking up

Problem

• Summarization

• Visualization


• Indexing

15

• Mining facial expression of one subject

Problem

• Summarization

• Embedding

I d i


• Indexing

• Mining facial expression for one subject

Problem

• Summarization

• Embedding


• Indexing

k-means and kernel k-means2||||),( FJ MGXGM

(MacQueen 67, Ding et al. 02, Dhillon et al. 04, Zass and Shashua 05, De la Torre 06)

xyX

)(G

MG xy

24

57

y

G )))((()( 1n GGGGIKG TTtrJM xy


13

4 6

8

9

10

x)()( XXK T

Problem formulation for ACA (I)

)..[ 21 ssX )..[ 43 ssX )..[ 1 mm ss X

1 2 3 1Labels (G)

1s 2s 3s 4sStart and end of the segments (s)

s 2)(),,( FacaJ MGXGM )..[)..[)..[ 13221 ,...,, mm ssssss XXX


16

Problem formulation for ACA (II)

2

)..[)..[)..[ ),...,,(),,( 13221 Fssssssaca mmJ MGXXXSGM

k

ccSS

m

ici mg ii

1

2

2)..[1

1X

Dynamic Time Alignment Kernel (Shimodaira et al. 01)

X[Si , Si+1) mc

X [Si , Si+1)


[ i , i+1)

mc

Matrix formulation for ACA

GGGGILKL 1n )(with)( TTkmk trJ

)()( XXK T

men

ts

GHGGGHILWLK 1n )(with))o(( TTTaca trJ

ers


samples

segm

2371,0 H

2323RW

clus

te

segments 731,0 G

Optimizing ACA (forward step)• Efficient Dynamic Programming

i =23 i =25 i =29

2.11.81.7

2.41.21.8

2.41.91.5


maxw

Optimizing ACA (backward step)

)( max2wnO


17

Honey bee dance data (Oh et al. 08)

Three behaviors: 1‐waggle, 2‐left turn, 3‐right turn

Seq 1 Seq 2 Seq 3 Seq 4 Seq 5 Seq 6ACA 0.845 0.925 0.600 0.922 0.878 0.928


PS- SLDS (Oh et al 08) 0.759 0.924 0.831 0.934 0.904 0.910

HDP- VAR(1)-HMM (Fox et al 08)

0.465 0.441 0.456 0.832 0.932 0.887

Spectral Clustering 0.698 0.631 0.509 0.671 0.577 0.649

Facial image features• Active Appearance Models (Baker and Matthews ‘04)

Appearance

Upper face

Shape• Image features


Lower face

• Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger)

Facial event discovery across subjects


• Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger)

Facial event discovery across subjects


ACA Spectral Clustering

(SC)0.87(.05) 0.56(.04)

• 10 sets of 30 people

18

Unsupervised facial event discovery


Clustering human motion


clustering of human motion II


Generative Models


BCD

Decomposition1) Robust PCA/SVD, PCA with uncertainty and missing data.2) Parameterized PCA3) Filtered Component Analysis4) Kernel PCA

• K-means and spectral clustering5) Aligned Cluster Analysis (ACA)


• Non-Negative Matrix Factorization• Independent Component Analysis.

19

“Intercorrelations among variables are the bane of the

multivariate researcher’s struggle for meaning”

Cooley and Lohnes, 1971


Part-based Representation

The firing rates of neurons are never negative. Independent representations.

NMF & ICA


NMF & ICA

Non-negative Matrix Factorization• Positive factorization.

• Leads to part-based representation.0||||)( CB,BCDCB, FE


Nonnegative Factorization

ij ijijdF

2

0,0)(min BC

CB Inference:

(Lee & Seung, 1999;Lee & Seung, 2000)

j

ij

ijijij )(

)(BVBDB

CC TT

Learning:

Tij

T

ijij )()(

BCCDC

BB

Derivatives:

ijijij

F )()( CBBCBC

TT

TTF )()( DCBCC


• Multiplicative algorithm can be interpreted as diagonally rescaled gradient descent.

ijTjj )(BCCijij

ij

)()( DCBCCB

20

Generative Models


BCD

Decomposition1) Robust PCA/SVD, PCA with uncertainty and missing data.2) Parameterized PCA3) Filtered Component Analysis4) Kernel PCA

• K-means and spectral clustering5) Aligned Cluster Analysis (ACA)


• Non-Negative Matrix Factorization• Independent Component Analysis.

Independent Component Analysis

• We need more than second order statistics to represent the signal.


ICA

• Look for si that are independent.• PCA finds uncorrelated variables, the independent

components have non Gaussian distributions

1 BWWDSCBCD(Hyvrinen et al., 2001)

components have non Gaussian distributions.• Uncorrelated E(sisj)= E(si)E(sj)• Independent E(g(si)f(sj))= E(g(si))E(f(sj)) for any non-

linear f,g


PCA ICA

ICA vs PCA


21

Many optimization criteria

• Minimize high order moments: e.g. kurtosiskurt(W) = E{s4} -3(E{s2}) 2

• Many other information criteria.

n

ii

n

iii S

11)(cBcd

Sparseness (e.g. S=| |)

(Olhausen & Field, 1996)• Also an error function:


(Chennubhotla & Jepson, 2001b; Zou et al., 2005; dAspremont et al., 2004;)

• Other sparse PCA.

Basis of natural images


Denoising

Originalimage Noisy Image

(30% i )(30% noise)

Denoise(Wi filt ) ICA


(Wiener filter) ICA

Outline• Introduction• Generative models

– Principal Component Analysis (PCA) and extensions– K-means, spectral clustering and extensions– Non-negative Matrix Factorization (NMF)– Independent Component Analysis (ICA)

• Discriminative models– Linear Discriminant Analysis (LDA) and extensions– Oriented Component Analysis (OCA)– Canonical Correlation Analysis (CCA) and extensions

• A unifying view of CA


• A unifying view of CA• Standard extensions of linear models

– Latent variable models.– Tensor factorization

22

Discriminative Models

• Linear Discriminant Analysis (LDA)7) Discriminative Cluster Analysis ) y8) Multimodal Oriented Discriminant Analysis

• Oriented Component Analysis (OCA)• Canonical Correlation Analysis (CCA)

9) Dynamical Coupled Component Analysis10) Canonical Time Warping


Linear Discriminant Analysis (LDA)

C C

(Fisher, 1938;Mardia et al., 1979; Bishop, 1995)

BΛSBSBSBBSBB bb t

tT

T

J ||||)(

C

i

C

j

Tjijib

1 1

))(( μμμμS

n

i

Tii

Tt

1

ddDDS

c C


• Optimal linear dimensionality reduction if classes are Gaussian with equal covariance matrix.

Tji

c

j

C

ijiw

i

)()(1 1

μdμdS

Error function for LDA

FTTT

LDAE ||)()(||),( 21

DBAGGGBA

(de la Torre & Kanade, 2006)

[d1 d2 ... dn]

Equations n×c Unknowns d×c

Ad=pixels

K=di

m

subs

pace

knnk

ijg

G

1G1

}1,0{

0...01...00...1

TG

c=cl

asse

s

n=samples


Equations n×c Unknowns d×c

• d>>n an UNDETERMINED system of equations! (over-fitting)

15

20

6

8

7-Discriminative Cluster Analysis (DCA)(de la Torre & Kanade, 2006)

FTTT

DCAE ||)()(||),,( 21

DBAGGGGBA

−10

−5

0

5

10 −10−5

05

10

−20

−15

−10

−5

0

5

10

15

YX

Z

−8 −6 −4 −2 0 2 4 6 8−10

−8

−6

−4

−2

0

2

4

6

X

Y

20PCA+k-means DCA


−10−5

05

10 −10−5

05

10−20

−15

−10

−5

0

5

10

15

20

Y

X

Z

−10

−5

0

5

10 −10−5

05

10

−20

−15

−10

−5

0

5

10

15

20

YX

Z

PCA+k means

23

Clustering faces20

40

60

80

100

TT GGGG 1)(

20 40 60 80 100 120 140

120

140

0

0.1

0.2

PCA

PCA DCA


−0.4−0.2

00.2

0.4

−0.2

0

0.2

0.4

0.6−0.2

−0.1

DCA vs. PCA+k-means

1

1.05

DCA

PCA+k−means

0.75

0.8

0.85

0.9

0.95

Acc

urac

y

PCA+k−means


5 10 15 20 25 30 35 400.65

0.7

0.75

Number of clusters (classes)

8- Multimodal Oriented Component Analysis (MODA)

• How to extend LDA to deal with:– Model class covariances.

(de la Torre & Kanade, 2005a)

– Multimodal classes.– Deal efficiently with huge covariance matrices

(e.g. 100*100).


Multimodality

−5

0

5

10

−200−100

0100

200

−200

0

200−10

−5

MODA

10

20

30

40

2

4

6

8

10

LDA


0 10 20 30 40 50 60−40

−30

−20

−10

0

10

0 10 20 30 40 50 60−10

−8

−6

−4

−2

0

2

24

MODAB that MAXIMIZES the Kullback-Leibler divergence between clusters among l

classes

Trj

ri

classesri

rj

rj

ri

ri

rj

ri

rj

Ttr 1111 )))())(((( 2121211212 BμμΣΣμμΣΣΣΣB

classes.


i

jiij Cr Cr

ijjiijiji j1 1 2

• 1 mode per class and equal covariances equivalent to LDA.

Optimization

1

1 ))()(()(i

iT

iTtrJ BABBΣBB

• Hard optimization problem

T(B)

)()()()(

00 BBBBB

JTJT

J(B)

• Iterative Majorization (Kiers, 1995; Leeuw, 1994)


W1

W0

Related LDA work

• Face recognition (Belhumeur et al., 1997;Zhao, 2000;Martinez & Kak, 2003)

• Small sample problem (Chen et al., 2000; Yu & Yang, 2001)• Mixture (Hastie et al., 1995; Zhu & Martinez, 2006;)• Neural approaches (Gallinari et al., 1991; Lowe & Webb, 1991)• Heteroscedastic discriminant analysis (Kumar &

Andreou, 1998; Fukunaga, 1990; Mardia et al., 1979; Saon et al., 2000;)







25

Oriented Component Analysis (OCA)

T bΣbsignal

OCAb

• Generalized eigenvalue problem:

OCAT

OCAOCA

noiseOCA

signal

bΣb

bΣb

keki bΣbΣ

noise


Generalized eigenvalue problem:• boca is steered by the distribution of noise

keki

(de la Torre et al., 2005a)

Representational Oriented Component Analysis (ROCA)

kT

kTk

ek

i

bΣbbΣb

1

1

jTj

jTj

e

i

bΣbbΣb

2

2







Canonical Correlation Analysis (CCA)

• Learn relations between multiple data sets? (e.g. find features in one set related to another data set)

• Given two sets , CCA finds the pair of directions w and w that maximize the correlation

ndnd and 21 YX

(Mardia et al., 1979; Borga 98)

of directions wx and wy that maximize the correlation between the projections (assume zero mean data)

• Several ways of optimizing it:Ty

TTy

Tx

TTx

yTT

x

YwYwXwXw

YwXw

TT w0XXYX0


• An stationary point of r is the solution to CCA.

y

xddddT

ddddT w

ww

YY00XX

Β0YXYX0

A )()()()( 21212121 ,

ΒwAw

26

9- Dynamic Coupled Component Analysis (DCCA)

Data 1Data 1 Data 2Data 2

(de la Torre & Black, 2001a)

• Learning the coupling


Learning the coupling.• High dimensional data.• Limited training data.

Solutions?• PCA independently and general mapping

PCA PCA


• Signals dependent signals with small energy can be lost.

DCCA

ˆ

n

iiicca

i

E1

2

1ˆˆˆ)ˆ,,,,ˆ,(

WcBμdμμCABB

ectio

nec

tion

BReconstructionReconstruction

B̂


n

iii

n

ii

Ti

i

ii 1

2

3121

2

2

1

)(WW

AccμdBc DynamicsDynamicsPro

jePr

oje

Dynamic Coupled Component Analysis


27

10- Canonical Time Warping (CTW)


Canonical Correlation Analysis (CCA)(Hotelling 1936)

• CCA minimizes:different #rows, same #columns

TT

ndnd yx YX ,

CCASpatial transformation

2),(

F

Ty

TxyxccaJ YVXVVV b

yTT

y

xTT

xts IVYYV

VXXV

.


A least-square formulation for DTW

same #rows, different #columnsyx ndnd YX ,

2),(

F

Ty

TxyxdtwJ YWXWWW


Canonical Time Warping (CTW)

Reminder 2

2

),(

),(

F

Ty

Txyxdtw

F

Ty

Txyxcca

J

J

YWXWWW

YVXVVV

different #rows, different #columns

spatial transformation

F

Ty

Ty

Tx

TxyxyxctwJ YWVXWVVVWW ),,,(

2

yyxx ndnd YX ,


temporal alignment

by

Ty

Ty

Ty

xT

xTx

Txts I

VYWYWV

VXWXWV

..

28

Facial expression alignment


Facial expression alignment


Aligning human motion

Boxing O i bi t


Boxing Opening a cabinet

Aligning motion capture and video


29


– Principal Component Analysis (PCA) and extensions– K-means, spectral clustering and extensions– Non-negative Matrix Factorization (NMF)– Independent Component Analysis (ICA)

• Discriminative models– Linear Discriminant Analysis (LDA) and extensions– Oriented Component Analysis (OCA)– Canonical Correlation Analysis (CCA) and extensions

• A unifying view of CA


• A unifying view of CA• Standard extensions of linear models

– Latent variable models.– Tensor factorization

The fundamental equation of CA

FTE ||)(||),(0 WBAWBA

Given two datasets : nxnd and XD

CC

FcrE ||)(||),(0 WBAWBA

Weightsfor rows

Weightsfor columns

Regressionmatrices XD )()(

C


AB

Properties of the cost function• E0(A,B) has a unique global minimum (Baldi and Hornik-89).

• Closed form solutions for A and B are:

)()()( 22120 AWWWAAWAA TrTTTT ccctrE

))(()()( 221222120 BWWWWWBBWBB TTTTTtrE


))(()()(0 BWWWWWBBWBB rcccrrtrE

Principal Component Analysis (PCA)

• PCA finds the directionsof maximum variation ofthe data based on linear

(Pearson, 1901; Hotelling, 1933;Mardia et al., 1979; Jolliffe, 1986; Diamantaras, 1996)

the data based on linearcorrelation.

• Kernel PCA finds the


directions of maximumvariation of the data inthe feature space.

),,(),,(),( 32122

212121 zzzxxxxxx

30

PCA-Kernel PCA

FTE ||)(||),(0 WBAWBA

• Error function for KPCA: (Eckardt & Young, 1936; Gabriel & Zamir, 1979; Baldi & Hornik, 1989; Shum et al., 1995; de la Torre & Black, 2003a)

• The primal problem:

FcrE ||)(||),(0 WBAWBA F

TkpcaE ||)(||),( BADBA

)(D

)()()( 1 BDDBBBB TTTkpca trE


))()(()()( 1 ADDAAAA TTTkpca trE

)()()(kpca• The dual problem:

Linear Discriminant Analysis (LDA)(Fisher, 1938;Mardia et al., 1979; Bishop, 1995)

C

i

C

j

Tjijib

1 1))(( μμμμS

n

BΛSBSBSBBSBB bb tTtTtrJ )()()( 1


• Optimal linear dimensionality reduction if classes are Gaussian with equal covariance matrix.

n

i

Tii

Tt

1

ddDDS


• Given two sets , CCA finds the pair of directions wx and wy that maximize the correlation between the projections (assume zero mean data)

ndnd and 21 DX

(Fisher 36;Mardia et al., 1979;)

between the projections (assume zero mean data)

Td

Td

Tx

TTx

dTT

x

DwDwXwXwDwXw

T


FcT

rE ||)(||),(0 WBAWBA TTE ||)()(||)( 2

1

XBADDDBA


FTT

CCAE ||)()(||),( 2 XBADDDBA

0...01...00...1

TG

c=cl

asse

s

n=samples


• CCA is the same as LDA changing the label matrix by a new set X

31

K-means

FcT

rE ||)(||),(0 WBADWBA (Ding et al., ‘02, Torre et al ‘06)

xyD

TBA xy

57

y

TA

yD

B xy


1

2

3

45

6

7

8

9

10

x

Normalized cuts

FcT

rE ||)(||),(0 WBAΓWBA

(Dhillon et al., ‘04, Zass & Shashua, 2005; Ding et al., 2005, De la Torre et al ‘06)

)(DΓ )](...)()([ 21 ndddΓ

Affinity Matrix

Normalized Cuts (Shi & Malik ’00)Ratio-cuts(Hagen & Kahng ’02)


Other Connections

• The LS-KRRR (E0) is also the generative model for:– Laplacian Eigenmaps, Locality Preserving projections, MDS,

Partial least-squaresPartial least squares, ….

• Benefits of LS framework:– Common framework to understand difference and communalities

between different CA methods (e.g. KPCA, KLDA, KCCA, Ncuts)– Better understanding of normalization factors and

generalizations– Efficient numerical optimization less than θ(n3) or θ(d3), where n


p ( ) ( )is number of samples and d dimensions


– Principal Component Analysis (PCA).– Non-negative Matrix Factorization (NMF).– Independent Component Analysis (ICA).

• Discriminative models– Linear Discriminant Analysis (LDA).– Oriented Component Analysis (OCA).– Canonical Correlation Analysis (CCA).

• A unifying view of CASt d d t i f li d l


• Standard extensions of linear models– Latent variable models.– Tensor factorization

32

Latent Variable Models


Factor Analysis• A Gaussian distribution on the coefficients and noise is

added to PCA Factor Analysis.

k NpNp BcμdBcdI0,ccηBcμd

),|(),|()|()(

(Mardia et al., 1979)

• Inference (Roweis & Ghahramani, 1999;Tipping & Bishop, 1999a)

TT

d

k

E

diagNppp

BBμdμdd

0,cημ,

)))((()cov(

),...,,()|()(),|(),|()|()(

21

),|()( Vmcd|c Np

),( dcp Jointly Gaussian


11

1

)()()(

),|()(

BBIVμdBBBm

|

T

TT

p

PCA reconstruction low error.FA high reconstruction error (low likelihood).

Ppca• If PPCA.• If is equivalent to PCA. TTTT BBBBBB 11 )()(0

dTE Iηη )(

0

• Probabilistic visual learning (Moghaddam & Pentland, 1997;)

2)(

2)(

1

21

2

21

21

2

)()()(21

21

2

)()(21

)2()2()2()2()()()(

2

1

211

kdk

ii

d

c

ddeeeedppp

k

i i

iTT

d

μdIBBμdμdμd T

ccc|dd

iT

i dBc


More on PPCA

• Tracking (Yang et al 1999; Yang et al 2000a; Lee et al 2005; de la Torre et(Tipping & Bishop, 1999b; Black et al., 1998; Jebara et al., 1998)

• Extension to mixtures of Ppca (mixture of subspaces).

Tracking (Yang et al., 1999; Yang et al., 2000a; Lee et al., 2005; de la Torre et al., 2000b)

• Recognition/Detection (Moghaddam et al., 2000; Shakhnarovich & Moghaddam, 2004; Everingham & Zisserman, 2006)

• PCA for the exponential family (collins et al., 2001)


33

Tensor Factorization


Tensor faces(Vasilescu & Terzopoulos, 2002; Vasilescu & Terzopoulos, 2003)

people

expressions


viewsilluminations

Eigenfaces• Facial images (identity change)

• Eigenfaces bases vectors capture the variability in facial appearance (do not decouple pose, illumination, …)


Data Organization• Linear/PCA: Data Matrix

– Rpixels x images

– a matrix of image vectorsD

Pixe

ls

ImagesD

• Multilinear: Data Tensor– Rpeople x views x illums x express x pixels

– N-dimensional matrix

Views

D

D


N dimensional matrix– 28 people, 45 images/person– 5 views, 3 illuminations,

3 expressions per personexilvpp ,,,iIl

lum

inat

ions

34

N-Mode SVD Algorithm

N = 3

pixelsxexpressxillums.xviews xpeoplex 51 UUUUU . ZD 432


PCA:

TensorFaces:


Strategic Data Compression = Perceptual Quality

• TensorFaces data reduction in illumination space primarily degrades illumination effects (cast shadows, highlights)

• PCA has lower mean square error but higher perceptual errorTensorFaces

Mean Sq. Err. = 409.153 illum + 11 people param.

33 basis vectors

PCA

Mean Sq. Err. = 85.7533 parameters

33 basis vectors

Original

176 basis vectors

TensorFaces

6 illum + 11 people param.66 basis vectors

• PCA has lower mean square error but higher perceptual error


Acknowledgments• The content of some of the slides has been taken from previous

presentations/papers of:– Ales Leonardis.– Horst BischofHorst Bischof.– Michael Black.– Rene Vidal.– Anat Levin.– Aleix Martinez.– Juha Karhunen.– Andrew Fitzgibbon.– Daniel Lee.– Chris Ding.


– M. Alex Vasilescu.– Sam Roweis.– Daoqiang Zhang.– Ammon Shashua.

35

CA

Thanks


Bibliography


Component Analysis for CV & PR F. De la Torre CVPR Easter School-2011 139 Component Analysis for CV & PR F. De la Torre CVPR Easter School-2011 140

36

Component Analysis for CV & PR F. De la Torre CVPR Easter School-2011 141 Component Analysis for CV & PR F. De la Torre CVPR Easter School-2011 142


Bibliography

Zhou F., De la Torre F. and Hodgins J. (2008) "Aligned Cluster Analysis for Temporal Segmentation of Human Motion“ IEEE Conference on Automatic Face and Gestures Recognition, September, 2008.

De la Torre, F. and Nguyen, M. (2008) “Parameterized Kernel Principal Component Analysis: Theory and Applications to Supervised and Unsupervised Image Alignment“ IEEE Conference on Computer Vision and Pattern Recognition, June, 2008.


Component Analysis for CV & PRcvpr11.cecs.anu.edu.au/files/australia_summer_school_4_Fernando.pdf · array processing (e.g. cocktail problem, eco cancellation), blind source Component

Documents