1 Image Analysis PCA and Eigenfaces Christophoros Nikou [email protected]PCA and Eigenfaces University of Ioannina - Department of Computer Science Images taken from: D. Forsyth and J. Ponce. Computer Vision: A Modern Approach, Prentice Hall, 2003. Computer Vision course by Svetlana Lazebnik, University of North Carolina at Chapel Hill. Computer Vision course by Michael Black, Brown University. Research page ofAntonio Torralba, MIT. 2 Face detection and recognition C. Nikou – Image Analysis (T-14)
22
Embed
10 PCA and Eigenfaces - cs.uoi.grcnikou/Courses/Image_Analysis/2011-2012/10_PCA_… · (PCA). • We want to construct a low-dimensional linear subspace that C. Nikou – Image Analysis
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
University of Ioannina - Department of Computer Science
Images taken from:D. Forsyth and J. Ponce. Computer Vision: A Modern Approach, Prentice Hall, 2003.Computer Vision course by Svetlana Lazebnik, University of North Carolina at Chapel Hill.Computer Vision course by Michael Black, Brown University.Research page of Antonio Torralba, MIT.
P. Viola and M. Jones. Robust real-time face detection. International Journal of Computer Vision, 57(2), 2004. Also in CVPR 2001.
M. Turk and A. Pentland. "Eigenfaces for recognition". Journal of Cognitive Neuroscience, 3(1):71–86, 1991. Also in CVPR 1991.
8 The space of all face images
• When viewed as vectors of pixel values, face images are extremely high-dimensional
– 100x100 image = 10,000 dimensions
• However, relatively few 10,000-dimensional vectors correspond to valid face
C. Nikou – Image Analysis (T-14)
correspond to valid face images.
• Is there a compact and effective representation of the subspace of face images?
5
9 The space of all face images
• Insight into principal t l icomponent analysis
(PCA).• We want to construct
a low-dimensional linear subspace that
C. Nikou – Image Analysis (T-14)
linear subspace that best explains the variation in the set of face images.
10 Principal Component Analysis
• Given: N data points x1, … ,xN in Rd
• We want to find a new set of features that are linear combinations of the original ones:
w(xi) = uT(xi – µ)( : mean of data points)
C. Nikou – Image Analysis (T-14)
(µ: mean of data points)
• What unit vector u in Rd captures the most variance of the data?
6
11Principal Component Analysis
(cont.)• The variance of the projected data:
Projection of data point
( ) ( )1 1
1 1var ( ) ( ) ( ) ( ) ( )N N TT T T
i i i i ii i
w w wN N= =
= = − −∑ ∑x x x u x μ u x μ
1 1( )( ) ( )( )N N
T T Ti i i i
⎡ ⎤= − − = − −⎢ ⎥∑ ∑u x μ x μ u u x μ x μ u
C. Nikou – Image Analysis (T-14)
Covariance matrix of data
1 1
( )( ) ( )( )i i i ii iN N= =
⎢ ⎥⎣ ⎦∑ ∑u x μ x μ u u x μ x μ u
T= u Σu
12Principal Component Analysis
(cont.)• We now estimate vector u maximizing the variance:
Tu Σusubject to: 2|| || 1T = =u u ubecause any multiple of u maximizes the objective function.
• The Lagrangian is ( ; ) (1 )T TJ λ λ= + −u u Σu u u
u Σu
C. Nikou – Image Analysis (T-14)
The Lagrangian isleading to the solution:
( ; ) (1 )J λ λ= +u u Σu u uλ=Σu u
which is an eigenvector of Σ. The one maximizing J corresponds to the largest eigenvalue of Σ.
7
13Principal Component Analysis
(cont.)• The direction that captures the maximum
covariance of the data is the eigenvector gcorresponding to the largest eigenvalue of the data covariance matrix.
• The top k orthogonal directions that
C. Nikou – Image Analysis (T-14)
capture the most variance of the data are the k eigenvectors corresponding to the klargest eigenvalues.
14 Eigenfaces: Key idea
• Assume that most face images lie on a low-dimensional subspace determineda low dimensional subspace determined by the first k (k<d) directions of maximum variance.
• Use PCA to determine the vectors or “eigenfaces” u1,…uk that span that
b
C. Nikou – Image Analysis (T-14)
subspace.• Represent all face images in the dataset
as linear combinations of eigenfaces.
8
15 Eigenfaces example
Training images
C. Nikou – Image Analysis (T-14)
16 Eigenfaces example (cont.)Top eigenvectors
Mean: μ
C. Nikou – Image Analysis (T-14)
9
17 Eigenfaces example (cont.)
uk
3μ - uk kλ
C. Nikou – Image Analysis (T-14)
3μ uk kλ+
18 Eigenfaces example (cont.)
• Face x in “face space” coordinates:
( ) ( ) ( )TT T T⎡ ⎤
[ ]1 2
1 2
( ), ( ),..., ( )
, ,...,
x u x μ u x μ u x μT T Tk
Tkw w w
⎡ ⎤→ − − −⎣ ⎦
=• Reconstruction:
C. Nikou – Image Analysis (T-14)
+=
+x̂ = μ 1 1 2 2 ...u u uk kw w w+ + +
10
19 Eigenfaces example (cont.)
• Any face of the training set may be expressed as a linear combination of a pnumber of eigenfaces.
• In matrix-vector form:
[ ]1
21 2x μ u u uk
ww⎡ ⎤⎢ ⎥⎢ ⎥= +⎢ ⎥
1=K
ii kk KE
λ
λ= +∑
∑
Error:
C. Nikou – Image Analysis (T-14)
+=
[ ]1 2μ k
kw⎢ ⎥⎢ ⎥⎣ ⎦
1
Kiiλ
=∑
20 Eigenfaces example (cont.)
Intra-personal subspace (variations in expression)
C. Nikou – Image Analysis (T-14)
Extra-personal subspace (variations between people)
11
21 Face Recognition with eigenfaces
Process the training images:• Find mean µ and covariance matrix Σ.• Find k principal components (eigenvectors of Σ) u u• Find k principal components (eigenvectors of Σ) u1,…uk.• Project each training image xi onto subspace spanned by
principal components:(wi1,…,wik) = (u1
T(xi – µ), … , ukT(xi – µ))
Given a new image x:• Project onto subspace:
T T
C. Nikou – Image Analysis (T-14)
(w1,…,wk) = (u1T(x – µ), … , uk
T(x – µ)).• Optional: check the reconstruction error x – x to determine
whether the new image is really a face.• Classify as closest training face in k-dimensional subspace.
^
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
22 Computation of eigenvectors
• Given the d-dimensional vectorized images xi(i=1,2…,N), we form matrix X=[x1, x2,…, xN], d >>N.
• Covariance matrix:
1 TXXN
Σ =
• Σ has dimensions dxd which, for example, for a
C. Nikou – Image Analysis (T-14)
200x200 image is very large (40000x40000).• How do we compute its eigenvalues and
eigenvectors?
12
23 Computation of eigenvectors (cont.)
• We create the NxN matrix:1 TT X X1 TT X XN
=
• T has N eigevalues (γi) and eigenvectors (ei):
i i iTe eγ= 1 Ti i iX Xe e
Nγ⇔ =
1 Ti i iXX Xe Xe
Nγ⇔ =
C. Nikou – Image Analysis (T-14)
N N( ) ( )i i iXe Xeγ⇔ Σ =
• γi is an eigenvalue of Σ and Xei is the correspoding eigenvector.
24The importance of the mean image
and the variations
C. Nikou – Image Analysis (T-14)
Average of 100 of the images from the Caltech-101 dataset. Reproduced from the web page of A. Torralba http://web.mit.edu/torralba/www/gallery/gallery11.html
What is this?
13
25The importance of the mean image
and the variations (cont.)
C. Nikou – Image Analysis (T-14)
The context plays an important role. Is it a car or a person? Both blobs correspond to the same shape after a 90 degrees rotation.
Reproduced from the web page of A. Torralba.http://web.mit.edu/torralba/www/gallery/gallery11.html
26The importance of the mean image
and the variations (cont.)
C. Nikou – Image Analysis (T-14)
Average fashion model. Average of 60 aligned face images of fashion models.
Reproduced from the Perception Lab, University of St Andrews in Fife, UK. http://perception.st-and.ac.uk
14
27Application: MRF features for texture
representation• Each pixel is described by its 7x7 neighborhood
in every Lab channel.• This results in a 7x7x3 = 147-dimensional vector
per pixel.• Applying PCA to these vectors leads to a
compact 6- to 8-dimensional representation of natural images.
C. Nikou – Image Analysis (T-14)
• It captures more than 90% of the variation of the image pixels neighborhoods.
28Application: MRF features for texture
representation (cont.)
C. Nikou – Image Analysis (T-14)
15
29 Application: view-based modeling
• PCA on various views of a 3D object.
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
30Application: view-based modeling
(cont.)• Subspace of the first three PC (varying view angle).
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
16
31Application: view-based modeling
(cont.)• View angle recognition.
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
32Application: view-based modeling
(cont.)• Extension to more 3D objects with varying pose and
illumination.
C. Nikou – Image Analysis (T-14)
H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995.
17
33 Application: EigenheadsPrincipal components 1 and 2.
C. Nikou – Image Analysis (T-14)
34 Application: Eigenheads (cont.)Principal components 3 and 4.
C. Nikou – Image Analysis (T-14)
18
35 Limitations
• Global appearance method– not robust to misalignment, backgroundnot robust to misalignment, background
variation.
C. Nikou – Image Analysis (T-14)
36 Limitations (cont.)
• Projection may suppress important detail– The smallest variance directions may not be
unimportant. PCA assumes Gaussian data.
C. Nikou – Image Analysis (T-14)
The shape of this dataset is not well described by its principal components
19
37 Limitations (cont.)
• PCA does not take the discriminative task into account
– Typically, we wish to compute features that allow good discrimination.
• Projection onto the major axis can not separate the green from the red data.
C. Nikou – Image Analysis (T-14)
• The second principal component captures what is required for classification.
38 Canonical Variates
• Also called “Linear Discriminant Analysis”• A labeled training set is necessaryA labeled training set is necessary.• We wish to choose linear functions of the
features that allow good discrimination.– Assume class-conditional covariances are
the same.
C. Nikou – Image Analysis (T-14)
– We seek for linear feature maximizing the spread of class means for a fixed within-class variance.
20
39 Canonical Variates (cont.)
• We have a set of vectorized images xi(i=1,2…,N).( , , )
• The images belong to C categories each having a mean μj, ( j=1,2,…,C ).
• The mean of the class means:
1 C
C. Nikou – Image Analysis (T-14)
1
1j
jCμ μ
=
= ∑
40 Canonical Variates (cont.)
• Between class covariance:C
T( )( )TB i i i
i 1
S N μ μ μ μ=
= − −∑with Ni being the number of images in class i.
• Within class covariance:
C. Nikou – Image Analysis (T-14)
• Within class covariance:
( )( )iNC T
W k ,i i k ,i ii 1 k 1
S x xμ μ= =
= − −∑∑
21
41 Canonical Variates (cont.)
Using the same argument as in PCA (and graph-based segmentation) we seek vector u maximizing
which is equivalent to
maxT
BTu
W
u S uu S u
⎧ ⎫⎨ ⎬⎩ ⎭
max{ }TBu
u S u s.t. 1TWu S u =
C. Nikou – Image Analysis (T-14)
B WS u S uλ=
The solution is the top eigenvector of the generalized eigenvalue problem:
42 Canonical Variates (cont.)
C. Nikou – Image Analysis (T-14)
• 71 views of 10 objects at a variety of poses at a black background.• 60 images were used to determine a set of canonical variates.• 11 images were used for testing.
22
43 Canonical Variates (cont.)
C. Nikou – Image Analysis (T-14)
• The first two canonical variates. The clusters are tight and well separated (a different symbol per object is used).
• We could probably get a quite good classification.