© 2003 by Davi GeigerComputer Vision September 2003 L1.1 Face Recognition Recognized Person Face Recognition.

Computer Vision September 2003 L1.1© 2003 by Davi Geiger

Face Recognition

Recognized Person

Face Recognition


Recognized Person

Face Recognition

Query Image

Face Recognition• Definition:

– Given a database of labeled facial images: Recognize an individual from an image formed from new and varying conditions (pose, expression, lighting etc.)

• Sub-Problems:

– Representation:

• How do we represent images of faces?

• What information do we store?

– Classification:

• How do we compare stored information to a new sample?

– Search


Representation Shape Representation:

• Generalized cylinders, Superquadrics …

Appearance Based Representation

• Refers to the recognition of 3D objects from ordinary images.

• PCA – Eigenfaces, EigenImages

• Fisher Linear Discriminant


Image Representation

klkl

l

l

ii

i

iii

I

1)1(

1

21

.

.

...

....

kli

i

i

2

1

i

1

0

0

0

0

1

0

0

0

1

21

kliii

kli

i

i

2

1

10

10

001

Basis Matrix

vector of coefficients

• An image I is a point in dimensional space1klIR

lkIRI pixel 1

pix

el k

l

pixel 2

2550

255

255

1 klIRi..

..

.........

...

. ...... .

....


Toy Example-Dimensionality Reduction

• Consider a set of images of 1person under fixed viewpoint & N lighting condition Each image is made up of 3 pixels and pixel 1 has the same value as pixel 3 for all images

pixel 1

pixel 3

pixe

l 2

...

..

........

...

. ...... .

...

Nn1 and .s.t 31

3

2

1

nn

n

n

n

n ii

i

i

i

i

1

0

0

0

1

0

0

0

1

321 nnnn iiii

0

1

0

1

0

1

21 nn ii n

ni

ni

Bc

2

1

01

10

01

.

11

01

10

01

ci

D, data matrix

NN ccciii 2121

01

10

01

D, data matrix

C, coefficient matrix

DBC 1CBD

new

new

new

newnewiii

3

2

11

010

505 ..iBc


Idea: Eigenimages and PCA

pixel 1

pix

el k

l

pixel 2

2550

255

255 ..

..

.........

...

. ...... .

....• Eigenimages are eigenvectors of image

ensemble

• The Principle Behind Principal Component Analysis(1) (also called the “Hotteling Transform”(2) or the “Karhunen-Loeve Method”(3).)

Find an orthogonal coordinate system such that the correlation between different axis is

minimized.• Eigenvectors are typically computed using the

Singular Value Decomposition (SVD)

(1) I.T.Jolliffe; Principle Component Analysis; 1986(2) R.C.Gonzalas, P.A.Wintz; Digital Image Processing; 1987(3) K.Karhunen; Uber Lineare Methoden in der Wahrscheinlichkeits Rechnug; 1946

M.M.Loeve; Probability Theory; 1955


...

..

.........

...

. ...... .

...

PCA-Dimensionality Reduction• Consider the same set of images

• PCA chooses axis in the direction of highest variability of the data, maximum scatter

pixel 1

pixel 3

pixe

l 2

1st axis

2nd axis

Nn1 and .s.t 31321 nnT

nnnn iiiiii

• Each image is now represented by a vector of coefficients in a reduced dimensionality space.

ninc

|||

ccc

|||

Biii NN 2121

|||

|||

data matrix, D

D) of (svd TUSVD UB set

dentitythat such I BBBSB TT

TE

• B minimizes the following energy function

Computer Vision© 2003 by Davi Geiger

The Covariance Matrix

• Define the covariance (scatter) matrix of the input samples:

(where is the sample mean)

N

nnnT

1

T)Cov( μ)μ)(i(iSD

μi

μi

μi

μiμiμiS

N

NT

2

1

21

T))(( μDμDS T


PCA: Some Properties of the Covariance/Scatter Matrix

• The matrix ST is symmetric

• The diagonal contains the variance of each parameter

(i.e. element ST,ii is the variance in the i’th direction).

• Each element ST,ij is the co-variance between the two directions i and j, represents the level of correlation

(i.e. a value of zero indicates that the two dimensions are uncorrelated).


PCA: Goal Revisited• Look for: - B• Such that:

– [c1 … cN] = BT [i1 … iN] ...

and the correlation is minimized

OR

Cov(C) is diagonal Note that Cov(C) can be expressed via Cov(D) and B as :

CCT = BT (D)(D)T B


Selecting the Optimal B

How do we find such B ?

DDTbiibi

Bopt contains the eigenvectors of the covariance of D

Bopt = [b1|…|bd]

BBS T


SVD of a Matrix

D) of (svd TVUD

UB set ) of (svd TTT SUUDD 2

TT DDS

UB set


Data Reduction: Theory

• Each eigenvalue represents the the total variance in its dimension.

• Throwing away the least significant eigenvectors in Bopt means throwing away the least significant variance information !


PCA for Recognition-Eigenfaces

• Consider the same set of images

• PCA chooses axis in the direction of highest variability of the data

• Given a new image, , compute the vector of coefficients associated with the new basis

Tnew

Tnew BBiBc 1

Nn1 and .s.t 31321 nnT

nnnn iiiiii

pixel 1

pixel 3

pixe

l 2

1st axis

2nd axis

....

..

........

...

. ...... .

...

newi

• Next, compare a reduced dimensionality representation of against all coefficient vectors

•One possible classifier: nearest-neighbor classifier

newcnewi

Nnn 1 c

© 2002 by M. Alex O. Vasilescu


Data and Eigenfaces

• Each image below is a column vector in the basis matrix B

• Data is composed of 28 faces photographed under same lighting and viewing conditions



PCA for Recognition - EigenImages• Consider a set of images of 2 people under fixed viewpoint & N lighting condition

• Each image is made up of 2 pixels

1st axis

2nd axis

1st axis

2nd axis

• Reduce dimensionality by throwing away the axis along which the data varies the least

• The coefficient vector associated with the 1st basis vector is used for classifiction

• Possible classifier: Mahalanobis distance

• Each image is represented by one coefficient vector

• Each person is displayed in N images and therefore has N coefficient vectors

pixe

l 2

...

..

........

...

. ...... .

...

...

..

........

...

. ...... .

...person 1

person 2

pixel 1

pixe

l 2

...

..

........

...

. ...... .

... ...

..

........

...

. ...... .

...person 1

person 2

pixel 1



PIE Database (Weizmann)


EigenImages-Basis Vectors

• Each image bellow is a column vector in the basis matrix B

• PCA encodes encodes the variability across

images without distinguishing between variability in people, viewpoints and illumination



Fisher’s Linear Discriminant

• Objective: Find a projection which separates data clusters

Good separation

Poor separation


Fisher Linear Discriminant

• The basis matrix B is chosen in order to maximize ratio of the determinant between class scatter matrix of the projected samples to the determinant within class scatter matrix of the projected samples

• B is the set of generalized eigenvectors of SBtw and SWin corresponding with a set of decreasing eigenvalues

BSB

BSBB

Bin

T

btwT

maxarg

BSBS withinbtw



FLD: Data Scatter

• Within-class scatter matrix

• Between-class scatter matrix

• Total scatter matrix

C

c

Tcncn

DW

cn1))(( iiS

i

C

c

TcccB D

1))(( μμμμS

SSS BWT


Fisher Linear Discriminant• Consider a set of images of 2 people under fixed viewpoint & N lighting condition

pixel 1

pixe

l 2..

..

.........

...

. ...... .

... ...

..

........

...

. ...... .

...person 1

person 2

2nd axis

1st axis

• Each image is represented by one coefficient vector

• Each person is displayed in N images and therefore has N coefficient vectors


© 2003 by Davi GeigerComputer Vision September 2003 L1.1 Face Recognition Recognized Person Face Recognition.

Documents

pca pixel

images pixel

x1x1 x2x2 x x x x x

x pca e1e1 e2e2 x x

davi geigercomputer

coefficient matrix slide

x y1y1 y2y2 slide

maximum scatter pixel