Recognition – PCA and Templates. Recognition Suppose you want to find a face in an imageSuppose you want to find a face in an image One possibility: look.

Recognition –Recognition –PCA and TemplatesPCA and Templates

RecognitionRecognition

• Suppose you want to find a face in an Suppose you want to find a face in an imageimage

• One possibility: look for something One possibility: look for something that looks sort of like a face (oval, dark that looks sort of like a face (oval, dark band near top, dark band near bottom)band near top, dark band near bottom)

• Another possibility: look for pieces of Another possibility: look for pieces of faces (eyes, mouth, etc.) in a specific faces (eyes, mouth, etc.) in a specific arrangementarrangement

RecognitionRecognition

• Suppose you want to recognize aSuppose you want to recognize aparticularparticular face face

• How does How does thisthis face differ from face differ from average faceaverage face

TemplatesTemplates

• Model of a “generic” or “average” Model of a “generic” or “average” faceface– Learn templates from example dataLearn templates from example data

• For each location in image, look for For each location in image, look for template at that locationtemplate at that location– Optionally also search over scale, Optionally also search over scale,

orientationorientation

TemplatesTemplates

• In the simplest case, based on intensityIn the simplest case, based on intensity– Template is average of all faces in Template is average of all faces in

training settraining set– Comparison based on e.g. SSDComparison based on e.g. SSD

• More complex templatesMore complex templates– Outputs of feature detectorsOutputs of feature detectors– Color histogramsColor histograms– Often combine position and frequency Often combine position and frequency

information (wavelets)information (wavelets)

Face Detection ResultsFace Detection Results

Sample ImagesSample Images

WaveletWaveletHistogramHistogramTemplateTemplate

Detection ofDetection offrontal / profilefrontal / profile

facesfaces

More Face Detection ResultsMore Face Detection Results

Schneiderman and KanadeSchneiderman and Kanade

How to Recognize Specific How to Recognize Specific People?People?

• Consider variation from average faceConsider variation from average face

• Not all variations equally importantNot all variations equally important– Variation in a single pixel relatively Variation in a single pixel relatively

unimportantunimportant

• If image is high-dimensional vector, If image is high-dimensional vector, want to find directions in this space want to find directions in this space along which variation is highalong which variation is high

Principal Components Principal Components AnalaysisAnalaysis

• Principal Components Analysis (PCA): Principal Components Analysis (PCA): approximating a high-dimensional approximating a high-dimensional data set with a lower-dimensional data set with a lower-dimensional subspacesubspace

Original axesOriginal axes

****

******

**** **

**

********

**

**

****** **

**** ******

Data pointsData points

First principal componentFirst principal componentSecond principal componentSecond principal component

Principal ComponentsPrincipal Components

• Computing PCA:Computing PCA:– Subtract out mean (“whitening”)Subtract out mean (“whitening”)– Find eigenvalues of covariance matrixFind eigenvalues of covariance matrix– Equivalently, compute SVD of data Equivalently, compute SVD of data

matrixmatrix

PCA on Faces: “Eigenfaces”PCA on Faces: “Eigenfaces”

AverageAveragefaceface

First principal componentFirst principal component

OtherOthercomponentscomponents

For all except average,For all except average,“gray” = 0,“gray” = 0,

“white” > 0,“white” > 0,““black” < 0black” < 0

Using PCA for RecognitionUsing PCA for Recognition

• Store each person as coefficients of Store each person as coefficients of projection onto first few principal projection onto first few principal componentscomponents

• Compute projections of target image, Compute projections of target image, compare to databasecompare to database

max

0iEigenfaceimage

i

iia

max

0iEigenfaceimage

i

iia

Recognition UsingRecognition UsingRelations Between TemplatesRelations Between Templates

• Often easier to recognize a small featureOften easier to recognize a small feature– e.g., lips easier to recognize than facese.g., lips easier to recognize than faces– For articulated objects (e.g. people), For articulated objects (e.g. people),

template for whole class usually template for whole class usually complicatedcomplicated

• So, identify small pieces and look for So, identify small pieces and look for spatial arrangementsspatial arrangements– Many false positives from identifying Many false positives from identifying

piecespieces

Graph MatchingGraph Matching

HeadHead

LegLegLegLeg

ArmArm

ArmArm

BodyBody

HeadHead

HeadHead

BodyBody

BodyBody

ArmArm

ArmArm LegLeg

LegLeg LegLeg

LegLeg

ModelModel Feature detection resultsFeature detection results

Graph MatchingGraph Matching

HeadHead

LegLegLegLeg

ArmArm

ArmArm

BodyBody

ConstraintsConstraints

Next to, right of

Next to, right ofNext to

, below

Next to, below

Graph MatchingGraph Matching

HeadHead

LegLegLegLeg

ArmArm

ArmArm

BodyBody

HeadHead

HeadHead

BodyBody

BodyBody

ArmArm

ArmArm LegLeg

LegLeg LegLeg

LegLeg

Combinatorial searchCombinatorial search

Graph MatchingGraph Matching

HeadHead

LegLegLegLeg

ArmArm

ArmArm

BodyBody

HeadHead

HeadHead

BodyBody

BodyBody

ArmArm

ArmArm LegLeg

LegLeg LegLeg

LegLeg

Combinatorial searchCombinatorial search

OKOK

Graph MatchingGraph Matching

HeadHead

LegLegLegLeg

ArmArm

ArmArm

BodyBody

HeadHead

HeadHead

BodyBody

BodyBody

ArmArm

ArmArm LegLeg

LegLeg LegLeg

LegLeg

Combinatorial searchCombinatorial search

Already assignedAlready assigned

Graph MatchingGraph Matching

HeadHead

LegLegLegLeg

ArmArm

ArmArm

BodyBody

HeadHead

HeadHead

BodyBody

BodyBody

ArmArm

ArmArm LegLeg

LegLeg LegLeg

LegLeg

Combinatorial searchCombinatorial search

ViolatesViolatesconstraintconstraint

Graph MatchingGraph Matching

• Large search spaceLarge search space– Heuristics for pruningHeuristics for pruning

• Missing featuresMissing features– Look for maximal consistent Look for maximal consistent

assignmentassignment

• Noise, spurious featuresNoise, spurious features

• Incomplete constraintsIncomplete constraints– Verification step at endVerification step at end