This lecture: models of shape and texture • Active Shape Models • Eigenfaces • Active Appearance Models • Class Projects 1 Automatic Erosion Detection in RA Monitoring | Georg Langs 1 1. Active Shape Models Adjust to texture Fit to shape model
This lecture: models of shape and texture
Active Shape Models Eigenfaces Active Appearance Models Class Projects
1 Automatic Erosion Detection in RA Monitoring | Georg Langs 1 1. Active Shape Models
Adjust to texture Fit to shape model
Example: bone contours
2 2
Task: localize anatomical structures
Task: Analyze a hand radiograph
3 Automatic Erosion Detection in RA Monitoring | Georg Langs
Task: Analyze a hand radiograph
4 Automatic Erosion Detection in RA Monitoring | Georg Langs
Assume: we are looking for proximal phalanx 2
PP3
PP2
PP4
PP5
MC2
MC3
MC4
MC5
MP5
MP4
MP3
MP2
Analyzing a hand radiograph
5 Automatic Erosion Detection in RA Monitoring | Georg Langs
We have a priori knowledge about the typical appearance: e.g. bone shapes and texture
PP2
How can we represent this knowledge? How can we exploit it?
How to capture a priori knowledge?
6 Automatic Erosion Detection in RA Monitoring | Georg Langs
Simple shape representation: shape consists of a set of points defined by their individual coordinates very powerful very general very unspecific
Alternative: capture common properties of the bone find a representation that is restricted to plausible bones.
Prior knowledge: first works
Snakes
Deformable templates
7
A.L.Yuille, D. Cohen, and P.Hallinan, Feature extraction from faces using deformable templates, CVPR 89, IJCV 92.
Data-driven
Prior knowledge
M. Kass, A. Witkin and D. Terzopoulos, `Snakes, ICCV 1987, IJCV 88
Current work: Statistical Shape Models
8 1. Active Shape Models
Each example is represented by a vector containing the coordinates of the landmarks.
Learning: Model Acquisition Inference: Model Fitting
Model learning
How can we learn what a bone looks like?
Objectives: Expressiveness (model can account for all bones) Compactness (as simple as possible)
Method: Principal Components Analysis (PCA)
9
To build a statistical model of the data we need many examples.
Bone shapes: vectors in
Goal: project data onto a low-dimensional linear
subspace that best explains their variation.
The space of all bone shapes
PCA: feature space
11 1. Active Shape Models
PCA: probability distribution
12 1. Active Shape Models
Example with high probability
Example with low probability
New subspace: `better coordinate system
13 1. Active Shape Models
New coordinates reflect the distribution of the data. Few coordinates suffice to represent a high dimensional vector They can be viewed as parameters of a model
Mean
Principal Component Analysis
Given: N data points x1, ,xN in Rd We want to find a new set of features that are linear
combinations of original ones:
u(xi) = uT(xi )
(: mean of data points)
What unit vector u in Rd captures the most variance of the data?
Principal Component Analysis Variance of projection on u:
Projection of data point
Covariance matrix of data
The direction that maximizes the variance: the eigenvector associated with the largest eigenvalue of
N
N
Direction: Unit norm vector
Principal component analysis
The direction that captures the maximum covariance of the data is the eigenvector corresponding to the largest eigenvalue of the data covariance matrix
Furthermore, the top k orthogonal directions that capture the most variance of the data are the k eigenvectors corresponding to the k largest eigenvalues
Covariance matrix:
Uncorrelated coordinates: diagonal covariance
Covariance matrix reminder
Height, Income Height, Weight
PCA: decorrelation of random variables
PCA: projection onto eigenvectors of covariance matrix
18
Dimensionality reduction by using only leading eigenvectors
PCA: step by step
1. Compute the empirical mean and subtract it from the data, and compute empirical covariance matrix
2. Compute eigenvalue decomposition of the covariance matrix resulting in
3. Retain only the k eigenvectors with highest corresponding eigenvalues.
1. Active Shape Models
PCA: step by step
4. Given a new input vector
5. project it into the eigenspace
6. resulting in the coefficient vector or parameter vector
1. Active Shape Models
Only first k eigenvectors
Using PCA to model shape
21 1. Active Shape Models
+ + + = +
Active shape models (ASM)
A set of training examples (images) A set of landmarks, that are present on
all images Build a statistical model of shape
variation (PCA) Build a statistical model of the local
texture (PCA) Use the model for the search in a new
image
22 1. Active Shape Models
ASM: model building
23
m Landmarks represented as
1. Active Shape Models
Training set: Instances vary in - shape - texture - orientation - scale - position
pose
ASM: alignment of examples
24 1. Active Shape Models
s1
s2
scale center (translate) rotate center of
gravity
The alignment removes rotation, scale and translation differences Minimizes Procrustes distance:
ASM: alignment of examples
Center all training shapes Choose one example as initial mean shape and scale
Align all shapes to by minimizing the Procrustes distance
Recalculate the mean shape from the aligned set.
Iterate until the mean shape converges
25 1. Active Shape Models
Given: a set of aligned shapes Perform PCA on the set of shapes, resulting in
a set of Eigenvectors and corresponding Eigenvalues
Each shape can be reconstructed by
ASM: shape model
26 1. Active Shape Models
ASM search
27 1. Active Shape Models
Adjust to texture Fit to shape model
Initialize
Update the instance according to shape model and pose
Initialize
Generate model pose instance Find pose parameters to map to Project into the model coordinates Update model parameters to match Apply constraints
ASM search: shape
28 1. Active Shape Models
ASM search
29 1. Active Shape Models 29
Modelling the appearance
To peform a search with the shape model, we have to capture the appearance, too. Local appearance at the landmark positions:
Active shape model (ASM)
Appearance of the enclosed texture Active appearance model (AAM)
30 Automatic Erosion Detection in RA Monitoring | Georg Langs
This lecture: models of shape and texture
Active Shape Models Eigenfaces Active Appearance Models Class Projects
31 Automatic Erosion Detection in RA Monitoring | Georg Langs
= +
+ w1u1+w2u2+w3u3+w4u4+ ^ x =
Face Recognition Problem
Face Recognition Problem
[Sinha and Poggio 2002]
Challenges: Image Variability
Expression Illumination Pose
Short Term
Facial Hair Makeup Eyewear
Hairstyle Piercings Aging
Long Term
Invariance Problem
Invariance Problem
Images as Points in Euclidean Space
Let an n-pixel image to be a point in an n-D space, x Rn.
Each pixel value is a coordinate of x.
x 1
x 2
x n
Face Recognition: Euclidean Distances
D2 > 0
D3 = 0
D1 > 0
~
Face Recognition: Euclidean Distances
D2 > 0
D3 > D1 or 2
D1 > 0
[Hallinan 1994] [Adini, Moses, and Ullman 1994]
Nearest Neighbor Classifier
Variations on distance function (e.g. L1, robust distances)
Multiple templates per class- perhaps many training images per class.
Expensive to compute k distances, especially when each image is big (N dimensional).
May not generalize well to unseen examples. Some solutions:
Bayesian classification Dimensionality reduction
Image Variability: Appearance Manifolds
x2
x1
xn
Lighting x Pose [Murase and Nayar 1993]
Image Variability: From Few to Many
x2
x1
xn
Lighting x Pose
[Georghiades, Belhumeur, and Kriegman 1999]
Real
Synthetic
Modeling Image Variability
Can we model illumination and pose variability in images of a face?
Yes, if we can determine the shape and texture of the face. But how?
Appearance modelling for faces When viewed as vectors of pixel values, face images are
extremely high-dimensional 100x100 image = 10,000 dimensions
Very few vectors correspond to valid face images
Original coordinates are not revealing about face properties We want to model the subspace of face images Same story as shape modelling.
Eigenfaces: Key idea Assume that most face images lie on
a low-dimensional subspace determined by the first k (k
Eigenfaces example Training images x1,,xN
Eigenfaces example Top eigenvectors: u1,uk
Mean:
Eigenfaces example Principal component (eigenvector) uk
+ 3kuk
3kuk
Eigenfaces example
Face x in face space coordinates:
=
Eigenfaces example
Face x in face space coordinates:
Reconstruction:
= +
+ w1u1+w2u2+w3u3+w4u4+
=
^ x =
Recognition with eigenfaces Process labeled training images:
Find mean and covariance matrix Find k principal components (eigenvectors of ) u1,uk Project each training image xi onto subspace spanned by
principal components: (wi1,,wik) = (u1T(xi ), , ukT(xi ))
Given novel image x: Project onto subspace:
(w1,,wk) = (u1T(x ), , ukT(x )) Check reconstruction error x x to determine whether image
is really a face Classify as closest training face in k-dimensional subspace
^
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
Challenges addressed by Eigenfaces
Expression Illumination Pose
Short Term
Facial Hair Makeup Eyewear
Hairstyle Piercings Aging
Long Term
Limitations
Global appearance method: not robust to misalignment, background variation
Linear Discriminant Analysis
We wish to choose linear functions of the features that allow good discrimination. Assume class-conditional covariances are the same Want linear feature that maximises the spread of class means for
a fixed within-class variance
This lecture: models of shape and texture
Active Shape Models Eigenfaces Active Appearance Models Class Projects
61 Automatic Erosion Detection in RA Monitoring | Georg Langs
Challenges: Image Variability
Expression Illumination Pose
Short Term
Facial Hair Makeup Eyewear
Hairstyle Piercings Aging
Long Term
DArcy Thompson, On Growth and Form (1917)
Modelling Shape Variability via Deformations
Modelling Shape Variability via Deformations
DArcy Thompson, On Growth and Form (1917)
Active Appearance Models
Appearance = Shape + Texture Shape: deformation from `template domain to
instance domain
Texture: intensity (or color) patch of an image in the template domain.
I(S (x)) = T (x)
X S (X )
T emplate Ins tance
Shape-free texture
An attempt to eliminate texture variation due to shape
Given instance I and a reference template T we warp our image so that points of I move into the corresponding points of T
warped to becomes
T I
AAM training
Training set: annotated grayscale images
Shape: displacements
from mean shape
Texture (shape-free)
Model of shape Model of texture
PCA PCA Optional: Interpolate to recover dense deformation field
Active Appearance Models
Shape:
Appearance:
Synthesis:
I(S (x)) = T (x)
X S (X )
T emplate Ins tance
AAM model Shape Eigenvectors
Texture Eigenvectors
Playing with the Parameters
First two modes of shape variation First two modes of gray-level variation
First four modes of appearance variation
Active Appearance Model fitting
Minimize reconstruction error
How? Alternate between estimating s and t
For t: projection of deformed image onto PCA basis For s?
Given: 1) an appearance model, 2) a new image, 3) a starting approximation
Find: the best matching synthetic image
AAM parameter estimation: shape
Iterative scheme (Newton-Raphson minimization)
72
Active Appearance Model Search (Results)
Model fitting + Image segmentation AAM fitting
Variational Segmentation
EM formulation E-step: Assign data to models M-step: Estimate model parameters
Joint segmentation & recognition E-step: segmentation M-step: deformable model fitting
M-step E-step Atomic Regions
Model fitting + Image segmentation
Challenges: Image Variability
Expression Illumination Pose
Short Term
Facial Hair Makeup Eyewear
Hairstyle Piercings Aging
Long Term
Modeling Image Variability: 3-D Faces
Laser Range Scanners Stereo Cameras Structured Light
Photometric Stereo
What can we do with 3d shape?
[Blanz and Vetter 1999, 2003]
Building a Morphable Face Model
[Blanz and Vetter 1999, 2003]
3-D Morphable Models
[Blanz and Vetter 1999, 2003]
3D Morphable models
Recover Shape
Synthesize new views
Synthesize new expressions
82
Rough manual initialization Gradient descent to minimize reconstruction
error functional
And then
3-D Morphable Models fitting
83
3D AAM for face tracking
CMU group: I. Matthews, S. Baker, R. Gross (230 Frames per second)
84
3D AAM for face tracking
Playing with Facial Attributes Several classes of attributes are modeled: Facial expressions (smile, frown) Individual characteristics (double chin, hooked
nose, maleness) Distinctiveness
Manipulating Facial Attributes via Deformations For each face in the database, two scans are recorded:
Sneutral, and Sexpression. The difference vector DS = Sexpression - Sneutral is saved and
later on simply added to the 3D reconstruction of the input image.
88