Top Banner
Computer vision: models, learning and inference Chapter 17 Models for shape Please send errata to [email protected]
72
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: 17 cv mil_models_for_shape

Computer vision: models, learning and inference

Chapter 17 Models for shape

Please send errata to [email protected]

Page 2: 17 cv mil_models_for_shape

2

Structure

2Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 3: 17 cv mil_models_for_shape

3

Motivation: fitting shape model

3Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 4: 17 cv mil_models_for_shape

What is shape?

• Kendall (1984) – Shape “is all the geometrical information that remains when location scale and rotational effects are filtered out from an object”

• In other words, it is whatever is invariant to a similarity transformation

4Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 5: 17 cv mil_models_for_shape

Representing Shape

• Algebraic modelling– Line:

– Conic:

– More complex objects? Not practical for spine.

5Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 6: 17 cv mil_models_for_shape

Landmark Points

6Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Landmark points can be thought of as discrete samples from underlying contour– Ordered (single continuous contour)– Ordered with wrapping (closed contour)– More complex organisation (collection of closed and open)

Page 7: 17 cv mil_models_for_shape

Snakes

• Provide only weak information: contour is smooth• Represent contour as N 2D landmark points

• We will construct terms for – The likelihood of observing an image x given

landmark points W. Encourages landmark points to lie on border in the image

– The prior of the landmark point. Encourages the contours to be smooth.

7Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 8: 17 cv mil_models_for_shape

Snakes

8Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Initialise contour and let it evolve until it grabs onto an object Crawls across the image – hence called snake or active contour

Page 9: 17 cv mil_models_for_shape

Snake likelihood

9Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Has correct properties (probability high at edges), but flat in regions distant from the contour. Not good for optimisation.

Page 10: 17 cv mil_models_for_shape

Snake likelihood (2)

10Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Compute edges (here using Canny) and then compute distance image – this varies smoothly with distance from the image

Page 11: 17 cv mil_models_for_shape

Prior

• Encourages smoothness

– Encourages equal spacing

– Encourages low curvature

11Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 12: 17 cv mil_models_for_shape

Inference

• Maximise posterior probability

• No closed form solution • Must use non-linear optimisation method• Number of unknowns = 2N

12Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 13: 17 cv mil_models_for_shape

Snakes

13Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Notice failure at nose – falls between points. A better model would sample image between landmark points

Page 14: 17 cv mil_models_for_shape

Inference

• Maximise posterior probability

• Very slow. Can potentially speed it up by changing spacing element of prior:

• Take advantage of limited connectivity of associated graphical model

14Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 15: 17 cv mil_models_for_shape

Relationships between models

15Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 16: 17 cv mil_models_for_shape

16

Structure

16Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 17: 17 cv mil_models_for_shape

Shape template model

• Shape based on landmark points• These points are assumed known• Mapped into the image by transformation• What is left is to find parameters of transformation• Likelihood is based on distance transform:

• No prior on parameters (but could do)

17Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 18: 17 cv mil_models_for_shape

Shape template model

18Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 19: 17 cv mil_models_for_shape

Inference

• Use maximum likelihood approach

• No closed form solution • Must use non-linear optimization• Use chain rule to compute derivatives

19Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 20: 17 cv mil_models_for_shape

Iterative closest points

20Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Find nearest edge point to each landmark point

• Compute transformation in closed form

• Repeat

Page 21: 17 cv mil_models_for_shape

21

Structure

21Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 22: 17 cv mil_models_for_shape

Statistical shape models

• Also called– Point distribution models– Active shape models (as they adapt to the image)

• Likelihood:

• Prior:

22Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 23: 17 cv mil_models_for_shape

Learning

• Usually, we are given the examples after they have been transformed

• Before we can learn the normal distribution we must compute the inverse transformation

• Procedure is called generalized Procrustes analysis

23Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 24: 17 cv mil_models_for_shape

Generalized Procrustes analysis

24Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Training data Before alignment After alignment

Page 25: 17 cv mil_models_for_shape

Generalized Procrustes analysisAlternately– Update all transformations to map landmark

points to current mean

– Update mean to be average of transformed values

Then learn mean and variance parameters.

25Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 26: 17 cv mil_models_for_shape

Inference

• Map inference:

• No closed form solution • Use non-linear optimisation• Or use ICP approach• However, many parameters, and not clear they are all

needed• more efficient to use subspace model

26Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 27: 17 cv mil_models_for_shape

Face model

27Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Three samples from learnt model for faces

Page 28: 17 cv mil_models_for_shape

Subspace shape model

• Generate data from model:

– is the mean shape– the matrix contains K

basis functions in it columns– is normal noise with covariance

• Can alternatively write

28Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 29: 17 cv mil_models_for_shape

Approximating with subspace

29Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Subspace model

Can approximate an vector w with a weighted sum of the basis functions

Surprising how well this works even with a small number of basis functions

Page 30: 17 cv mil_models_for_shape

Subspace shape model

30Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 31: 17 cv mil_models_for_shape

Probabilistic PCA

31Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Generative eq:

Probabilistic version:

Add prior:

Density:

Page 32: 17 cv mil_models_for_shape

Learning PPCA

32Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Learn parameters and from data ,where .

Learn mean:

Then set and compute eigen-decomposition

Choose parameters

Page 33: 17 cv mil_models_for_shape

Properties of basis functions

33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Learning of parameters based on eigen-decomposition:

Parameters

Notice that:• Basis functions in are orthogonal• Basis functions in are ordered

Page 34: 17 cv mil_models_for_shape

Learnt hand model

34Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 35: 17 cv mil_models_for_shape

Learnt spine model

35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Mean Manipulating first principal component

Manipulating second principal component

Page 36: 17 cv mil_models_for_shape

Inference

36Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

likelihood prior

To fit model to an image:

ICP Approach:

• Find closest points to current prediction• Update weightings h• Find closest points to current prediction• Update transformation parameters y

Page 37: 17 cv mil_models_for_shape

Inference

37Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

1. Update weightings h

If transformation parameters can be represented as a matrix A

2. Update transformation parameters y• Using one of closed form solutions

Page 38: 17 cv mil_models_for_shape

Fitting model

38Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Much better to use statistical classifier instead of just distance from edges

Page 39: 17 cv mil_models_for_shape

39

Structure

39Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 40: 17 cv mil_models_for_shape

3D shape models

40Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 41: 17 cv mil_models_for_shape

41

Structure

41Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 42: 17 cv mil_models_for_shape

Statistical models for shape and appearance

42Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 43: 17 cv mil_models_for_shape

Statistical models for shape and appearance

43Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

1. We draw a hidden variable from a prior

Page 44: 17 cv mil_models_for_shape

Statistical models for shape and appearance

44Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

1. We draw a hidden variable h from a prior2. We draw landmark points w from a subspace model3. We draw image intensities x.• Generate image intensities in standard template shape• Transform the landmark points (parameters y) • Transform the image to landmark points• Add noise

Page 45: 17 cv mil_models_for_shape

Shape and appearance model

45Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Shape model Intensity model Shape and intensity

Page 46: 17 cv mil_models_for_shape

Warping images

46Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Piecewise affine transformation

Triangulate image points using Delaunay triangulation.

Image in each triangle is warped by an affine transformation.

Page 47: 17 cv mil_models_for_shape

Learning

47Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Goal is to learn parameters :

Problem• We are given the transformed landmark points• We are given the warped and transformed images

Solution• Use Procrustes analysis to un-transform landmark points• Warp observed images to template shape

Page 48: 17 cv mil_models_for_shape

Learning

48Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Now have aligned landmark points w, and aligned images x, we can learn the simpler model:

Can write generative equation as:

Has the form of a factor analyzer

Page 49: 17 cv mil_models_for_shape

Inference

49Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Likelihood of observed intensities

To fit the model use maximum likelihood

This has the least squares form

Page 50: 17 cv mil_models_for_shape

Inference

50Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

This has the least squares form

Use Gauss-Newton method or similar

Where the Jacobian J is a matrix with elements

Page 51: 17 cv mil_models_for_shape

Statistical models for shape and appearance

51Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 52: 17 cv mil_models_for_shape

52

Structure

52Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 53: 17 cv mil_models_for_shape

Non-linear models

• The shape and appearance models that we have studied so far are based on the normal distribution

• But more complex shapes might need more complex distributions– Could use mixture of PPCAs or similar– Or use a non-linear subspace model

• We will investigate the Gaussian process latent variable model (GPLVM)

• To understand the GPLVM, first think about PPCA in terms of regression.

53Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 54: 17 cv mil_models_for_shape

PPCA as regression

PPCA model:

• First term in last equation looks like regression • Predicts w for a given h• Considering each dimension separately, get linear regression

54Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 55: 17 cv mil_models_for_shape

PPCA as regression

55Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Joint probability distribution

Regress 1st dimension against

hidden variable

Regress 2nd dimension against

hidden variable

Page 56: 17 cv mil_models_for_shape

Gaussian process latent variable model

• Idea: replace the linear regression model with a non-linear regression model

• As name suggests, use Gaussian process regression

• Implications– Can now marginalize over parameters m and F– Can no longer marginalize over variable h

56Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 57: 17 cv mil_models_for_shape

GPLVM as regression

57Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Joint probability distribution

Regress 1st dimension against

hidden variable

Regress 2nd dimension against

hidden variable

Page 58: 17 cv mil_models_for_shape

Learning

• In learning the Gaussian process regression model , we optimized the marginal likelihood of the data with respect to the parameter s2.

58Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 59: 17 cv mil_models_for_shape

Learning• In learning the GPLVM, we still optimized the

marginal likelihood of the data with respect to the parameter s2, but must also find the values of the hidden variables that we regress against .

• Use non-linear optimization technique

59Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 60: 17 cv mil_models_for_shape

Inference• To predict a new value of the data using a hidden variable

• To compute density

• Cannot be computed in closed from

60Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 61: 17 cv mil_models_for_shape

GPLVM Shape models

61Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 62: 17 cv mil_models_for_shape

62

Structure

62Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 63: 17 cv mil_models_for_shape

Articulated Models

63Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformations of parts applied one after each other

• Known as a kinematic chain

• e.g. Foot transform is relative to lower leg, which is relative to upper leg etc.

• One root transformation that describes the position of model relative to camera

Page 64: 17 cv mil_models_for_shape

Articulated Models

64Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 65: 17 cv mil_models_for_shape

Articulated Models

65Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• One possible model for an object part is a quadric

• Represents spheres, ellipsoids, cylinders, pairs of planes and others

• Make truncated cylinders by clipping with cylinder with pair of planes

• Projects to conic in the image

Page 66: 17 cv mil_models_for_shape

66

Structure

66Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Snakes• Template models• Statistical shape models• 3D shape models• Models for shape and appearance• Non-linear models

• Articulated models• Applications

Page 67: 17 cv mil_models_for_shape

3D morphable models

67Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 68: 17 cv mil_models_for_shape

3D morphable models

68Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 69: 17 cv mil_models_for_shape

3D morphable models

69Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 70: 17 cv mil_models_for_shape

3D body model

70Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 71: 17 cv mil_models_for_shape

3D body model applications

71Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Page 72: 17 cv mil_models_for_shape

Conclusions

72Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Introduced a series of models for shape

• Assume different forms of prior knowledge• Contour is smooth (snakes)• Shape is known, but not position (template)• Shape class is known (statistical models)• Structure of shape known (articulated model)

• Relates to other models• Based on subspace models• Tracked using temporal models