Computer vision: models, learning and inference Chapter 15 Models for transformations.

Computer vision: models, learning and inference

Chapter 15 Models for transformations

2

Structure

2Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

3

Transformation models

3Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Consider viewing a planar scene• There is now a 1 to 1 mapping between points

on the plane and points in the image• We will investigate models for this 1 to 1

mapping– Euclidean– Similarity– Affine transform– Homography

4

Motivation: augmented reality tracking

4Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

5

• Consider viewing a fronto-parallel plane at a known distance D.

• In homogeneous coordinates, the imaging equations are:

Euclidean Transformation

5Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

3D rotation matrix becomes 2D (in plane)

Plane at known distance D

Point is on plane (w=0)

6

Euclidean transformation

6Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Simplifying

• Rearranging the last equation

7

Euclidean transformation

7Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Simplifying

• Rearranging the last equation

8

Euclidean transformation

8Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Pre-multiplying by inverse of (modified) intrinsic matrix

9

Euclidean transformation

9Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

or for short:

Homogeneous:

Cartesian:

For short:

10

Similarity Transformation

10Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Consider viewing fronto-parallel plane at unknown distance D

• By same logic as before we have

• Premultiplying by inverse of intrinsic matrix

11

Similarity Transformation

11Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Simplifying:

Multiply each equation by :

12

Similarity Transformation

12Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Simplifying:

Incorporate the constants by defining:

13

Similarity Transformation

13Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Homogeneous:

Cartesian:

For short:

14

Affine Transformation

14Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Homogeneous:

Cartesian:

For short:

15

Affine Transform

15Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Affine transform describes mapping well when the depth variation within the

planar object is small and the camera is far away

When variation in depth is comparable to distance to object then the affine

transformation is not a good model. Here we need the homography

16

Projective transformation / collinearity / homography

16Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Start with basic projection equation:

Combining these two matrices we get:

17

Homography

17Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Homogeneous:

Cartesian:

For short:

18

Modeling for noise

18Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• In the real world, the measured image positions are uncertain.

• We model this with a normal distribution• e.g.

19

Structure

19Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

20

Learning and inference problems

20Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Learning – take points on plane and their projections into the image and learn transformation parameters

• Inference – take the projection of a point in the image and establish point on plane

21

Learning and inference problems

21Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

22

• Maximum likelihood approach

• Becomes a least squares problem

Learning transformation models

22Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

23

Learning Euclidean parameters

23Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Solve for transformation:

Remaining problem:

24

• This is an orthogonal Procrustes problem. To solve: • Compute SVD• And then set

Learning Euclidean parameters

24Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Has the general form:

25

Learning similarity parameters

25Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Solve for rotation matrix as before• Solve for translation and scaling factor

26

Learning affine parameters

26Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Affine transform is linear

• Solve using least-squares solution

27

Learning homography parameters

27Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Homography is not linear – cannot be solved in closed form. Convert to other homogeneous coordinates

• Both sides are 3x1 vectors; should be parallel, so cross product will be zero

28

• Write out these equations in full

• There are only 2 independent equations here – use a minimum of four points to build up a set of equations

Learning homography parameters

28Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

29

• These equations have the form , which we need to solve with the constraint

• This is a “minimum direction” problem– Compute SVD– Take last column of

• Then use non-linear optimization

Learning homography parameters

29Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

30

Inference problems

30Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Given point x* in image, find position w* on object

31

In the absence of noise, we have the relation , or in homogeneous coordinates

Inference

31Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

To solve for the points, we simply invert this relation

32

Structure

32Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

33

Problem 1: exterior orientation

33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

34

Problem 1: exterior orientation

34Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Writing out the camera equations in full

• Estimate the homography from matched points• Factor out the intrinsic parameters

35

Problem 1: exterior orientation

35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• To estimate the first two columns of rotation matrix, we compute this singular value decomposition

• Then we set

36

Problem 1: exterior orientation

36Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Find the last column using the cross product of first two columns

• Make sure the determinant is 1. If it is -1, then multiply last column by -1.

• Find translation scaling factor between old and new values

• Finally, set

37

Structure

37Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

38

Problem 2: calibration

38Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

39

Structure

39Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

One approach (not very efficient) is to alternately

• Optimize extrinsic parameters for fixed intrinsic

• Optimize intrinsic parameters for fixed extrinsic

Calibration

40Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Then use non-linear optimization.

41

Structure

41Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

42

Problem 3 - reconstruction

42Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Transformation between plane and image:

Point in frame of reference of plane:Point in frame of reference of camera

43

Structure

43Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

44

Transformations between images

44Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• So far we have considered transformations between the image and a plane in the world

• Now consider two cameras viewing the same plane

• There is a homography between camera 1 and the plane and a second homography between camera 2 and the plane

• It follows that the relation between the two images is also a homography

45

Properties of the homography

45Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Homography is a linear transformation of a ray

Equivalently, leave rays and linearly transform image plane – all images formed by all planes that cut the same ray bundle are related by homographies.

46

Camera under pure rotation

46Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Special case is camera under pure rotation.Homography can be showed to be

47

Structure

47Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

48

Robust estimation

48Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Least squares criterion is not robust to outliers

For example, the two outliers here cause the fitted line to be quite wrong.

One approach to fitting under these circumstances is to use RANSAC – “Random sampling by consensus”

49

RANSAC

49Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

50

RANSAC

50Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

51

Fitting a homography with RANSAC

51Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Original images Initial matches Inliers from RANSAC

52

Piecewise planarity

52Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Many scenes are not planar, but are nonetheless piecewise planarCan we match all of the planes to one another?

53

Approach 1 – Sequential RANSAC

53Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Problems: greedy algorithm and no need to be spatially coherent

54

Approach 2 – PEaRL (propose, estimate and re-learn)

54Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Associate label l which indicates which plane we are in• Relation between points xi in image1 and yi in image 2

• Prior on labels is a Markov random field that encourages nearby labels to be similar

• Model solved with variation of alpha expansion algorithm

55

Approach 2 – PEaRL

55Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

56

Structure

56Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Transformation models• Learning and inference in transformation models• Three problems– Exterior orientation– Calibration– Reconstruction

• Properties of the homography• Robust estimation of transformations• Applications

57

Augmented reality tracking

57Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

58

Fast matching of keypoints

58Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

59

Visual panoramas

59Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

60

Conclusions

• Mapping between plane in world and camera is one-to-one

• Takes various forms, but most general is the homography

• Revisited exterior orientation, calibration, reconstruction problems from planes

• Use robust methods to estimate in the presence of outliers

60Computer vision: models, learning and inference. ©2011 Simon J.D. Prince