14 cv mil_the_pinhole_camera

Computer vision: models, learning and inference

Chapter 14 The pinhole camera

Please send errata to [email protected]

2

Structure

• Pinhole camera model• Three geometric problems• Homogeneous coordinates• Solving the problems– Exterior orientation problem– Camera calibration– 3D reconstruction

• Applications

2Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Motivation

3Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Sparse stereo reconstruction

Compute the depth at a set of sparse matching points

Pinhole camera

4Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Real camera image is inverted

Instead model impossible but more convenient virtual image

Pinhole camera terminology

5Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Normalized Camera

6Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

By similar triangles:

Focal length parameters

7Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Can model both• the effect of the distance to the focal plane• the density of the receptors

with a single focal length parameter f

In practice, the receptors may not be square:

So use different focal length parameter for x and y dirns

Focal length parameters

8Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Offset parameters

9Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Current model assumes that pixel (0,0) is where the principal ray strikes the image plane (i.e. the center)

• Model offset to center

• Finally, add skew parameter• Accounts for image plane being not exactly perpendicular to

the principal ray

Skew parameter

10Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Position w=(u,v,w)T of point in the world is generally not expressed in the frame of reference of the camera.

• Transform using 3D transformation

or

Position and orientation of camera

11Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Point in frame of reference of camera

Point in frame of reference of world

• Intrinsic parameters (stored as intrinsic matrix)

• Extrinsic parameters

Complete pinhole camera model

12Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

For short:

Add noise – uncertainty in localizing feature in image

Complete pinhole camera model

13Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Radial distortion

14Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

15

Structure

• Pinhole camera model• Three geometric problems• Homogeneous coordinates• Solving the problems– Exterior orientation problem– Camera calibration– 3D reconstruction

• Applications

15Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Problem 1: Learning extrinsic parameters (exterior orientation)

16Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Use maximum likelihood:

Problem 2 – Learning intrinsic parameters (calibration)

17Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Use maximum likelihood:

Calibration

18Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Use 3D target with known 3D points

Problem 3 – Inferring 3D points (triangulation / reconstruction)

19Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Use maximum likelihood:

Solving the problems

20Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• None of these problems can be solved in closed form

• Can apply non-linear optimization to find best solution but slow and prone to local minima

• Solution – convert to a new representation (homogeoneous coordinates) where we can solve in closed form.

• Caution! We are not solving the true problem – finding global minimum of wrong problem. But can use as starting point for non-linear optimization of true problem

21

Structure

• Pinhole camera model• Three geometric problems• Homogeneous coordinates• Solving the problems– Exterior orientation problem– Camera calibration– 3D reconstruction

• Applications

21Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Homogeneous coordinates

22Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Convert 2D coordinate to 3D

To convert back

Geometric interpretation of homogeneous coordinates

23Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Pinhole camera in homogeneous coordinates

24Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Camera model:

In homogeneous coordinates:

(linear!)

Pinhole camera in homogeneous coordinates

25Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Writing out these three equations

Eliminate l to retrieve original equations

Adding in extrinsic parameters

26Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Or for short:

Or even shorter:

27

Structure

• Pinhole camera model• Three geometric problems• Homogeneous coordinates• Solving the problems– Exterior orientation problem– Camera calibration– 3D reconstruction

• Applications

27Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Problem 1: Learning extrinsic parameters (exterior orientation)

28Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Use maximum likelihood:

29

Exterior orientation

29Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Start with camera equation in homogeneous coordinates

Pre-multiply both sides by inverse of camera calibration matrix

30

Exterior orientation

30Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Start with camera equation in homogeneous coordinates

Pre-multiply both sides by inverse of camera calibration matrix

31

Exterior orientation

31Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

The third equation gives us an expression for l

Substitute back into first two lines

32

Exterior orientation

32Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Linear equation – two equations per point – form system of equations

33

Exterior orientation

33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Minimum direction problem of the form , Find minimum of subject to .

To solve, compute the SVD and then set to the last column of .

34

Exterior orientation

34Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Now we extract the values of and from .

Problem: the scale is arbitrary and the rows and columns of the rotation matrix may not be orthogonal.

Solution: compute SVD and then choose .

Use the ratio between the rotation matrix before and after to rescale

Use these estimates for start of non-linear optimisation.

35

Structure

• Pinhole camera model• Three geometric problems• Homogeneous coordinates• Solving the problems– Exterior orientation problem– Camera calibration– 3D reconstruction

• Applications

35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Problem 2 – Learning intrinsic parameters (calibration)

36Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Use maximum likelihood:

Calibration

37Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

One approach (not very efficient) is to alternately

• Optimize extrinsic parameters for fixed intrinsic

• Optimize intrinsic parameters for fixed extrinsic

Then use non-linear optimization.

Intrinsic parameters

38Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Maximum likelihood approach

This is a least squares problem.

Intrinsic parameters

39Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

The function is linear w.r.t. intrinsic parameters. Can be written in form

Now solve least squares problem

40

Structure

• Pinhole camera model• Three geometric problems• Homogeneous coordinates• Solving the problems– Exterior orientation problem– Camera calibration– 3D reconstruction

• Applications

40Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Problem 3 – Inferring 3D points (triangulation / reconstruction)

41Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Use maximum likelihood:

Reconstruction

42Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Write jth pinhole camera in homogeneous coordinates:

Pre-multiply with inverse of intrinsic matrix

Reconstruction

43Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Last equations gives

Substitute back into first two equations

Re-arranging get two linear equations for [u,v,w]

Solve using >1 cameras and then use non-linear optimization

44

Structure

• Pinhole camera model• Three geometric problems• Homogeneous coordinates• Solving the problems– Exterior orientation problem– Camera calibration– 3D reconstruction

• Applications

44Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Depth from structured light

45Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Depth from structured light

46Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Depth from structured light

47Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Shape from silhouette

48Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Shape from silhouette

49Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Shape from silhouette

50Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Conclusion

51Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Pinhole camera model is a non-linear function who takes points in 3D world and finds where they map to in image

• Parameterized by intrinsic and extrinsic matrices

• Difficult to estimate intrinsic/extrinsic/depth because non-linear

• Use homogeneous coordinates where we can get closed from solutions (initial solns only)