Stereo Course web page: vision.cis.udel.edu/~cv April 11, 2003 Lecture 21.

Stereo

Course web page:vision.cis.udel.edu/~cv

April 11, 2003 Lecture 21

Announcements

• No class on Monday• Read Forsyth & Ponce, Chapter 11-

11.1 and Hartley & Zisserman, Chapter 10-10.2 on calculating the fundamental matrix for next Wednesday

• HW4 will be assigned next Wednesday and due Monday, April 28

Outline

• Epipolar geometry• Fundamental matrix• Depth recovery

Two-View Geometry

• The relationship of two views of a scene taken from different camera positions to one another

• Interpretations– “Stereo vision” generally means two

synchronized cameras or eyes capturing images

– Could also be two sequential views from the same camera in motion

• Assuming a static scene

Mapping Points between Images

• What is the relationship between the images x, x’ of the scene point X in two views?

• Intuitively, it depends on:– The rigid transformation between cameras

(derivable from the camera matrices P, P’)

– The scene structure (i.e., the depth of X)• Parallax: Closer points appear to move more

Example: Two-View Geometry

courtesy of F. Dellaert

x1x’1

x2x’2

x3 x’3

Is there a transformation relating the points xi to x’i ?

Epipolar Geometry

• Baseline: Line joining camera centers C, C’• Epipolar plane ¦: Defined by baseline and

scene point X

from Hartley& Zisserman

baseline

Epipolar Lines

• Epipolar lines l, l’: Intersection of epipolar plane ¦ with image planes

• Epipoles e, e’: Where baseline intersects image planes– Equivalently, the image in one view of the other camera center.

C C’

from Hartley& Zisserman

Epipolar Pencil

• As position of X varies, epipolar planes “rotate” about the baseline– This set of planes is called the epipolar pencil

• Epipolar lines “radiate” from epipole—this is the pencil of epipolar lines

from Hartley& Zisserman

Epipolar Constraint• Camera center C and image point define ray in 3-D space

that projects to epipolar line l’ in other view (since it’s on the epipolar plane)

• 3-D point X on this ray, so image of X in other view x’ must be on l’

• In other words, the epipolar geometry defines a mapping x ! l’ of points in one image to lines in the other

from Hartley& Zisserman

C C’

x’

Example: Epipolar Lines for Converging Cameras

from Hartley & ZissermanLeft view Right view

Intersection of epipolar lines = Epipole ! Indicates location of other camera center

Special Case: Translation Parallel to Image

Plane

Note that epipolar lines are parallel and corresponding points lie on correspond-ing epipolar lines (the latter is true for all kinds of camera motions)

Special Case: Translation along Optical Axis

• Epipoles coincide at focus of expansion

• Not the same (in general) as vanishing point of scene lines

from Hartley & Zisserman

The Fundamental Matrix F• Mapping of point in one image to epipolar

line in other image x ! l’ is expressed algebraically by the fundamental matrix F

• Write this as l’ = F x• Remember that the point-on-line

relationship is given by l’ ¢ Fx = 0, which can also be written as l’T

Fx = (Fx)T l’ =

0•F is 3 x 3, rank 2 (not invertible, in contrast

to homographies)– 7 DOF (homogeneity and rank constraint take

away 2 DOF)

line point

The Fundamental Matrix F• Since x’ is on l’, by the point-on-line definition we

know that x’T l’ = 0

• Combined with l’ = F x, we can thus relate corres-ponding points in the camera pair (P, P’) to each other with the following:

x’T F x = 0

• The fundamental matrix of (P’, P) is the transpose FT

from Hartley& Zisserman

x’

Deducing Epipoles from F• For any x (besides e), l’ = F x

contains e’, so e’T (F x) = 0. Thus, e’T

F = 0 ! FT e’ = 0 – Similarly, Fe = 0

• Solve systems Fe = 0, FTe’ = 0– Same as last step of DLT (recall that there we

were trying to solve Ap = 0)– E.g., to solve Fe = 0 in Matlab:

• Take SVD of F with [U,S,V] = svd(F)• Last column of V is the answer e

x’

from Hartley & Zisserman

Cross Products & Skew-Symmetric Matrices

• If a = (a1, a2, a3), then define

• With this we have the following identity:

Building F from KnownCamera Matrices P, P’

• P, P’ determine unique fundamental matrix by

F = [e’]£P’P+

– Pseudoinverse A+ (pinv in Matlab): Like inverse for non-square matrices (i.e., AA+A = A, etc.)

• Defined as A+ = (ATA)-1AT when inverse exists

• Can also get camera matrices from F– Canonical form: P = K[Id j 0], P’ = K’[R j t]– However, there is an ambiguity up to a projective

transformation H such that we actually get PH, P’H

The Essential Matrix E• Fundamental matrix when calibration

matrices K, K’ of the two cameras are known is called the essential matrix E:

• Allows computation of camera matrices P, P’ up to a scale ambiguity

Extracting Structure

• The key aspect of epipolar geometry is its linear constraint on where a point in one image can be in the other

• By correlation-matching pixels (or features) along epipolar lines and measuring the disparity between them, we can construct a depth map (scene point depth is inversely proportional to disparity)

View 1 View 2 Computed depth mapcourtesy of P. Debevec