Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision CS484: Introduction to Computer Vision Min H. Kim KAIST School of Computing Epipolar Geometry (Szeliski: 11)
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
CS484: Introduction to Computer Vision
Min H. KimKAIST School of Computing
Epipolar Geometry(Szeliski: 11)
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
SINGLE-VIEW GEOMETRY
2Acknowledgment: Many slides from James Hays, Derek Hoiem and Grauman & Leibe 2008 AAAI Tutorial
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Single-view geometry
3Odilon Redon, Cyclops, 1914
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Our goal: Recovery of 3D structure• Recovery of structure from one image is
inherently ambiguous
4
x
X?X?
X?
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Our goal: Recovery of 3D structure• Recovery of structure from one image is
inherently ambiguous
5
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Our goal: Recovery of 3D structure• Recovery of structure from one image is
inherently ambiguous
6
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Ames Room
7
http://en.wikipedia.org/wiki/Ames_room
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Our goal: Recovery of 3D structure• We will need multi-view geometry
8
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
3D-2D Projective mapping
9
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Recall: Pinhole camera model
10
• Principal axis: line from the camera center perpendicular to the image plane
• Normalized (camera) coordinate system:camera center is at the origin and the principal axis is the z-axis
Note that from this slide, the z-axis looks at the object to simplify equations by removing the sign!
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
(X ,Y ,Z )! ( f X / Z , f Y / Z )XYZ1
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
!
f Xf YZ
⎛
⎝
⎜⎜⎜
⎞
⎠
⎟⎟⎟=
f 0f 01 0
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
XYZ1
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
Recall: Pinhole camera model
11x = PX
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Principal point
12
• Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system).
• Normalized coordinate system: origin is at the principal point.• Image coordinate system: origin is in the corner.• How to go from normalized coordinate system to image coordinate
system?
px
py
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
(X ,Y ,Z )! ( f X / Z + px , f Y / Z + py )
XYZ1
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
!
f X + Z pxf Y + Z pyZ
⎛
⎝
⎜⎜⎜
⎞
⎠
⎟⎟⎟=
f px 0
f py 0
1 0
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
XYZ1
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
Principal point offset
13
principal point: ( px , py )
px
py
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Principal point offset
14
P =K I | 0⎡⎣ ⎤⎦K =
f pxf py
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
Camera matrix
principal point: ( px , py )
px
py
f X + Zpxf Y + ZpyZ
⎛
⎝
⎜⎜⎜
⎞
⎠
⎟⎟⎟=
f pxf py
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
1 01 01 0
⎡
⎣
⎢⎢⎢
⎤
⎦
⎥⎥⎥
XYZ1
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
Projection(extrinsics)
Camera(intrinsics)
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
K =
mxmy
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
f pxf py
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=
α x βxα y β y
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
Pixel coordinates
15
• mx pixels per meter in horizontal direction, my pixels per meter in vertical direction
1mx
× 1my
Pixel size:
pixels/m m pixelsUnits:
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
!Xcam = R !X- !C( )
Camera rotation and translation
16
• In general, the camera coordinate frame will be related to the world coordinate frame by a rotation and a translation
3D coords. of point in camera frame
3D coords. of camera center in world frame
3D coords. of a pointin world frame (nonhomogeneous)
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
!Xcam = R !X- !C( )
Xcam = R −R !C0 1
⎡
⎣⎢⎢
⎤
⎦⎥⎥
!X1
⎛
⎝⎜⎞
⎠⎟= R −R !C
0 1
⎡
⎣⎢⎢
⎤
⎦⎥⎥X
x =K I | 0⎡⎣ ⎤⎦Xcam =K R |−R !C⎡⎣ ⎤⎦X P =K R | t⎡⎣ ⎤⎦ , t = −R !C
Camera rotation and translation
17
In non-homogeneous (~ symbol)coordinates:
Note: C is the null space of the camera projection matrix (PC=0)
Xcam and X are homogeneous coordinates
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera parameters
18
• Intrinsic parameters– Principal point coordinates– Focal length– Pixel magnification factors– Skew (non-rectangular pixels)– Radial distortion
K =
mxmy
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
f pxf py
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=
α x βxα y β y
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Radial distortion
• To model lens distortion– Use above projection operation
instead of standard projection matrix multiplication19
2. Apply radial distortion
3. Apply focal length translate image center
Slide by Steve Seitz
1. Project to “normalized” image coordinates(where the center is the origin)
x, y, z( ) ′xn = x / z′yn = y / z
r 2 = ′xn2 + ′y 2
n
′xd = ′xn 1+ k1r2 + k2r
4( )′yd = ′yn 1+ k1r
2 + k2r4( )
′x = f ′xd + xc′y = f ′yd + yc
K =
f pxf py
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera parameters• Intrinsic parameters
– Principal point coordinates– Focal length– Pixel magnification factors– Skew (non-rectangular pixels)– Radial distortion
• Extrinsic parameters– Rotation and translation relative to world coordinate
system
20
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera Calibration (Affine Projection)
Source: D. Hoiem
λ xλ yλ
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=* * * ** * * ** * * *
⎡
⎣
⎢⎢⎢
⎤
⎦
⎥⎥⎥
XYZ1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
x =K R t⎡⎣
⎤⎦ X
PAffine projection
Also called “para-perspective”
How to find out?
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera calibration• Given n points with known 3D coordinates Xi
and known image projections xi, estimate the camera parameters
22
? P
Xi
xi
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera calibration: Linear method
23
λx i = PX i x i × PX i = 0 xiyi1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
×
PxTX i
PyTX i
PzTX i
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
= 0
0 −X iT yiX i
T
X iT 0 −xiX i
T
− yiX iT xiX i
T 0
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
PxPyPz
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
= 0
Two linearly independent equations
∵"a ×"b = "a
"b sinθ
? P
Xi
xi
P∈!3×4
Px ,y ,z ∈!4×1
X i ∈!4×1
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera calibration: Linear method
24
• p has 11 degrees of freedom (3x4=12 parameters but the last element is 1, but scale is arbitrary)
• One 2D/3D correspondence gives us two linearly independent equations• Homogeneous least squares• 6 correspondences (6x2=12) needed for a minimal solution
∴Ap = 0
0T X1T − y1X1
T
X1T 0T −x1X1
T
! ! !0T Xn
T − ynXnT
XnT 0T −xnXn
T
⎡
⎣
⎢⎢⎢⎢⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥⎥⎥⎥⎥
PxPyPz
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
= 0
Suppose we have n points’ coordinates
A p
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera calibration: Linear method• Advantages: easy to formulate and solve• Disadvantages
– Doesn’t directly tell you camera parameters– Doesn’t model radial distortion– Can’t impose constraints, such as known focal length and
orthogonality
• Non-linear methods are preferred– Define error as difference between projected points and
measured points– Minimize error using Newton’s method or other non-line
ar optimization
Source: D. Hoiem
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
EPIPOLAR GEOMETRY
27Acknowledgment: Many slides from James Hays, Derek Hoiem and Grauman & Leibe 2008 AAAI Tutorial
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Triangulation• Given projections of a 3D point in two or more
images (with known camera matrices), find the coordinates of the point
28
O1 O2
x1x2
X?
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Triangulation• We want to intersect the two visual rays
corresponding to x1 and x2, but because of noise and numerical errors, they don’t meet exactly
29
O1 O2
x1x2
X?
R1R2
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Triangulation: Geometric approach• Find shortest segment connecting the two viewing
rays and let X be the midpoint of that segment
30
O1 O2
x1x2
X
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Recap: Triangulation: Linear approach
31
x × PX =xy1
⎡
⎣
⎢⎢⎢
⎤
⎦
⎥⎥⎥×
PxTX
PyTX
PzTX
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=
0 −XT y XT
XT 0 −x XT
− y XT x XT 0
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
PxPyPz
⎛
⎝
⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟
= [x× ]PX
λ1x1 = P1Xλ2x2 = P2X
x1 × P1X = 0x2 × P2X = 0
[x1× ]P1X = 0[x2× ]P2X = 0
Cross product as matrix multiplication:
Pi ∈!3×4
X∈!4×1
x i ∈!3×1
λi ∈!1×1
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Recap: Triangulation: Linear approach
32
Two independent equations each in terms of three unknown entries of X
λ1x1 = P1Xλ2x2 = P2X
x1 × P1X = 0x2 × P2X = 0
[x1× ]P1X = 0[x2× ]P2X = 0
Pi ∈!3×4
X∈!4×1
x i ∈!3×1
λi ∈!1×1
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Triangulation: Nonlinear approach
minimize d 2(x1 ,P1X)+ d 2(x2 ,P2X)
33
• Find X that minimizes distances in the screen spaces
O1 O2
x1x2
X?
x’1
x’2
= ′x1 = ′x2
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Two-view geometry
34
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
• Epipolar Plane – plane containing baseline (1D family)• Epipoles = intersections of baseline with image planes = projections of the other camera center
• Baseline – line connecting the two camera centers
Epipolar geometry
35
X
x x’
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
• Epipolar Plane – plane containing baseline (1D family)• Epipoles= intersections of baseline with image planes = projections of the other camera center
• Epipolar Lines - intersections of epipolar plane with imageplanes (always come in corresponding pairs)
• Baseline – line connecting the two camera centers
Epipolar geometry
36
X
x x’
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Example: Converging cameras
37
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Example: Motion parallel to image plane
38
Motion parallel
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Example: Motion perpendicular to image plane
39
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Example: Motion perpendicular to image plane
40
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
e
e’
Example: Motion perpendicular to image plane
41
Epipole has same coordinates in both images.Points move along lines radiating from e: “Focus of expansion”
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Epipolar constraint
42
• If we observe a point x in one image, where can the corresponding point x’ be in the other image?
x x’
X
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
• Potential matches for x have to lie on the corresponding epipolar line l’.
• Potential matches for x’ have to lie on the corresponding epipolar line l.
Epipolar constraint
43
x x’
X
x’
X
x’
X
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Epipolar constraint example
44
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
X
Epipolar constraint: Calibrated case
45
• Assume that the intrinsic and extrinsic parameters of the cameras are known
• We can also set the global coordinate system to the coordinate system of the first camera. Then the projection matrix of the first camera is [I | 0].
x’x
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Epipolar constraint: Calibrated case
x x’
X
ˆ ′x = ′K -1 ′x = ′Xx =K-1x = XHomogeneous 2D point (3D ray towards X)in normalized screen coordinates
2D pixel coordinate (homogeneous)
3D scene point
3D scene point in 2nd
camera’s 3D coordinates
• We can multiply the projection matrix of each camera (and the image points) by the inverse of the calibration matrix to get normalized image coordinates
x x '
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Camera1: projectionmatrix is [ I | 0 ], same as theworld coordinate. Point X has coordinate in the camera1’s normalized image coordinates
Epipolar constraint: Calibrated case
47
R, tCamera2: projection matrix is [ R | t ].Point X has coordinate in the camera2’snormalized image coordinates
NOTE: R and t are the coordinate transformation. (t = -RO’ )e.g., X’ can be written RX + t in the camera2’s normalized image coordinate.
X (=X’ in the camera2 frame)
x x '
x x '
ˆ ′x = Rx + t
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
X’=RX + t
Epipolar constraint: Calibrated case
48
Those three vectors are coplanar.Writing in the camera2’s normalized image coordinates,the vectors , , and t are coplanar.(we only care about direction!)
Camera1: extrinsic matrix is [ I | 0 ]Point X has coordinate in the camera1’s normalized image coordinate
Camera2: extrinsic matrix is [ R | t ]Point X has coordinate in the camera2’snormalized image coordinate
t = -RO’
(x in the camera2 frame) Rx
x '
ˆ ′x = Rx + t ˆ ′x ⋅ t × Rx⎡⎣ ⎤⎦ = 0
Rx x '
x x '
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Essential Matrix(Longuet-Higgins, 1981)
Epipolar constraint: Calibrated case
49
′x ⋅[t × (Rx)]= 0 ′x TEx = 0 with E = [t× ]R
X
The vectors , and t are coplanar.
essential matrix gives us the relative rotation and translation
t
Rx x '
(x in the camera2 frame) Rx
x '
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
X
Epipolar constraint: Calibrated case
50
• forms the epipolar line l’ associated with ( )• forms the epipolar line l associated with ( )• Epipoles and satisfy that and • E is singular (rank two)• E has five degrees of freedom (rotation + translation)
′x ⋅[t × (Rx)]= 0 ′x TEx = 0 with E = [t× ]R
Ex E
T x ' x
x '∀p '∈ l ', p 'Ex = 0∀p∈ l, pET x ' = 0
x x '
e e ' Ee = 0 ET e ' = 0
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Epipolar constraint: Uncalibrated case
51
• The camera matrices K and K’ of the two cameras are unknown
• We can write the epipolar constraint in terms of unknown normalized coordinates :
• Here are screen coordinates
X
ˆ ˆT¢ =x Ex 0 x =Kx, ′x = ′K ˆ ′x
K =
f pxf py
1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
x and ˆ ′x
x’x
x and x '
x x '
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Epipolar constraint: Uncalibrated case
52
X
Fundamental Matrix(Faugeras and Luong, 1992)
x =K −1xˆ ′x = ′K −1 ′x
weak calibration
ˆ ˆT¢ =x Ex 0 1withT T- -¢ ¢= =x Fx K EK0 F
x’x
x x '
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Epipolar constraint: Uncalibrated case
53
• F x forms the epipolar line l’ associated with x (∀p’∈l’, p’Fx=0)• FTx’ forms the epipolar line l associated with x’ (∀p∈l, pFTx’=0)• Epipoles satisfy that F e = 0 and FTe’ = 0• F is singular (rank two)• F has seven degrees of freedom
X
ˆ ˆT¢ =x Ex 0 1withT T- -¢ ¢= =x Fx K EK0 F
x x x '
x’
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
The eight-point algorithm
54
• Meaning of error
sum of Euclidean distances between points x’iand epipolar lines Fxi (or points xi and epipolarlines FTx’i) multiplied by a scale factor
• Nonlinear approach:
minimize 2 2
1d ( , ) d ( , )
N
i i i ii=
¢ ¢é ù+ë ûå Tx Fx x F x
2
1( ) :
NTi i
i=
¢å x Fx
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
The eight-point algorithm
55
x = (u, v, 1)T, x’ = (u’, v’, 1)T
Minimize:
under the constraintF33 = 1
2
1( )
NTi i
i=
¢å x Fx
, where both x and x’ are screen coordinates.
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Example with eight-point algorithm
56
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Example with eight-point algorithm
57
• Poor numerical conditioning• Can be fixed by rescaling the data
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
The normalized eight-point algorithm
• Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels
• Use the eight-point algorithm to compute F from the normalized points
• Enforce the rank-2 constraint (for example, take SVD of F and throw out the smallest singular value)
• Transform fundamental matrix back to original units: if T and T’ are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is T’T F T
58(Hartley, 1995)
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
Comparison of estimation algorithms
59
8-point Normalized 8-point Nonlinear least squares
Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel
Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel
Lecturer: Min H. Kim (KAIST) CS484: Introduction to Computer Vision
From epipolar geometry to camera calibration
• Estimating the fundamental matrix is known as “weak calibration”
• If we know the calibration matrices of the two cameras, we can estimate the essential matrix: E = K’TFK
• The essential matrix gives us the relative rotation and translation between the cameras, or their extrinsic parameters
60