Camera Projection Model
Camera Projection Model
3D Point Projection (Pixel Space)
O
Projection plane ( , , )X Y Z
Z
img x
Xu f p
Z
img y
Yv f p
Z
CCD sensor (mm) Image (pixel)
u vccd ccd( , )
u vimg img( , )
ccdw
ccdh
(0,0)
(0,0) imgw
imgh
x yp p( , )
img imgu v( , )
f
img xZu fX p Z
img yZv fY p Z
img
img
11
x
y
u f p X
v f p Y
Z
K
+
Camera intrinsic parameter : metric space to pixel space
Metric space Pixel space
2D Inverse Projection
O
Projection plane
img
img
11
x
y
u f p X
v f p Y
Z
K
Metric space
f
img imgu v( , )
Pixel space
u X
v Y
Z
K
img
-1img
1
u
vK
img
-1img
1
( , , )X Y Z
2D point == 3D ray
3D ray
The 3D point must lie in the 3D ray passing through the origin and 2D image point.
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
= ( , , )X x y z
Camera
C C C C= ( , , )X x y z
World
RCW SO(3)
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C CW W 3 W, det 1 R R I R• Orthogonal matrix
Degree of freedom?
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
= ( , , )X x y z
Camera
C C C C= ( , , )X x y z
World
RCW SO(3)
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C CW W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
rX
Yr
Zr
rX
Yr
Z X Y= r r r
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RCW SO(3)
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C CW W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
rX
Yr
Zr
rX
Yr
Z X Y= r r r
rX : camera x axis seen from the world coord.
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RCW SO(3)
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C CW W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
rX
Yr
Zr
rX
Yr
Z X Y= r r r
rX : camera x axis seen from the world coord.
Yr : camera y axis seen from the world coord. Zr
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RCW SO(3)
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C CW W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
1r
2r
3 1 2= r r r1r : world x axis seen from
the camera coord.
r1 2r 3r
Ground plane
Camera
3D world
Origin at world coordinate
Coordinate Transform (Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
RCW SO(3)
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
T
C C CW W 3 W, det 1 R R I R• Orthogonal matrix
• Right hand rule
1r
2r
3 1 2= r r r1r : world x axis seen from
the camera coord.
r1 2r 3r
2r : world y axis seen from the camera coord.
3r
x1 x2 x3
y1 y2 y3
z1 z2
C
z3
C
C1 1
1
x
y
x
y
u f p
v f p
f p r r r
f p r r r
r
X
Y
Zr r
X
Y
Z
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection (Pure Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
1r : world x axis seen from the camera coord.
r1 2r 3r
2r : world y axis seen from the camera coord.
K
Camera projection of world point:
3r
x1 x2 x3
y1 y2 y3
z1 z2
C
z3
C
C1 1
1
x
y
x
y
u f p
v f p
f p r r r
f p r r r
r
X
Y
Zr r
X
Y
Z
Ground plane
Camera
3D world
Origin at world coordinate
Camera Projection (Pure Rotation)
X
1C
WR
Coordinate transformation from world to camera:
Camera
World
x1 x2 x3
Cy1 y2 y3 W
z1 z
C
2 z3
= =
X XRX
r r r
r r r
r r r
1r : world x axis seen from the camera coord.
r1 2r 3r
2r : world y axis seen from the camera coord.
K
Camera projection of world point:
K
3r
Ground plane
Camera
3D world
Origin at world coordinate
Euclidean Transform=Rotation+Translation
X
C CW ,R t
World
= ( , , )X x y z
Ground plane
Camera
3D world
Origin at world coordinate
X
Camera
World
= ( , , )X x y zC C C C= ( , , )X x y z
C CW ,R t
Euclidean Transform=Rotation+Translation
Ground plane
Camera
3D world
Origin at world coordinate
X
Camera
World
= ( , , )X x y zC C C C= ( , , )X x y z
C CW ,R t
Euclidean Transform=Rotation+Translation
Ground plane
Camera
3D world
Origin at world coordinate
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C CW y1 y2 y3 y
z1 z2 z3 z
C = =1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C= ( , , )X x y z
C CW ,R t
Euclidean Transform=Rotation+Translation
C t
C twhere is the world orgin seen from camera.
Ground plane
Camera
3D world
Origin at world coordinate
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C CW y1 y2 y3 y
z1 z2 z3 z
C = =1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C= ( , , )X x y z
C CW ,R t
Euclidean Transform=Rotation+Translation
C t
C twhere is the world orgin seen from camera.
Ground plane
Camera
3D world
Origin at world coordinate
Geometric Interpretation
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C CW y1 y2 y3 y
z1 z2 z3 z
C = =1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C= ( , , )X x y z
C CW ,R t
C t
Rotate and then, translate.
C twhere is the world orgin seen from camera.
Ground plane
Camera
3D world
Origin at world coordinate
Geometric Interpretation
X
Coordinate transformation from world to camera:
Camera
x1 x2 x3 x
C CW y1 y2 y3 y
z1 z2 z3 z
C = =1
RX
XX t
r r r t
r r r t
r r r t
World
= ( , , )X x y zC C C C= ( , , )X x y z
C CW ,R t
Rotate and then, translate.
x1 x2 x3 x
CW y1 y2 y3 y
z1 z2 z3 z
C
1
= = 11
1
XXR CX
r r r -C
r r r -C
r r r -C
cf) Translate and then, rotate.
where is the camera location seen from world. C
C
C twhere is the world orgin seen from camera.
Projective Line
Point-Point in Image
1 1 0 au bv c
1 1( , )u v
1xx2
2 2( , )u v
2 2 0 au bv c
x lT1 0 x lT
2 0
1
1 1
1
x
u
v2
2 2
1
x
u
v
a
b
c
lwhere
A 2D line passing through two 2D points:
xl 0
x
T1
T2
0 0 A
null A l x x1 2=or
2x3
Two 2D lines in an image intersect at a 2D point:
Line-Line in Image
l xT1 0 l xT
2 0
lx 0
l
T1
T2
1 2= x l l
1
1 1
1
l
a
b
c
( , )u vx
2
2 2
2
l
a
b
c
1 1 1 0 a u b v c2 2 2 0 a u b v c
1
x
u
v1
1 1
1
l
a
b
c
where
2
2 2
2
l
a
b
c
2x3
0 0 A
null A
or
(1804,934)
Vanishing Point
11l
1l 2
Ground plane
1u
2u3u
4u
Parallel lines:
11 4 3 l u u 12 1 2 l u u
Vanishing Point
11l
Ground plane
4u
Parallel lines:
11 4 3 l u u 12 1 2 l u u
21 4 1 l u u 22 3 4 l u u
2l 1
l221l 2
1u
2u3u
Vanishing Point
11l
x1
Ground plane
4u
Parallel lines:
11 4 3 l u u 12 1 2 l u u
21 4 1 l u u 22 3 4 l u u
2l 1
l221l 2
1u
2u3u
x l l1 11 12
Vanishing points:
x l l2 21 22
x2
Vanishing Point
11l
1v
Ground plane
4u
Parallel lines:
11 4 3 l u u 12 1 2 l u u
21 4 1 l u u 22 3 4 l u u
2l 1
l221l 2
1u
2u3u
1 11 12 v l l
Vanishing points:
2 21 22 v l l
2v Vanishing line:
1 2 l v vl
Where was I (how high)?
Taken from my hotel room (6th floor) Taken from beach
Vanishing point
Vanishing point
Multiple vanishing point
Vanishing point
Vanishing point
Vanishing line: Horizon
Vanishing line for horizon
Keller Hall
Vanishing point
Vanishing point
What can vanishing line tell us about me? • Horizon • Camera pitch angle (looking down) • Camera roll angle (tilted toward right)
Vanishing line for horizon
Ground plane
Single Vanishing Point Camera
3D world
r1
r2
r3
v
0 0
0 0
1
1 0
Z
C1 2 3 W
0
0
1
0
v K r r r t
-13
K v r
3v Kr
Ground plane
Single Vanishing Point Camera
3D world
r1
r2
r3
v
0 0
0 0
1
1 0
Z
C1 2 3 W
0
0
1
0
v K r r r t
-13
K v r
3v Kr
K v
rK v
-1
3 -1 because r3 is a unit vector.
Z vanishing point tells us about the surface normal of the ground plane
Ground plane
Single Vanishing Point Camera
3D world
r1
r2
r3
v
0 0
0 0
1
1 0
Z
C1 2 3 W
0
0
1
0
v K r r r t
-13
K v r
3v Kr
K v
rK v
-1
3 -1 because r3 is a unit vector.
Z vanishing point tells us about the surface normal of the ground plane
Ground plane
Rotation ambiguity
Ground plane
Two Vanishing Points Camera
X
1
0
0
0
v K r r r t X
Kr
CX 1 2 3 W
1
0
1
0
0
Y
3D world
r1
r2
r3
Yv
vX
3D world
Ground plane
Two Vanishing Points Camera
X
1
0
0
0
CY 1 2 3 W
2
v K r r r t Y
Kr
v K r r r t X
Kr
CX 1 2 3 W
1
0
1
0
0
Y
3D world
r1
r2
r3
Yv
vX
3D world
Ground plane
Two Vanishing Points Camera
X
1
0
0
0
-1 -1
X Y1 2-1 -1
X Y
,
K v K v
r rK v K v
3 1 2 r r r : Orthogonality constraint
CY 1 2 3 W
2
v K r r r t Y
Kr0
1
0
0
Y
3D world
r1
r2
r3
v K r r r t X
Kr
C
X 1 2 3 W
1
Yv
vX
3D world
Single View Metrology
Cross Ratio
Ground plane
A B
C D
Reference points
Target points
AC ac a c
BC bc
BD bd b d
AD ad a db c
a b
c
d a' b' c'
d'
Cross ratio (perspective transformation invariant)
v
sh_f
jp_f
jp_h p2
p3
p1
p4
h_prime = norm(p1-p2); h_R = norm(p1-p3); h_prime_inf = norm(p4-p2); h_inf = norm(p4-p3); H = H_R * h_prime * h_prime_inf / h_R / h_inf
ComputeHeightFromCrossRatio.m
h
Rh
vZ
h
h
R
R
h
hH
h
hH
H = 1.6779 Ground truth: 1.7m
Homography
Perspective Transform (Homography)
I1 I2
x 11 12 13 x x
y 21 22 23 y y
31 32 11 1 1
H
v h h h u u
v h h h u u
h h
: General form of plane to plane linear mapping
Ground plane
Camera plane
u
Perspective Transform (Homography)
u K R t X
0
1
X =
X
Y
uGround plane Camera plane
1 2 30
1
u K r r r t
X
Y
1 2
1
u K r r t
X
Y
x
y
11
H
u X
u Y
Perspective Transform (Homography)
I1 I2Invariant properties
• Cross ratio
• Concurrency
• Colinearity
x 11 12 13 x x
y 21 22 23 y y
31 32 11 1 1
H
v h h h u u
v h h h u u
h h
Degree of freedom
8 (9 variables 1 scale)
Hierarchy of Transformations
Euclidean (3 dof) Similarity (4 dof) Affine (6 dof) Projective (8 dof)
• Length • Angle • Area
• Length ratio • Angle
• Parallelism • Ratio of area • Ratio of length
• Cross ratio • Concurrency • Colinearity
x
y
cos sin
sin cos
1
t
t
θ θ
θ θ
x
y
cos sin
sin cos
1
t
t
θ θ
θ θ
11 12 13
21 22 23
0 0 1
a a a
a a a11 12 13
21 22 23
31 32 1
h h h
h h h
h h
Image Transform via Plane
Keller entrance left Keller entrance right
Plane
Image Transform via 3D Plane
X
YX =
0
1
1 2
1 2
=
1
=
1
u K r r t
v K r r t
X
Y
X
Y
u
u K R t X
v
How are two image coordinates (u,v) related?
-1-1 -1 -1
1 2 1 2
-1 -11 2 1 2
= =
1
=
=
r r t K u r r t K v
v K r r t r r t K u
v Hu
X
Y
Camera
Image Transform by Pure 3D Rotation
X
Camera
Image Transform by Pure 3D Rotation
X
2 = v KRX
1 = u KX
-1 T -11 2= = X K u R K v
-1=v KRK u
-1=H KRK-1=R K HK
Homography Computation
x 11 12 13 x
y 21 22 23 y
31 32 33
=
1 1
v h h h u
v h h h u
h h h
Homography Computation
x 11 12 13 x
y 21 22 23 y
31 32 33
=
1 1
v h h h u
v h h h u
h h h
11 12 13
31 32 33
21 22 23
31 3
x y
x
x y
x y
y
x y2 33
+
+ +
+
+ +
u uv
u u
u uv
h h h
h h h
h
h u
h
h hu
h
Homography Computation
x 11 12 13 x
y 21 22 23 y
31 32 33
=
1 1
v h h h u
v h h h u
h h h
11 12 13
31 32 33
21 22 23
31 3
x y
x
x y
x y
y
x y2 33
+
+ +
+
+ +
u uv
u u
u uv
h h h
h h h
h
h u
h
h hu
h
x y x x y x x
x y
11 12 13 31 32 33
21 22 23 31 32 3x y y y y3
+ + + 0
+ + + + 0
h h h h h h
h
u u u v u v v
u u u v u vh h vh h h
Unknowns: 11 33, ,h h
Equations: 2 per correspondence
Homography Computation
x 11 12 13 x
y 21 22 23 y
31 32 33
=
1 1
v h h h u
v h h h u
h h h
11 12 13
31 32 33
21 22 23
31 3
x y
x
x y
x y
y
x y2 33
+
+ +
+
+ +
u uv
u u
u uv
h h h
h h h
h
h u
h
h hu
h
11 12 13 31 3x y x x y x x
x y x
2 33
21 22 23 31 32 33y y y y
+ 0
+ 0
h h h h h h
h
u u u v u v v
u u u v u vh h vh h h
11
12
13
21
22
x y x x y x x
x y x y y y y23
31
32
33
1 0
1 0
h
h
h
h
h
h
u u -u v -u v -v
u u -u v -u v
h
-v
h
h
Homography Computation
x 11 12 13 x
y 21 22 23 y
31 32 33
=
1 1
v h h h u
v h h h u
h h h
11 12 13
31 32 33
21 22 23
31 3
x y
x
x y
x y
y
x y2 33
+
+ +
+
+ +
u uv
u u
u uv
h h h
h h h
h
h u
h
h hu
h
11 12 13 31 3x y x x y x x
x y x
2 33
21 22 23 31 32 33y y y y
+ 0
+ 0
h h h h h h
h
u u u v u v v
u u u v u vh h vh h h
11
12
13
21
22
x y x x y x x
x y x y y y y23
31
32
33
1 0
1 0
h
h
h
h
h
h
u u -u v -u v -v
u u -u v -u v
h
-v
h
h
A x 02x9
Homography Computation
x 11 12 13 x
y 21 22 23 y
31 32 33
=
1 1
v h h h u
v h h h u
h h h
11
12
13
21
22
x y x x y x x
x y x y y y y23
31
32
33
1 0
1 0
h
h
h
h
h
h
u u -u v -u v -v
u u -u v -u v
h
-v
h
h
A x 02x9
How many correspondences are needed?
Homography Computation
x 11 12 13 x
y 21 22 23 y
31 32 33
=
1 1
v h h h u
v h h h u
h h h
How many correspondences are needed? 4
11
12
13
21
22
x y x x y x x
x y x y y y y23
31
32
33
1 0
1 0
h
h
h
h
h
h
u u -u v -u v -v
u u -u v -u v
h
-v
h
h
A x 08x9
:end= nullx V A
Epipolar Geometry
X
u
Bob Alice
v
u v
Properties of Fundamental Matrix
Fundamental Matrix
P K I 0bob 3 3 1
P K R talice
• Transpose: if F is for Pbob, Palice, then FT is for Palice, Pbob.
• Epipolar line:
• Epipole:
• rank(F)=2: degree of freedom 9 (3x3 matrix)-1 (scale)-1 (rank)=7
ealiceebob
ul Fu vl F vT
ealiceebob
-T -1
F K t RK
rank 2 matrix
Fe 0bob = F e 0Talice =
u v
Bob Alice
Essential Matrix
Essential Matrix:
-T -1 -T -1
=
=
F F(R,t)
K t RK K EK
T=E K FKCalibrated fundamental matrix
=
E t Rwhere
Property of essential matrix:
T T
1
= 1
0
E UDV = U V
HW #4 RANSAC Fundamental matrix
v
Bob Alice
X
bobP aliceP
u
Baseline
Triangulation General camera pose
1 bob=1 1
u XP
Two 3D vectors are parallel.
bob1 1
u XP 0
bob1 1
u XP 0 : Knowns
: Unknowns
alice1
vP
mikeP
mike1
wP
PnP
u
X
3D-2D correspondence: u X
= P K R t (3)R SOwhere
11
12
13
14x x x x1 1 1 1 1 1 1
21y y y y1 1 1 1 1 1 1
22
23x x x x
24y y y y
31
32
33
34
1
1
1
1
m m m m m m m
m m m m m m m
p
p
p
pX Y Z -u X -u Y -u Z -u
pX Y Z -u X -u Y -u Z -u
p
pX Y Z -u X -u Y -u Z -u
pX Y Z -u X -u Y -u Z -u
p
p
p
p
0
0
0
0
x 11 12 13 14
31 32 33 34
+ +
+ +
p X p Y p Z pu
p X p Y p Z py 21 22 23 24
31 32 33 34
+ +
+ +
p X p Y p Z pu
p X p Y p Z p
2mx12
Bundle Adjustment
Nonlinear System
E
x
T T( ) ( )( )
x xx b
x x
f ff
Find x such that the following equation is satisfied:
How?
Strategy: Given x, move such that x
x x x
( ) ( ) x x xE E
Taylor expansion:
T T( ) ( ) ( )( )
x x xx x b
x x x
f f ff
T T( ) ( ) ( )
( )
x x xx b x
x x x
f f ff
= x TA bTA A-1
Cf.)
( )( ) ( ) H.O.T.
xx x x x
x
ff f
X
bobP
uu
Point Jacobian
-1
T T( ) ( ) ( )( )
x x xx b x
x x x
f f ff
X
Black: given variables Red: unknowns
XKR -C
u
v
w
where
( )
Xf
u
w
v
w
2 2
geom
Ev
wx y
u
w
2
2
( )
X XX
X X
X X
u ww u
u
w w
v u wv v
ww
f
KR
X
u
v
w
-1
T T( ) ( ) ( )( )
x x xx b x
x x x
f f ff
2
2
( )
X XX
X X
X X
u ww u
u
w w
v u wv v
ww
f
Damping factor (Levenberg-Marquardt algorithm)
-1
T T( ) ( ) ( )( )
x x xx I b x
x x x
f f ff
X
bobP
uu
Camera Jacobian
p
-1
T T( ) ( ) ( )( )
p p pp b p
p p p
f f ff
Black: given variables Red: unknowns
X CK -R
u
v
w
where
2 2
geom
Ev
wx y
u
w
-KC
R
u
v
w
11 1 3 13
1 3 22 23
1 3 1 3
- -
- -
-
K X C 0 K X C
0 K X C K X CR
0 0 X C
u
v
w
1
2
3
=
C
rp
r
r
3+9 parameters
X
bobP
uu
Camera Jacobian
p
Black: given variables Red: unknowns
X CK -R
u
v
w
where
2 2
geom
Ev
wx y
u
w
-KC
R
u
v
w
11 1 3 13
1 3 22 23
1 3 1 3
- -
- -
-
K X C 0 K X C
0 K X C K X CR
0 0 X C
u
v
w
=
Cp
q
3+4 parameters
R
q R q
u u
v v
w w
: Chain rule
Quaternion jacobian
v
Bob Alice
1X
bobP aliceP
u
pp
X
Black: given variables Red: unknowns
Camera & Point Jacobian ( ) ( )
p X
p ,X p ,XJ J J
p Xj i j i
ij ij ij
j i
f f
J1,bobpJ 2 70 1,bobXJ
1,aliceXJ1,alicepJ2 70
v
Bob Alice
1X
bobP aliceP
u
pp
X
Black: given variables Red: unknowns
Camera & Point Jacobian ( ) ( )
p X
p ,X p ,XJ J J
p Xj i j i
ij ij ij
j i
f f
J1,bobpJ 2 70 1,bobXJ
1,aliceXJ1,alicepJ2 70
2X
2,bobpJ 2 70 2,bobXJ
2,aliceXJ2,alicepJ2 70
2 30
2 30
2 30
2 30
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 3x4
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
J JTsize of : 24x24
x J J J b f(x)-1T T
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
J JTsize of : 24x24
x J J J b f(x)-1T T
Main computational bottle neck
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3