Page 1
CEE598 - Visual Sensing for
Civil Infrastructure Eng. & Mgmt.
Session 5 – Single View Metrology
Department of Civil and Environmental Engineering, University of Illinois at Urbana-Champaign
Mani Golparvar-Fard Department of Civil and Environmental Engineering
3129D, Newmark Civil Engineering Lab
e-mail: [email protected]
Page 2
Reminders
“App Turns iPhone into a Smarted Camera” • Technology Review:
http://www.technologyreview.com/computing/32235/?p1=MstRcnt&a=f
http://www.youtube.com/watch?v=b0zLgCF42Vk&feature=player_embedded
“Google’s Art Project” • http://www.googleartproject.com/
(Based on an image-based reconstruction algorithm):
• http://www.youtube.com/watch?v=RAvnJCBYHgE&feature=player_embedded
Camera Calibration
Wikipage
Assignment 1 will be online Next Tuesday
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
2
Page 3
Single View Metrology
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
3
Vermeer’s Music Lesson
Reconstructions by Criminisi et al.
Page 4
New Papers in this area
Yu Chen, Duncan Robertson and Roberto Cipolla.
A Practical System for Modelling Body Shapes
from Single View Measurements. Proceedings of the
British Machine Vision Conference, pages 82.1-82.11.
BMVA Press, September 2011.
http://dx.doi.org/10.5244/C.25.82
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
4
Page 5
Outline
Single View Metrology
• Review calibration
• Points and Lines
• Vanishing Points
• Measuring Height
5
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013 Some slides in this lecture are courtesy to Profs. D. Hoiem and S. Savarese
Reading: [HZ] Chapters 2,3,8
Page 6
Outline
Single View Metrology
• Review calibration
• Points and Lines
• Vanishing Points
• Measuring Height
6
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Page 7
jC
ii PMP
i
i
iv
up
In pixels
TRKM
100
v0
ucot
100
00
001
K o
o
sin
1
World ref. system
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
7
Page 8
jC
ii PMP
i
i
iv
up
Need at least 6 correspondences
11 unknown
TRKM
In pixels World ref. system
Calibration Problem
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
8
Page 9
Calibrating the Camera
Method 1: Use an object (calibration grid) with
known geometry
• Correspond image points to 3d points
• Get least squares solution (or non-linear solution)
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
w
wv
wu
9 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
9
Page 10
Estimating the Projection Matrix
Place a known object in the scene
• Identify correspondence between image and
scene
• Compute mapping from scene to image
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
10
11 34333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
v
u
TRKM
Page 11
Direct Linear Calibration
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
11
Page 12
Direct Linear Calibration
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
12
Page 13
Direct Linear Calibration
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
13
Page 14
Direct Linear Calibration
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
14
Can solve for 𝑚𝑖𝑗 by linear least squares
Page 15
Calibration with linear method
Advantages: easy to formulate and solve
Disadvantages • Doesn’t tell you camera parameters • Doesn’t model radial distortion • Can’t impose constraints, such as known focal
length • Doesn’t minimize right error function (see HZ p.
181)
Non-linear methods are preferred • Define error as difference between projected points
and measured points • Minimize error using Newton’s method or other
non-linear optimization 15
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
15
Page 16
Pinhole perspective projection
TRKM
C
Ow
-Internal parameters K are known
-R, T are known – but these can only relate C to the calibration rig
P p
Can I estimate P from the measurement p from a single image?
No - in general [P can be anywhere along the line defined by C and p]
Once the camera is calibrated...
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
16
Page 17
Pinhole perspective projection
C
Ow
P p
unknown known Known/
Partially known/
unknown
Recovering structure from a single view
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
17
Page 18
Outline
Single View Metrology
• Review calibration
• Points and Lines
• Vanishing Points
• Measuring Height
18
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Page 19
Camera Calibration
What if world coordinates are not known?
Can we use scene features(vanishing points)?
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
19
Page 20
Calibrating the Camera
Method 2: Use vanishing points
• Find vanishing points corresponding to orthogonal directions
Vanishing point
Vanishing line
Vanishing point
Vertical vanishing
point (at infinity)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
20
Page 21
Vanishing Points and Lines
Scene contains lines along directions that are
orthogonal
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
21
Page 22
Calibration by orthogonal vanishing points
Intrinsic camera matrix
• Use orthogonality as a constraint
• Model K with only f, u0, v0
What if you don’t have three finite vanishing points?
• Two finite VP: solve f, get valid u0, v0 closest to image center
• One finite VP: u0, v0 is at vanishing point; can’t solve for f
ii KRXp 0j
T
i XX
For vanishing points
22 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
22
Page 23
Calibration by vanishing points
Intrinsic camera matrix
Rotation matrix
• Set directions of vanishing points
e.g., X1 = [1, 0, 0]
• Each VP provides one column of R
• Special properties of R
inv(R)=RT
Each row and column of R has unit length
ii KRXp
23 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
23
Page 24
(0,0,0)
The projective plane Why do we need homogeneous coordinates?
• represent points at infinity, homographies, perspective projection, multi-view relationships
What is the geometric intuition? • a point in the image is a ray in projective space
(sx,sy,s)
• Each point (x,y) on the plane is represented by a ray (sx,sy,s)
– all points on the ray are equivalent: (x, y, 1) (sx, sy, s)
image plane
(x,y,1)
-y
x -z
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
24
Page 25
Projective lines
What does a line in the image correspond to in
projective space?
• A line is a plane of rays through origin
– all rays (x,y,z) satisfying: ax + by + cz = 0
z
y
x
cba0 :notationvectorin
• A line is also represented as a homogeneous 3-vector l
l p
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
25
Page 26
l l
Point and line duality
• A line l is a homogeneous 3-vector
• It is to every point (ray) p on the line: l p=0
p1 p2
What is the intersection of two lines l1 and l2 ?
• p is to l1 and l2 p = l1 l2
Points and lines are dual in projective space
• given any formula, can switch the meanings of points and lines
to get another formula
l1
l2
p
What is the line l spanned by rays p1 and p2 ?
• l is to p1 and p2 l = p1 p2
• l is the plane normal
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
26
Page 27
Ideal points and lines
Ideal point (“point at infinity”)
• p (x, y, 0) – parallel to image plane
• It has infinite image coordinates
(sx,sy,0) -y
x -z image plane
Ideal line
• l (a, b, 0) – parallel to image plane
(a,b,0)
-y
x
-z image plane
• Corresponds to a line in the image (finite coordinates)
– goes through image origin (principle point) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
27
Page 28
Homographies of points and lines
Computed by 3x3 matrix multiplication
• To transform a point: 𝒑′ = 𝑯𝒑
• To transform a line: 𝒍𝒑 = 0 𝑙′𝑝′ = 0
0 = 𝒍𝒑 = 𝒍𝑯−𝟏𝑯𝒑 = 𝒍𝑯−𝟏𝒑′ 𝒍′ = 𝒍𝑯−𝟏
lines are transformed by post multiplication of 𝑯−𝟏
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
28
Page 29
3D projective geometry
These concepts generalize naturally to 3D
Homogeneous coordinates
Projective 3D points have four coords:
𝑷 = (𝑋, 𝑌, 𝑍, 𝑊)
Duality
A plane 𝑵 is also represented by a 4-vector
Points and planes are dual in 3D: 𝑵 𝑷 = 0
Projective transformations
Represented by 4x4 matrices 𝑻:
𝑷′ = 𝑻𝑷, 𝑵′ = 𝑵 𝑇−1
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
29
Page 30
3D to 2D: “perspective” projection
Matrix Projection: ΠPp
1************
ZYX
w
wywx
What is not preserved under perspective
projection?
• Length, angles, parallelism
What IS preserved?
• Lines, incidences
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
30
Page 31
Vanishing points
Vanishing point
• projection of a point at infinity
image plane
camera center
ground plane
vanishing point
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
31
Page 32
Vanishing points (2D)
image plane
camera center
line on ground plane
vanishing point
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
32
Page 33
Vanishing points
Properties
• Any two parallel lines have the same vanishing point v
• The ray from C through v is parallel to the lines
• An image may have more than one vanishing point
in fact every pixel is a potential vanishing point
image plane
camera center
C
line on ground plane
vanishing point V
line on ground plane
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
33
Page 34
Vanishing lines
Multiple Vanishing Points • Any set of parallel lines on the plane define a vanishing point • The union of all of these vanishing points is the horizon line also called vanishing line
• Note that different planes define different vanishing lines
v1 v2
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
34
Page 35
Vanishing lines
Multiple Vanishing Points • Any set of parallel lines on the plane define a vanishing point • The union of all of these vanishing points is the horizon line also called vanishing line
• Note that different planes define different vanishing lines
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
35
Page 36
Computing vanishing points
V
DPP t 0
P0
D
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
36
Page 37
Computing vanishing points
Properties • 𝑷 is a point at infinity, 𝒗 is its projection
• They depend only on line direction
• Parallel lines 𝑷0 + 𝑡𝑫, 𝑷1 + 𝑡𝑫 intersect at 𝑷
V
DPP t 0
0/1
/
/
/
1
Z
Y
X
ZZ
YY
XX
ZZ
YY
XX
tD
D
D
t
t
DtP
DtP
DtP
tDP
tDP
tDP
PP
ΠPv
P0
D
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
37
Page 38
Computing vanishing lines
Properties
• 𝒍 is intersection of horizontal plane through 𝑪 with image
plane
• Compute 𝒍 from two sets of parallel lines on ground plane
• All points at same height as 𝑪 project to 𝒍 points higher than 𝐶 project above 𝑙
• Provides way of comparing height of objects in the scene
ground plane
l C
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
38
Page 39
Is the Parachute higher than the person
who is taking this picture?
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
39
Page 40
Fun with vanishing points
Perspective Cues?
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
40
Page 41
Perspective cues
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
41
Page 42
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
42
Perspective cues 42
Page 43
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
43
Perspective cues 43
Page 44
Ames Room
44
http://www.youtube.com/watch?v=hCV2Ba5wrcs&feature=related
http://www.youtube.com/watch?v=6aJlX0AEWys&feature=related
http://www.youtube.com/watch?v=6aJlX0AEWys&feature=related
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Page 45
Vanishing
Point
Comparing heights
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
45
Page 46
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
1
2
3
4
5 5.4
2.8
3.3
Camera height
Comparing heights
46
Page 47
q1
Computing vanishing points (from lines)
Intersect p1q1 with p2q2
v
p1
p2
q2
Least squares version
• Better to use more than two lines and compute the “closest” point of
intersection
• See notes by Bob Collins for one good way of doing this:
– http://www-2.cs.cmu.edu/~ph/869/www/notes/vanishing.txt
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
47
Page 48
C
Measuring height without a ruler
ground plane
Compute Z from image measurements
• Need more than vanishing points to do this
Z
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
48
Page 49
The cross ratio
A Projective Invariant
• Something that does not change under projective transformations
(including perspective projection)
P1
P2
P3 P4
1423
2413
PPPP
PPPP
The cross-ratio of 4 collinear points
Can permute the point ordering
• 4! = 24 different orders (but only 6 distinct values)
This is the fundamental invariant of projective geometry
1
i
i
i
iZ
Y
X
P
3421
2431
PPPP
PPPP
Proof available on Scholar CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
49
Page 50
vZ
r
t
b
tvbr
rvbt
Z
Z
image cross ratio
Measuring height
B (bottom of object)
T (top of object)
R (reference point)
ground plane
H C
TBR
RBT
scene cross ratio
1
Z
Y
X
P
1
y
x
pscene points represented as image points as
R
H
R
H
R
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
50
Page 51
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Measuring height
R H
vz
r
b
t
R
H
Z
Z
tvbr
rvbt
image cross ratio
H
b0
t0
v vx vy
vanishing line (horizon)
51
Page 52
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Measuring height vz
r
b
t0
vx vy
vanishing line (horizon)
v
t0
m0
What if the point on the ground plane b0 is not known?
• Here the guy is standing on the box, height of box is known
• Use one side of the box to help find b0 as shown above
b0
t1
b1
52
Page 53
Computing (X,Y,Z) coordinates
Okay, we know how to compute height (Z
coords)
• how can we compute X, Y?
• Exact same idea as before, but substitute X for
Z (e.g., need a reference object with known X
coordinates)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
53
Page 54
3D Modeling from a photograph
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
54
Page 55
Some Related Techniques
Image-Based Modeling and Photo Editing
• Mok et al., SIGGRAPH 2001
• http://graphics.csail.mit.edu/ibedit/
Single View Modeling of Free-Form Scenes
• Zhang et al., CVPR 2001
• http://grail.cs.washington.edu/projects/svm/
Tour Into The Picture
• Anjyo et al., SIGGRAPH 1997
• http://koigakubo.hitachi.co.jp/little/DL_TipE.html
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
55