3D Urban Modeling Marc Pollefeys Jan-Michael Frahm, Philippos Mordohai Fall 2006 / Comp 790-089 Tue & Thu 14:00-15:15
Dec 21, 2015
3D Urban Modeling
Marc PollefeysJan-Michael Frahm, Philippos Mordohai
Fall 2006 / Comp 790-089
Tue & Thu 14:00-15:15
3D Urban Modeling
• Basics: sensor, techniques, …• Video, LIDAR, GPS, IMU, …• SfM, Stereo, …• Robot Mapping, GIS, …
• Review state-of-the-art• Video and LIDAR-based systems
• Projects to experiment• Virtual UNC-Chapel Hill• DARPA Urban Challenge• …
3D Urban Modeling course schedule(tentative)
Sep. 5, 7 Introduction Video-based Urban 3D Capture
Sep. 12, 14 Cameras
Sep. 19, 21
Sep. 26, 28 Project proposals
Oct. 3, 5
Oct. 10, 12
Oct. 17, 19 (fall break)
Oct. 24, 26 Project Status Update
Oct. 31, Nov. 2
Nov. 7, 9
Nov. 14, 16
Nov. 21, 23 (Thanksgiving)
Nov. 28, 30
Dec. 5 Project demonstrations (classes ended)Note: Dec. 3 is CVPR deadline
Course material
Slides, notes, papers and references
see course webpage/wiki (later)
On-line “shape-from-video” tutorial:http://www.cs.unc.edu/~marc/tutorial.pdfhttp://www.cs.unc.edu/~marc/tutorial/
Geometry primer
l'lx
Incidence of point and line
x'xl
0xl T 0Xπ T
0ll * CT
1* CC
0QXX T 0πQπ * T
-1* QQ
Joins
Conics and dual conics
0xx CT
Incidence of point and plane
Joins
Conics and dual conics
0Xπππ
T
3
T
2
T
1
0πXXX
T
3
T
2
T
1
Geometry primer
Transformation for lines
ll' -TH
Transformation for conics-1-TCHHC '
Transformation for dual conics
THHCC **'
xx' HTransformation for points
XX' H
ππ' -TH
-1-TQHHQ'
THHQQ' **
Transformation for planes
Transformation for quadrics
Transformation for dual quadrics
Transformation for points
Geometry primer Hierarchy of 2D transformations
1002221
1211
y
x
taa
taa
1002221
1211
y
x
tsrsr
tsrsr
333231
232221
131211
hhh
hhh
hhh
1002221
1211
y
x
trr
trr
Projective8dof
Affine6dof
Similarity4dof
Euclidean3dof
Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio
Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l∞
Ratios of lengths, angles.The circular points I,J
lengths, areas.
invariantstransformed squares
Geometry primer Hierarchy of 3D transformations
vTv
tAProjective15dof
Affine12dof
Similarity7dof
Euclidean6dof
Intersection and tangency
Parallellism of planes,Volume ratios, centroids,The plane at infinity π∞
Angles, ratios of length
The absolute conic Ω∞
Volume
10
tAT
10
tRT
s
10
tRT
Projector model
Relation between pixels and rays in space(dual of camera)
(main geometric difference is vertical principal point offset to reduce keystone effect)
?
Pinhole camera model
TT ZfYZfXZYX )/,/(),,(
101
0
0
1
Z
Y
X
f
f
Z
fY
fX
Z
Y
X
linear projection in homogeneous coordinates!
homogeneous coordinates
non-homogeneous coordinates
Principal point offset
Tyx
T pZfYpZfXZYX )/,/(),,(
principal pointT
yx pp ),(
101
0
0
1
Z
Y
X
pf
pf
Z
ZpfY
ZpfX
Z
Y
X
y
x
x
x
Principal point offset
101
0
0
Z
Y
X
pf
pf
Z
ZpfY
ZpfX
y
x
x
x
camX0|IKx
1y
x
pf
pf
K calibration matrix
General projective camera
1yx
xx
p
ps
K
1yx
xx
p
p
K
t|IKRP
non-singular
11 dof (5+3+3)
t|RKP
intrinsic camera parametersextrinsic camera parameters
Camera matrix decomposition
Finding the camera center
0PC (use SVD to find null-space)
Finding the camera orientation and internal parameters
KR (use RQ decomposition ~QR)
Q R=( )-1= -1 -1QR
(if only QR, invert)
PXλC)P(Xx (for all X and λ C must be camera center)
t|RKP
Radial distortion
• Due to spherical lenses (cheap)• Model:
R
yxyxKyxKyx ...))()(1(),( 222
2
22
1R
http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpgstraight lines are not straight anymore
pincushion dist.
barrel dist.
Action of projective camera on points and lines
forward projection of line
μbaμPBPAμB)P(AμX
back-projection of line
lPT
PXlX TT PX x0;xlT
PXx projection of point
Action of projective camera on conics and quadricsback-projection to cone
CPPQ Tco 0CPXPXCxx TTT
PXx
projection of quadric
TPPQC ** 0lPPQlQ T*T*T
lPT
Direct Linear Transform (DLT)
Minimal solution
Over-determined solution
5½ correspondences needed (say 6)
P has 11 dof, 2 independent eq./points
n 6 points
Apminimize subject to constraint
1p
use SVD
Singular Value Decomposition
XXVT XVΣ T XVUΣ T
Homogeneous least-squares
TVUΣA
1X AXmin subject to nVX solution
TVΣU
X
Degenerate configurations
(i) Points lie on plane or single line passing through projection center
(ii) Camera and points on a twisted cubic
Scale data to values of order 1
1. move center of mass to origin2. scale to yield order 1 values
Data normalization
D3
D2
Line correspondences
Extend DLT to lines
ilPT
ii 1TPXl
(back-project line)
ii 2TPXl (2 independent eq.)
Gold Standard algorithmObjective
Given n≥6 2D to 2D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P
Algorithm
(i) Linear solution:
(a) Normalization:
(b) DLT
(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:
(iii) Denormalization:
ii UXX~ ii Txx~
UP~
TP -1
~ ~~
Calibration example
(i) Canny edge detection(ii) Straight line fitting to the detected edges(iii) Intersecting the lines to obtain the images corners
typically precision <1/10
(H&Z rule of thumb: 5n constraints for n unknowns)
Errors in the world
Errors in the image and in the world
ii XPx
iX
Errors in the image
iPXx̂
i
(standard case)
Restricted camera estimation
Minimize geometric error impose constraint through parametrization
Find best fit that satisfies• skew s is zero• pixels are square • principal point is known• complete camera matrix K is known
Minimize algebraic error assume map from param q P=K[R|-RC], i.e. p=g(q)minimize ||Ag(q)||
Restricted camera estimation
Initialization • Use general DLT• Clamp values to desired values, e.g. s=0, x= y
Note: can sometimes cause big jump in error
Alternative initialization• Use general DLT• Impose soft constraints
• gradually increase weights
A simple calibration device
(i) compute H for each square (corners (0,0),(1,0),(0,1),(1,1))
(ii) compute the imaged circular points H(1,±i,0)T
(iii) fit a conic to 6 circular points(iv) compute K from through cholesky factorization
(≈ Zhang’s calibration method)
Some typical calibration algorithmsTsai calibration
Zhangs calibration
http://research.microsoft.com/~zhang/calib/
Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.
Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.
http://www.vision.caltech.edu/bouguetj/calib_doc/
*
*
projection
constraints
Tiii
Tii ωΩ KKPP
Self-calibration using absolute conic
Absolute conic projection:
Translate constraints on K through projection equation
to constraints on *
Euclidean projection matrix:
Upgrade from projective to metric
Transform structure and motion so that * diag(1,1,1,0)
some constraints, e.g. constant, no skew,...
(Faugeras ECCV’92; Triggs CVPR’97; Pollefeys et al. ICCV’98; etc.)
Practical linear self-calibration
0Ω
0Ω
0Ω
0ΩΩ
23T
13T
12T
22T
11T
PP
PP
PP
PPPP
(Pollefeys et al., ECCV‘02)
100
0ˆ0
00ˆ2
2
* f
fTPPKK
0ΩΩ
0ΩΩ
33T
22T
33T
11T
PPPP
PPPP
(only rough aproximation, but still usefull to avoid degenerate configurations)
(relatively accurate for most cameras)
9
1
9
1
1.0
11.0
101.0
12.0
1
1yy
xx
cf
csf
K
Don’t treat all constraints equal
after normalization!
when fixating point at image-center not only absolute quadric diag(1,1,1,0) satisfies ICCV98 eqs., but also diag(1,1,1,a), i.e. real or imaginary spheres!