3D Computer Vision and Video Computing 3D Vision Topic 2 of Part II Calibration CSc I6716 Fall 2010 Zhigang Zhu, City College of New York [email protected]
Feb 15, 2016
3D Computer Visionand Video Computing 3D Vision
Topic 2 of Part IICalibration
CSc I6716Fall 2010
Zhigang Zhu, City College of New York [email protected]
3D Computer Visionand Video Computing Lecture Outline
n Calibration: Find the intrinsic and extrinsic parametersl Problem and assumptionsl Direct parameter estimation approachl Projection matrix approach
n Direct Parameter Estimation Approachl Basic equations (from Lecture 5)l Homogeneous Systeml Estimating the Image center using vanishing pointsl SVD (Singular Value Decomposition) l Focal length, Aspect ratio, and extrinsic parametersl Discussion: Why not do all the parameters together?
n Projection Matrix Approach (…after-class reading)l Estimating the projection matrix Ml Computing the camera parameters from Ml Discussion
n Comparison and Summaryl Any difference?
3D Computer Visionand Video Computing Problem and Assumptions
n Given one or more images of a calibration pattern, n Estimate
l The intrinsic parametersl The extrinsic parameters, orl BOTH
n Issues: Accuracy of Calibrationl How to design and measure the calibration pattern
n Distribution of the control points to assure stability of solution – not coplanar
n Construction tolerance one or two order of magnitude smaller than the desired accuracy of calibration
n e.g. 0.01 mm tolerance versus 0.1mm desired accuracyl How to extract the image correspondences
n Corner detection?n Line fitting?
l Algorithms for camera calibration given both 3D-2D pairs
n Alternative approach: 3D from un-calibrated camera
3D Computer Visionand Video Computing Camera Model
n Coordinate Systemsl Frame coordinates (xim, yim) pixelsl Image coordinates (x,y) in mml Camera coordinates (X,Y,Z) l World coordinates (Xw,Yw,Zw)
n Camera Parametersl Intrinsic Parameters (of the camera and the frame grabber): link the
frame coordinates of an image point with its corresponding camera coordinates
l Extrinsic parameters: define the location and orientation of the camera coordinate system with respect to the world coordinate system
Zw
Xw
Yw
Y
X
Zx
yO
Pw
P
p
xim
yim
(xim,yim)
Pose / Camera
Object / World
Image frame
Frame Grabber
3D Computer Visionand Video ComputingLinear Version of Perspective Projection
n World to Cameral Camera: P = (X,Y,Z)T
l World: Pw = (Xw,Yw,Zw)T
l Transform: R, T n Camera to Image
l Camera: P = (X,Y,Z)T
l Image: p = (x,y)T
l Not linear equationsn Image to Frame
l Neglecting distortionl Frame (xim, yim)T
n World to Framel (Xw,Yw,Zw)T -> (xim, yim)T
l Effective focal lengthsn fx = f/sx, fy=f/sy
zT
yT
xT
zwww
ywww
xwww
T
TT
TZrYrXr
TZrYrXrTZrYrXr
w
w
w
w
PR
PRPR
TRPP
3
2
1
333231
232221
131211
yyim
xximsoyysoxx)()(
) ,(),(ZYf
ZXfyx
zwww
ywwwyyim
zwww
xwwwxxim
TZrYrXrTZrYrXr
foy
TZrYrXrTZrYrXrfox
333231
232221
333231
131211
3D Computer Visionand Video Computing Direct Parameter Method
n Extrinsic Parametersl R, 3x3 rotation matrix
n Three angles a,b,gl T, 3-D translation vector
n Intrinsic Parametersl fx, fy :effective focal length in pixel
n a = fx/fy = sy/sx, and fxl (ox, oy): known Image center -> (x,y) knownl k1, radial distortion coefficient: neglect it in the basic algorithm
n Same Denominator in the two Equationsl Known : (Xw,Yw,Zw) and its (x,y)l Unknown: rpq, Tx, Ty, fx, fy
zwww
ywwwyyim
zwww
xwwwxxim
TZrYrXrTZrYrXr
foyy
TZrYrXrTZrYrXr
foxx
333231
232221
333231
131211
'
'
)(')(' 131211232221 xwwwxywwwy TZrYrXrfyTZrYrXrfx
'/)('/)( 131211232221 xTZrYrXrfyTZrYrXrf xwwwxywwwy
3D Computer Visionand Video Computing Linear Equations
n Linear Equation of 8 unknowns v = (v1,…,v8)
l Aspect ratio: a = fx/fyl Point pairs , {(Xi, Yi,, Zi) <-> (xi, yi) } drop the ‘ and subscript “w”
)(')(' 131211232221 xwwwywww TZrYrXryTZrYrXrx a
0)()()()( 131211232221 xiiiiiiiyiiiiiii TyrZyrYyrXyTxrZxrYxrXx aaaa
087654321 vyvZyvYyvXyvxvZxvYxvXx iiiiiiiiiiiiii
),,,,,,,(),,,,,,,(
131211232221
87654321
xy TrrrTrrrvvvvvvvv
aaaa
3D Computer Visionand Video Computing Homogeneous System
n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) }, i=1,2,…Nl 8 unknowns v = (v1,…,v8)T, 7 independent parameters
087654321 vyvZyvYyvXyvxvZxvYxvXx iiiiiiiiiiiiii
0Av
NNNNNNNNNNNNNN yZyYyXyxZxYxXx
yZyYyXyxZxYxXxyZyYyXyxZxYxXx
22222222222222
11111111111111
A
n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7l Can be determined from the SVD of A
3D Computer Visionand Video Computing
n Homework #3 online, due October 25 (Monday) before class
3D Computer Visionand Video Computing Homogeneous System
n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) }, i=1,2,…Nl 8 unknowns v = (v1,…,v8)T, 7 independent parameters
087654321 vyvZyvYyvXyvxvZxvYxvXx iiiiiiiiiiiiii
0Av
NNNNNNNNNNNNNN yZyYyXyxZxYxXx
yZyYyXyxZxYxXxyZyYyXyxZxYxXx
22222222222222
11111111111111
A
n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7l Can be determined from the SVD of A
3D Computer Visionand Video Computing SVD: definition
n Singular Value Decomposition:l Any mxn matrix can be written as the product of three
matricesTUDVA
n Singular values si are fully determined by An D is diagonal: dij =0 if ij; dii = si (i=1,2,…,n)
n s1 s2 … sN 0
n Both U and V are not uniquen Columns of each are mutual orthogonal vectors
nnnn
n
n
n
mmmm
m
m
mnmm
n
n
vvv
vvvvvv
uuu
uuuuuu
aaa
aaaaaa
21
22212
121112
1
21
22221
11211
21
22221
11211
000
000
s
ss
V1U1
Appendix A.6
3D Computer Visionand Video Computing SVD: properties
n 1. Singularity and Condition Numberl nxn A is nonsingular IFF all singular values are nonzerol Condition number : degree of singularity of A
n A is ill-conditioned if 1/C is comparable to the arithmetic precision of your machine; almost singular
n 2. Rank of a square matrix Al Rank (A) = number of nonzero singular values
n 3. Inverse of a square Matrixl If A is nonsingularl In general, the pseudo-inverse of A
n 4. Eigenvalues and Eigenvectors (questions)l Eigenvalues of both ATA and AAT are si
2 (si > 0)l The columns of U are the eigenvectors of AAT (mxm)l The columns of V are the eigenvectors of ATA (nxn)
TUDVA
nC ss /1
TUVDA 11 TUVDA 1
0
iiT
iuuAA 2s
iiT
ivAvA 2s
3D Computer Visionand Video Computing SVD: Application 1
n Least Squarel Solve a system of m equations for n unknowns x(m >= n)l A is a mxn matrix of the coefficients l b (0) is the m-D vector of the data l Solution:
l How to solve: compute the pseudo-inverse of ATA by SVDn (ATA)+ is more likely to coincide with (ATA)-1 given m > nn Always a good idea to look at the condition number of ATA
bAx
bAAxA TT bAAAx TT )(
nxn matrix Pseudo-inverse
3D Computer Visionand Video Computing SVD: Application 2
n Homogeneous Systeml m equations for n unknowns x(m >= n-1)l Rank (A) = n-1 (by looking at the SVD of A)l A non-trivial solution (up to a arbitrary scale) by SVD:l Simply proportional to the eigenvector corresponding to the
only zero eigenvalue of ATA (nxn matrix)n Note:
l All the other eigenvalues are positive because Rank (A)=n-1
l For a proof, see Textbook p. 324-325l In practice, the eigenvector (i.e. vn) corresponding to
the minimum eigenvalue of ATA, i.e. sn2
0Ax
iiT
ivAvA 2s
3D Computer Visionand Video Computing SVD: Application 3
n Problem Statementsl Numerical estimate of a matrix A whose entries are not
independentl Errors introduced by noise alter the estimate to Â
n Enforcing Constraints by SVDl Take orthogonal matrix A as an examplel Find the closest matrix to Â, which satisfies the constraints
exactlyn SVD of  n Observation: D = I (all the singular values are 1) if A is
orthogonaln Solution: changing the singular values to those expected
TUDVA ˆ
TUIVA
3D Computer Visionand Video Computing Homogeneous System
n Homogeneous System of N Linear Equations l Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) },
i=1,2,…Nl 8 unknowns v = (v1,…,v8)T, 7 independent parameters
n The system has a nontrivial solution (up to a scale) l IF N >= 7 and N points are not coplanar => Rank (A) = 7l Can be determined from the SVD of Al Rows of VT: eigenvectors {ei} of ATAl Solution: the 8th row e8 corresponding to the only zero
singular value l8=0
n Practical Considerationl The errors in localizing image and world points may make
the rank of A to be maximum (8)l In this case select the eigenvector corresponding to the
smallest eigenvalue.
0Av
TUDVA
8ev c
3D Computer Visionand Video ComputingScale Factor and Aspect Ratio
n Equations for scale factor g and aspect ratio a
n Knowledge: R is an orthogonal matrix
n Second row (i=j=2):
n First row (i=j=1)
),,,,,,,( 131211232221 xy TrrrTrrr aaaagv
T
T
T
ijrrrrrrrrr
r
3
2
1
333231
232221
131211
33RRR
R
ji if 0ji if 1
jTi RR
1222232221
rrr 23
22
21|| vvv g
1222131211
rrr 27
26
25|| vvv ga
a
||g
v1 v2 v3 v4 v5 v6 v7 v8
3D Computer Visionand Video Computing Rotation R and Translation T
n Equations for first 2 rows of R and T given a and |g|
n First 2 rows of R and T can be found up to a common sign s (+ or -)
n The third row of the rotation matrix by vector product
n Remaining Questions :l How to find the sign s?l Is R orthogonal?l How to find Tz and fx, fy?
),,,,,,,( || 131211232221 xy TrrrTrrrs aaaagv
yxTT sTsTss ,,, 21 RR
TTTTT ss 21213 RRRRR
T
T
T
ijrrrrrrrrr
r
3
2
1
333231
232221
131211
33RRR
R
3D Computer Visionand Video Computing Find the sign s
n Facts:l fx > 0l Zc >0l x knownl Xw,Yw,Zw known
n SolutionÞ Check the sign of Xc
Þ Should be opposite
to x zwww
ywwwyy
zwww
xwwwxx
TZrYrXrTZrYrXr
fZcYcfy
TZrYrXrTZrYrXrf
ZcXcfx
333231
232221
333231
131211
Zw
Xw
Yw
Y
X
Zx
yO
PwP
p
xim
yim
(xim,yim)
3D Computer Visionand Video Computing Rotation R : Orthogonality
n Question: l First 2 rows of R are calculated
without using the mutual orthogonal constraint
n Solution: l Use SVD of estimate R
TTTTT ss 21213 RRRRR
T
T
T
ijrrrrrrrrr
r
3
2
1
333231
232221
131211
33RRR
R
?ˆˆ IRR T
TUDVR ˆ TUIVR
Replace the diagonal matrix D with the 3x3 identity matrix
3D Computer Visionand Video Computing Find Tz, Fx and Fy
n Solutionl Solve the system of N linear
equations with two unknownn Tx, fx
l Least Square method
l SVD method to find inverse
zwww
xwwwx TZrYrXr
TZrYrXrfx
333231
131211
bA
x
zfT
)()( 333231131211 wwwxxwwwz ZrYrXrxfTZrYrXrxT
ai2ai1 bi
bAAA TT
x
z
f
T 1)(ˆ
ˆ
3D Computer Visionand Video ComputingDirect parameter Calibration Summary
n Algorithm (p130-131)1. Measure N 3D coordinates (Xi, Yi,Zi)2. Locate their corresponding image
points (xi,yi) - Edge, Corner, Hough3. Build matrix A of a homogeneous
system Av = 0 4. Compute SVD of A , solution v5. Determine aspect ratio a and scale |g|6. Recover the first two rows of R and the
first two components of T up to a sign7. Determine sign s of g by checking the
projection equation8. Compute the 3rd row of R by vector
product, and enforce orthogonality constraint by SVD
9. Solve Tz and fx using Least Square and SVD, then fy = fx / a
Yw
Xw
Zw
3D Computer Visionand Video Computing
n Homework #3 online, due October 25 before class
3D Computer Visionand Video Computing Discussions
n Questions
l Can we select an arbitrary image center for solving other parameters?
l How to find the image center (ox,oy)?
l How about to include the radial distortion?
l Why not solve all the parameters once ?
n How many unknown with ox, oy? --- 20 ??? – projection matrix method
zwww
ywwwyyim
zwww
xwwwxxim
TZrYrXrTZrYrXr
foyy
TZrYrXrTZrYrXrfoxx
333231
232221
333231
131211
3D Computer Visionand Video Computing Estimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in
a point on the image - the vanishing point, which is the common intersection of all the image lines
3D Computer Visionand Video Computing Estimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in
a point on the image - the vanishing point, which is the common intersection of all the image lines
VP1
3D Computer Visionand Video Computing Estimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in a point
on the image - the vanishing point, which is the common intersection of all the image lines
n Important property:l Vector OV (from the center of projection to the vanishing point)
is parallel to the parallel lines
O
VP1
Y
X
Z
3D Computer Visionand Video Computing Estimating the Image Center
n Vanishing points:l Due to perspective, all parallel lines in 3D space appear to meet in
a point on the image - the vanishing point, which is the common intersection of all the image lines
VP1
VP2
3D Computer Visionand Video Computing Estimating the Image Center
n Orthocenter Theorem:l Input: three mutually
orthogonal sets of parallel lines in an image
l T: a triangle on the image plane defined by the three vanishing points
l Image center = orthocenter of triangle T
l Orthocenter of a triangle is the common intersection of the three altitudes VP1
VP2
VP3
3D Computer Visionand Video Computing Estimating the Image Center
n Orthocenter Theorem:l Input: three mutually
orthogonal sets of parallel lines in an image
l T: a triangle on the image plane defined by the three vanishing points
l Image center = orthocenter of triangle T
l Orthocenter of a triangle is the common intersection of the three altitudes VP1
VP2
VP3
3D Computer Visionand Video Computing Estimating the Image Center
n Orthocenter Theorem:l Input: three mutually
orthogonal sets of parallel lines in an image
l T: a triangle on the image plane defined by the three vanishing points
l Image center = orthocenter of triangle T
l Orthocenter of a triangle is the common intersection of the three altitudes
n Orthocenter Theorem:l WHY?
VP1
VP2
VP3
h3
h1
h1
(ox,oy)
3D Computer Visionand Video Computing Estimating the Image Center
n Assumptions:l Known aspect ratiol Without lens distortions
n Questions:l Can we solve both
aspect ratio and the image center?
l How about with lens distortions?
VP1
VP2
VP3
h3
h1
h1
(ox,oy) ?
3D Computer Visionand Video ComputingDirect parameter Calibration Summary
n Algorithm (p130-131)0. Estimate image center (and aspect ratio)1. Measure N 3D coordinates (Xi, Yi,Zi)2. Locate their corresponding image (xi,yi) -
Edge, Corner, Hough3. Build matrix A of a homogeneous system
Av = 0 4. Compute SVD of A , solution v5. Determine aspect ratio a and scale |g|6. Recover the first two rows of R and the first
two components of T up to a sign7. Determine sign s of g by checking the
projection equation8. Compute the 3rd row of R by vector product,
and enforce orthogonality constraint by SVD
9. Solve Tz and fx using Least Square and SVD , then fy = fx / a
Yw
Xw
Zw
3D Computer Visionand Video ComputingRemaining Issues and Possible Solution
n Original assumptions:l Without lens distortionsl Known aspect ratio when estimating image centerl Known image center when estimating others including aspect ratio
n New Assumptionsl Without lens distortion l Aspect ratio is approximately 1, or a = fx/fy = 4:3 ; image center about
(M/2, N/2) given a MxN image
n Solution (?)1. Using a = 1 to find image center (ox, oy)2. Using the estimated center to find others including a3. Refine image center using new a ; if change still significant, go to step
2; otherwise stop
l Projection Matrix Approach
3D Computer Visionand Video Computing
n Homework #3 online, due October 25 before class
3D Computer Visionand Video Computing Linear Matrix Equation of
perspective projectionn Projective Space
l Add fourth coordinate n Pw = (Xw,Yw,Zw, 1)T
l Define (u,v,w)T such thatn u/w =xim, v/w =yim
n 3x4 Matrix Eextl Only extrinsic parametersl World to camera
n 3x3 Matrix Eintl Only intrinsic parametersl Camera to frame
n Simple Matrix Product! Projective Matrix M= MintMext
l (Xw,Yw,Zw)T -> (xim, yim)T
l Linear Transform from projective space to projective planel M defined up to a scale factor – 11 independent entries
1ww
w
ZYX
wvu
extintMM
zT
yT
xT
z
y
x
ext
T
TT
TTT
rrrrrrrrr
3
2
1
333231
232221
131211
R
RR
M
1000
0
int yy
xxofof
M
wvwu
yx
im
im//
3D Computer Visionand Video Computing Projection Matrix M
n World – Frame Transforml Drop “im” and “w” l N pairs (xi,yi) <-> (Xi,Yi,Zi)l Linear equations of m
n 3x4 Projection Matrix Ml Both intrinsic (4) and extrinsic (6) – 10 parameters
1ii
i
i
i
i
ZYX
wvu
M
34333231
24232221
34333231
14131211
mZmYmXmmZmYmXm
wuy
mZmYmXmmZmYmXm
wux
iii
iii
i
ii
iii
iii
i
ii
z
zyyy
zxxx
yyyyyy
xxxxxx
TToTfToTf
rrrrorfrorfrorfrorfrorfrorf
333231
332332223121
331332123111M
0Am
3D Computer Visionand Video ComputingStep 1: Estimation of projection matrix
n World – Frame Transforml Drop “im” and “w” l N pairs (xi,yi) <-> (Xi,Yi,Zi)
n Linear equations of m l 2N equations, 11 independent variablesl N >=6 , SVD => m up to a unknown scale
34333231
24232221
34333231
14131211
mZmYmXmmZmYmXm
wuy
mZmYmXmmZmYmXm
wux
iii
iii
i
ii
iii
iii
i
ii
0Am
1111111111
11111111111000000001
yYyYyXyZYXxZxYxXxZYX
A
Tmmmmmmmmmmmm 343332312423222114131211m
3D Computer Visionand Video ComputingStep 2: Computing camera parameters
n 3x4 Projection Matrix Ml Both intrinsic and extrinsic
n From M^ to parameters (p134-135)l Find scale |g| by using unit vector R3
T
l Determine Tz and sign of g from m34 (i.e. q43)l Obtain R3
T
l Find (Ox, Oy) by dot products of Rows q1. q3, q2.q3, using the orthogonal constraints of R
l Determine fx and fy from q1 and q2 (Eq. 6.19) Wrong???)l All the rests: R1
T, R2T, Tx, Ty
l Enforce orthognoality on R?
z
zyyy
zxxx
yyyyyy
xxxxxx
TToTfToTf
rrrrorfrorfrorfrorfrorfrorf
333231
332332223121
331332123111M
MM gˆ
43
42
41
3
2
1ˆ
qqq
qqq
M
3D Computer Visionand Video Computing Comparisons
n Direct parameter method and Projection Matrix method
n Properties in Common:l Linear system first, Parameter decomposition secondl Results should be exactly the same
n Differencesl Number of variables in homogeneous systems
n Matrix method: All parameters at once, 2N Equations of 12 variables
n Direct method in three steps: N Equations of 8 variables, N equations of 2 Variables, Image Center – maybe more stable
l Assumptionsn Matrix method: simpler, and more general; sometime projection
matrix is sufficient so no need for parameter decompostionn Direct method: Assume known image center in the first two steps,
and known aspect ratio in estimating image center
3D Computer Visionand Video Computing Guidelines for Calibration
n Pick up a well-known technique or a fewn Design and construct calibration patterns (with known 3D)n Make sure what parameters you want to find for your cameran Run algorithms on ideal simulated data
l You can either use the data of the real calibration pattern or using computer generated data
l Define a virtual camera with known intrinsic and extrinsic parametersl Generate 2D points from the 3D data using the virtual cameral Run algorithms on the 2D-3D data set
n Add noises in the simulated data to test the robustness n Run algorithms on the real data (images of calibration target)n If successful, you are all set n Otherwise:
l Check how you select the distribution of control pointsl Check the accuracy in 3D and 2D localizationl Check the robustness of your algorithms againl Develop your own algorithms NEW METHODS?
3D Computer Visionand Video Computing Next
n 3D reconstruction using two cameras
Stereo Vision
& project discussions
n Homework #3 online, due October 25 before class