Automatic Camera Calibration Lu Zhang Sep 22, 2005
Dec 24, 2015
Outline
Projective geometry and camera modals Principles of projective geometry Camera modals
Camera calibration methods Basic principles of self-calibration Stratified self-calibration
Projective Geometry Basic principles in projective spaces:
Points:A point of a dimensional projective space is represented by
an
n+1 vector of coordinates , are called
the homogeneous or projective coordinates of the points, x is called a
a coordinate vector. If for the two n+1 vectors represent the
same point.
np
121 ,...,, nxxxx ix
ii yx 11 ni
Projective Geometry The Projective Line The space is known as projective line. The standard projective basis of projective line is and . A point on the line
is , and are not both=0.
The point at infinity:
If let , when =0,
-> infinity, we call this point ‘point at infinity’
1p Te 0,11
Te 1,02 Te 1,13 2211 exexx 2x
2x21 eex 2
1
x
x
1x
Projective Geometry Projective space
Points:
Planes:
Lines:
A line is defined as the set of points that are linearly dependent on two points.
Plane at infinity: The points with =0 are said to be at infinity or ideal
points. The set of all ideal points may be written (X; Y; Z; 0). The set of all ideal points lies on a single plane, the plane at infinity.
Txxxxx 4321 ,,,
Tuuuuu 4321 ,,,
3P
4x
Camera models Camera models A point M on an object with coordinat
es (X,Y,Z) will be imaged at some point m=(x, y) in the image plane.
If consider the effect of focal length f, the relationship between image coordinate and 3-D space coordinate can be written as
here u=U/S v=V/S if S≠0, m=PMy
v
x
u
z
f
T
z
y
x
f
f
S
V
U
0100
000
000
Camera modals The general intrinsic matrix is
Intrinsic parameter indicates theproperty of camera itself. focal length pixel width pixel height, x coordinate at the optical
centre y coordinate at the optical
centre
0100
00
00
0
0
vfk
ufk
P v
u
vkuk
0u
0v
f
Camera models Camera Motion (Extrinsic parameters) If we go from the old coordinate system centered at C to
the new coordinate system centered at O by a rotation R followed by a translation T, in projective coordinates
The 4*4 matrix K is
newold KMM
103
T
tR
Camera models Intrinsic calibrationThe graph shows the transformation from retinal plane to itself.
The 3*3 matrix H is given by
Cause
We have
thus
oldnew Hmm
102
T
ts
MPm oldold
MHPm oldnew
KHPP oldnew
Self-calibration Self-calibration refers to the process of
calculating all the intrinsic parameters of the camera using only the information available in the images taken by that camera.
No calibration frame or known object is needed: the only requirement is that there is a static object in the scene, and the camera moves around taking images.
Self-calibration Why could we use self-calibratio
n? projective invariants
Epipolar geometry: The epipole is the point ofintersection of the linejoining the optical centerswith the image plane.
Thus the epipole is theimage, in one camera, of theoptical centre of the othercamera
Self-calibration a point x in one image gene
rates a line in the other on which its corresponding point x’ must lie.
With two views, the two camera coordinate systems are related by a rotation R and a translation T.
Self-calibration Therefore
Or if rewrite it as
Then we can define E, the essential matrix:
TRxx '
0)(' RxTx
0' Exx T
0
0
0
xy
xz
yz
tt
tt
tt
E
Self-calibration If we have two views of a point
M in three dimensional space, with M imaged at m in view 1 and m' in view 2
m=PM and m’=P’M If we set C(0,0,1), then we can
find p through PC=0,
where P=[P p] e’ is the projection of C on
view2, therefore
pPC 1
1''
1pPPe
Self-calibration
We also can get
Then can rewrite this equation as
mPPm 1''
1'
0''
'0'
''0
0'
PP
ee
ee
ee
F
Fmm
xy
xz
yz
T
Self-calibration A is the 3*3 left corner matrix of intrinsic matrix
P
We get the relationship between F and E
'
''
''
'
1
'
'
1 f
vv
uu
Av
u
and
f
vv
uu
Av
u
c
c
c
c
1' EAAF T
Self-calibration We can get three fundamenta
l matrices F By using the relationship bet
ween F,E and A, and the already known coordinate of epipolar points:
finally use a matrix equation
we can calculate the parameters in matrix A which is also the parameters in Matrix P, P is the intrinsic Matrix which include camera characteristics.
323123211312 ,,,,, eeeeee
Stratified self-calibration
What is Stratified self-calibration? Self-calibration is the process of determining internal cam
era parameters directly from multiple uncalibrated images.
First, Stratified self-calibration performs a projective reconstruction. Then, it obtains an affine reconstruction as the initial value. Finally, it applies metric reconstruction.
Stratification of geometry Projective Affine Metric
Stratified self-calibration Obtaining Projective camera matrix Determine the rectifying homography H is the projective camera matrix for each view i For the actual cameras, the internal parameter K is the same for each
view, but in general the calibration matrix will differ for each view. Therefore, the purpose for self-calibration is to find a rectifying homo
graphy H
Obtaining the metric reconstruction
iP
iP][ iiii tRKP
iK
][ iii tIKRHP
},{},{ 1j
ij
i XHHPXP
Stratified self-calibration Stratified self-calibration algorithm Step 1: affine calibration
a. formulate the modulus constraint for all pairs of views, need at least 3 views
b. (for n>3) solve the set of equations c. compute the affine projection matrices
Step 2: metric calibrationa. compute the position of plane at infinityb. find the intrinsic parameters Kc. compute the metric projection matrices
Pipeline Projective reconstruction
Relating images Initial reconstruction Adding views
Self-calibration Finding plane at infinity Compute K Metric reconstruction
Dense depth estimation Rectification Dense stereo matching
Modeling
Projective reconstruction Relating images
Detecting feature points Harris corner detector
Matching feature points RANSAC (RANdom Sampling Consensus) Determine Fundamental MatrixLeast square estimation
The whole procedure: Repeat
a. take minimal sample(8)b. compute Fc. estimate inliers
Until (inliers, trials)> 95% Refine F (using all inliers)
OKP
Projective reconstruction Results from projective reconstruction
Feature points detection (the original images are captured from a video sequence)
After applying Harris Corner Detector
Projective reconstruction Improvement by using RANSAC
Computation of Fundamental Matrix by applying least square estimation
0.2005- 0.0218- 0.0132
0.0218 0.0000 0.0000-
0.0121- 0.0000 0.0000-
F
Projective reconstruction Result for projective reconstruction (con’t)
Feature points detection
After applying Harris Corner Detector
Figure 9 Figure 10
Projective reconstruction Improvement by using RANSAC
Computation of Fundamental Matrix by applying least square estimation
0.1853 0.0043 0.0006
0.0044- 0.0000 0.0000-
0.0007- 0.0000 0.0000-
F
Self-calibration Finding initial Projective Matrix
Compute epipoles and epipolar lines Epipolar lines: & Epipoles: on all epipolar lines, thus x ,
Fxl '
]λe'|ve'F][[e'P' 0]|[IP T
'xFl T
0' Fxe T
0' Fe T 0Fe
Self-calibration Result from Initial Projective Matrix
Epipolar lines
Epipoles
0.0017
0.4982
0.8670
1e
0.0018
0.5067
0.8621
2e
Self-calibration Result from Initial Projective Matrix
Epipolar lines
Epipoles
0.0004-
0.1685-
0.9857
1e
0.0004-
0.1182-
0.9930
2e
Self-calibration Projective Matrix
0.0018 0.0250 0.0000- 0.0000-
0.5067 0.1729 0.0188 0.0114-
0.8621 0.1016- 0.0110- 0.0067
2P
0.0004- 0.0044- 0.0000 0.0000-
0.1182- 0.1840- 0.0042- 0.0006-
0.9930 0.0219- 0.0005- 0.0001-
2P
1.0000 0 0
0.1590 695.6221 0
0.5541 59.7937 677.2568
K
0.1853 0.0043 0.0006
0.0044- 0.0000 0.0000-
0.0007- 0.0000 0.0000-
F