Fundamentals of Photogrammetry
Fundamentals of Photogrammetry
Niclas Borlin, [email protected]
Department of Computing ScienceUmea University
Sweden
April 1, 2014
Fundamentals of Photogrammetry
Presentation
Ph.D. in Computing Science (2000).
Numerical Linear Algebra.
Non-linear least squares with non-linear equality constraint.
X-ray photogrammetry — Radiostereometry (RSA).
Post doc at Harvard Medical School, Boston, MA.
Fundamentals of Photogrammetry
Radiostereometric analysis (RSA)
Developed by Hallert (1960), Selvik (1974), Karrholm (1989),Borlin (2002, 2006), Valstar (2005).
Procedure
Dual X-ray setupCalibration cageMarkermeasurementsReconstruction ofprojection geometryMotion analysis
Software UmRSA Digital Measure running in Europe, NorthAmerica, Australia, Asia. Used to produce 150+ scientificpapers.
Fundamentals of Photogrammetry
Introduction
Definition
Photogrammetry — measuring from photographs
photos — “light”gramma — “that which is drawn or written”metron — “to measure”
Definition in Manual of Photogrammetry, 1st ed., 1944,American Society for Photogrammetry:
Photogrammetry is the science or art of obtainingreliable measurement by means of photographs
Fundamentals of Photogrammetry
Introduction
Overview
Principles
History
Mathematical models
Processing
Applications
Fundamentals of Photogrammetry
Principles
Principles
Non-contact measurements.
(Passive sensor.)
Collinearity.
Triangulation.
Fundamentals of Photogrammetry
Principles
Collinearity
Collinearity
The collinearity principle is the assumption that
the object points Q,the projection center C , andthe projected points q
are collinear, i.e. lie on a straight line.Q
C
q
Fundamentals of Photogrammetry
Principles
Triangulation
Triangulation
One image coordinate measurement (x , y) is too little todetermine the object point coordinates (X ,Y ,Z ).
We need at least two measurements of the same point.
C1
q1?
C1
q1
C2
q2
Q
Fundamentals of Photogrammetry
Principles
Triangulation
Triangulation (2)
The position of object points are calculated by triangulation,i.e. by angles, but without any range values.
B
α β
Q
C1 C2
Fundamentals of Photogrammetry
Principles
Triangulation
Other techniques
Trilateration, ranges but noangles (GPS).
C1r1
C2
r2
C3
r3
Q
Tachyometry, angles andranges (surveying, laserscanning)
C
Q
rα
Fundamentals of Photogrammetry
History
Overview
Principles
History
Mathematical models
Processing
Applications
Fundamentals of Photogrammetry
History
Pre-history
Pre-history
Geometry, perspective, pinhole camera model — Euclid (300BC).
Leonardo da Vinci (1480)
Perspective is nothing else than the seeing of anobject behind a sheet of glass, smooth and quitetransparent, on the surface of which all the thingsmay be marked that are behind this glass. All thingstransmit their images to the eye by pyramidal lines,and these pyramids are cut by the said glass. Thenearer to the eye these are intersected, the smallerthe image of their cause will appear.
Fundamentals of Photogrammetry
History
Plane table photogrammetry
First generation — Plane table photogrammetry
First photograph — Niepce,1825. Required 8 hour expo-sure.
Glass negative — Hershel,1839.
First use of terrestrial photographs for topographic maps —Laussedat, 1849 “Father of photogrammetry”. City map ofParis (1851).Film — Eastman, 1884.Architectural photogrammetry — Meydenbauer, 1893, coinedthe word “photogrammetry”.Measurements made on a map on a table. Photographs usedto extract angles.
Fundamentals of Photogrammetry
History
Analog photogrammetry
Second generation — Analog photogrammetry
Stereocomparator (Pulfrich, Fourcade, 1901). Requiredcoplanar photographs. Measurements made by floating mark.
Aeroplane (Wright 1903). First aerial imagery from aeroplanein 1909.Aerial survey camera for overlapping vertical photos (Messter1915).
Fundamentals of Photogrammetry
History
Analog photogrammetry
Second generation — Analog photogrammetry (2)
Opto-mechanical stere-oplotters (von Orel,Thompson, 1908, Zeiss1921, Wild 1926). Allowednon-coplanar photographs.
Wild A8 Autograph (1950)
Relative orientation determined by 6 points in overlappingimages — von Gruber points (1924)
Photogrammetry — the art of avoidingcomputations
Fundamentals of Photogrammetry
History
Analytical photogrammetry
Third generation — analytical photogrammetry
Finsterwalder (1899) — equations for analyticalphotogrammetry, intersection of rays, relative and absoluteorientation, least squares theory.von Gruber (1924) — projective equations and theirdifferentials,Computer (Zuse 1941, Turing, Flowers, 1943, Aiken 1944).Schmid, Brown multi-station analytical photogrammetry,bundle block adjustment (1953), adjustment theory.
The [Ballistic Research] laboratory had a virtualglobal monopoly on electronic computing power.This unique circumstance combined with Schmid setthe stage for the rapid transistion from classicalphotogrammetry to the analytic approach (Brown).
Ackermann independent models (1966).
Fundamentals of Photogrammetry
History
Analytical photogrammetry
Third generation — analytical photogrammetry (2)
Analytical plotter (Helava 1957) - image-map coordinatetransformation by electronic computation, servocontrol.
Zeiss Planicomp P3
Camera calibration (Brown 1966, 1971).
Direct Linear Transform (DLT) (Abdel-Azis, Karara, 1971).
Fundamentals of Photogrammetry
History
Digital photogrammetry
Digital photogrammetry
Charge-Coupled Device (CCD) (Boyle, Smith 1969).
Landsat (1972)
Digital camera (Sesson (Eastman Kodak) 1975 — 0.01Mpixels).
Flash memory (Masuoka (Toshiba) 1980).
Matching (Forstner 1986, Gruen 1985, Lowe 1999).
Projective Geometry (Klein 1939)
5-point relative orientation (Nister 2004)
Fundamentals of Photogrammetry
Mathematical models
Overview
Principles
History
Mathematical models
Processing
Applications
Fundamentals of Photogrammetry
Mathematical models
Preliminaries
Matrix multiplication
C = AB
a11 a12 a13
a21 a22 a23
a31 a32 a33
b11 b12
b21 b22
b31 b32
c11 c12
c21 c22
c31 c32
a 21× b 12
a 22× b 22
a23× b32
+
+
Fundamentals of Photogrammetry
Mathematical models
Preliminaries
Image plane placement
The projected coordinates q will be identicalif a (negative) sensor is placed behind the camera center orif a (positive) sensor is mirrored and placed in front of thecamera center. Q
C
q
q
Fundamentals of Photogrammetry
Mathematical models
The collinearity equations
The collinearity equations
Z
X
Y
C
Q
c
ZX
Y
Z0
X0
Y0
x
y qqp
The collinearity equationsx − xpy − yp−c
= kR
X − X0
Y − Y0
Z − Z0
describe the relationshipbetween the object point(X ,Y ,Z )T , the positionC = (X0,Y0,Z0)T of thecamera center and theorientation R of thecamera.
Fundamentals of Photogrammetry
Mathematical models
The collinearity equations
The collinearity equations (2)
Z
X
Y
C
Q
c
ZX
Y
Z0
X0
Y0
x
y qqp
The distance c is known asthe principal distance orcamera constant.
The point qp = (xp, yp)T iscalled the principal point.
The ray passing throughthe camera center C andthe principal point qp iscalled the principal ray.
Fundamentals of Photogrammetry
Mathematical models
The collinearity equations
The collinearity equations (3)
Fromx − xpy − yp−c
= kR
X − X0
Y − Y0
Z − Z0
, and R =
r11 r12 r13
r21 r22 r23
r31 r32 r33
,
we can solve for k and insert:
x = xp − cr11(X − X0) + r12(Y − Y0) + r13(Z − Z0)
r31(X − X0) + r32(Y − Y0) + r33(Z − Z0),
y = yp − cr21(X − X0) + r22(Y − Y0) + r23(Z − Z0)
r31(X − X0) + r32(Y − Y0) + r33(Z − Z0).
Fundamentals of Photogrammetry
Mathematical models
Projective geometry
Homogenous coordinates
In projective geometry, points, lines, etc. are represented byhomogenous coordinates.Any cartesian coordinates (x , y) may be transformed tohomogenous by adding a unit value as an extra coordinate:(
xy
)7→
xy1
.
All homogenous vector multiplied by a non-zero scalar kbelong to the same equivalence class and correspond to thesame object. Thus,x
y1
and k
xy1
=
kxkyk
, k 6= 0
all correspond to the same 2D point (x , y)T .
Fundamentals of Photogrammetry
Mathematical models
Projective geometry
Homogenous coordinates (2)
Any homogenous vector (x1, x2, x3)T , x3 6= 0 may betransformed to cartesian coordinates by dividing by the lastelement x1
x2
x3
7→x1/x3
x2/x3
x3/x3
=
x1/x3
x2/x3
1
.
A homogenous vector (x1, x2, x3)T with x3 = 0 is called anideal point and is “infinitely far away” in the direction of(x1, x2).
The point (0, 0, 0)T is undefined.
The space <3 \ (0, 0, 0)T is called the projective plane P2.
A homogenous point in P2 has 2 degrees of freedom.
Fundamentals of Photogrammetry
Mathematical models
Projective geometry
Interpretation of the projective plane P2
A homogenous vectorx ∈ P2 may beinterpreted as a linethrough the origin in <3.
The intersection with theplane x3 = 1 gives thecorresponding cartesiancoordinates.
x3
x1
x2
xO
idealpoint
1
Fundamentals of Photogrammetry
Mathematical models
Transformations
Transformations
Transformation of homogenous 2D points may be describedby multiplication by a 3×3 matrixu
v1
=
a11 a12 a13
a21 a22 a23
a31 a32 a33
xy1
,
or
q = Ap.
Fundamentals of Photogrammetry
Mathematical models
Transformations
Basic transformations — Translation
A translation of points in <2 may be described usinghomogenous coordinates as
q = T (x0, y0)p =
1 0 x0
0 1 y0
0 0 1
xy1
=
x + x0
y + y0
1
.
Fundamentals of Photogrammetry
Mathematical models
Transformations
Basic transformations — Rotation
A rotation may be described using homogenous coordinates as
R(ϕ)p =
cosϕ − sinϕ 0sinϕ cosϕ 0
0 0 1
xy1
=
x cosϕ− y sinϕx sinϕ+ y cosϕ
1
.
Fundamentals of Photogrammetry
Mathematical models
Transformations
Basic transformations — Scaling
Scaling of points in <2 along the coordinate axes may bedescribed using homogenous coordinates as
q = S(k , l)p =
k 0 00 l 00 0 1
xy1
=
kxly1
.
Fundamentals of Photogrammetry
Mathematical models
Transformations
Combination of transformations
Combinations of transformations are constructed by matrixmultiplication:
q = T (x0, y0)R(ϕ)T (−x0,−y0)p
Fundamentals of Photogrammetry
Mathematical models
Transformation classes
Transformation classes
Transformation may be classified based on their properties.
The most important transformations are
Similarity (rigid-body transformation).Affinity.Projectivity (homography).
Fundamentals of Photogrammetry
Mathematical models
Transformation classes
Similarity
A similarity transformation consists of a combinationof rotations, isotropic scalings, and translations.s cos θ −s sin θ tx
s sin θ s cos θ ty0 0 1
or (
sR t0 1
),
where the scalar s is the scaling, R is a 2× 2rotation matrix and t is the translation vector.
A 2D similarity has 4 degrees of freedom.
A similarity preserves angles (and “shape”).
Fundamentals of Photogrammetry
Mathematical models
Transformation classes
Affinity
For an affine transformation the rotation and scalingis replaced by any non-singular 2× 2 matrix Aa11 a12 tx
a21 a22 ty0 0 1
or (
A t0 1
).
A 2D affinity has 6 degrees of freedom.
A similarity preserves parallelity but not angles.
Fundamentals of Photogrammetry
Mathematical models
Transformation classes
Projectivity (Homography)
A projectivity or homography consists of anynon-singular 3× 3 matrix Hh11 h12 h13
h21 h22 h23
h31 h32 h33
.
A 2D projectivity has 8 degrees of freedom.
A projectivity preserves neither parallelity nor angles.
Fundamentals of Photogrammetry
Mathematical models
Transformation classes
The effect of different transformations
Similarity Affinity Projectivity
Fundamentals of Photogrammetry
Mathematical models
Planar rectification
Planar rectification
If the coordinates for 4 points pi and their mappings qi = Hpi
in the image are known, we may calculate the homography H.
From each point pair pi = (xi , yi , 1)T , qi = (x ′i , y′i , 1)T we get
the following equations:x ′iy ′i1
=
u/wv/w
1
, where
uvw
=
h11 h12 h13
h21 h22 h23
h31 h32 h33
xiyi1
or
x ′i = u/w =h11xi + h12yi + h13
h31xi + h32yi + h33,
y ′i = v/w =h21xi + h22yi + h23
h31xi + h32yi + h33.
Fundamentals of Photogrammetry
Mathematical models
Planar rectification
Planar rectification (2)
Rearranging
x ′i (h31xi + h32yi + h33) = h11xi + h12yi + h13,
y ′i (h31xi + h32yi + h33) = h21xi + h22yi + h23.
This equation is linear in hij .
Given 4 points we get 8 equations, enough to uniquelydetermine H assuming the points are in “standard position”,i.e. no 3 points are collinear.
Fundamentals of Photogrammetry
Mathematical models
Planar rectification
Planar rectification (3)
Given H we may apply H−1 to remove the effect of thehomography.
Fundamentals of Photogrammetry
Mathematical models
The camera model
The pinhole camera model
The most commonly used camera model is called the pinholecamera.In the pinhole camera model:
All object points Q are projected via a central projectionthrough the same point C , called the camera center.The object point Q, the camera center C , and the projectedpoint q are collinear.A pinhole camera is straight line-preserving.
Q
C
q
Fundamentals of Photogrammetry
Mathematical models
The camera model
The central projection
If the camera center is at the origin and the image plane is theplane Z = c , the world coordinate (X ,Y ,Z )T is mapped tothe point (cX/Z , cY /Z , c)T in space or (cX/Z , cY /Z ) in theimage plane, i.e.
(X ,Y ,Z )T 7→ (cX/Z , cY /Z )T
Z
X
Y
Q
Oq
x
y
qp
Fundamentals of Photogrammetry
Mathematical models
The camera model
The central projection (2)
If the camera center is at the origin and the image plane is theplane Z = c , the world coordinate (X ,Y ,Z )T is mapped tothe point (cX/Z , cY /Z , c)T in space or (cX/Z , cY /Z ) in theimage plane, i.e.
(X ,Y ,Z )T 7→ (cX/Z , cY /Z )T
O
Y
Z
q
qp
c
cY /Zα
Fundamentals of Photogrammetry
Mathematical models
The camera model
The central projection (3)
The corresponding expression in homogenous coordinates maybe written as
XYZ1
7→
cXcYZ
=
c 0c 0
1 0
XYZ1
.
q P Q
The matrix P is called the camera matrix and maps the worldpoint Q onto the image point q.
In more compact form P may be written as
P = diag(c , c , 1)(I | 0
),
where diag(c , c , 1) is a diagonal matrix and I is the 3× 3identity matrix.
Fundamentals of Photogrammetry
Mathematical models
The camera model
The principal point
If the principal point is not at the origin of the imagecoordinate system, the mapping becomes
(X ,Y ,Z )T 7→ (cX/Z + px , cY /Z + py )T ,
where (px , py )T are the image coordinates for the principalpoint qp.
Z
X
Y
Q
Oq
x
y
qp
Fundamentals of Photogrammetry
Mathematical models
The camera model
The principal point (2)
In homogenous coordinatesXYZ
7→ (cX/Z + px
cY /Z + py
)
becomesXYZ1
7→cX + Zpx
cY + Zpy
Z
=
c px 0c py 0
1 0
XYZ1
Fundamentals of Photogrammetry
Mathematical models
The camera model
The camera calibration matrix
If we write
K =
c px
c py1
,
the projection may be written as
q = K(I | 0
)Q.
The matrix K is known as the camera calibration matrix.
Fundamentals of Photogrammetry
Mathematical models
The camera model
The camera position and orientation
Introduce
Q ′ =
X ′
Y ′
Z ′
1
and q = K(I | 0
)Q ′.
to describe coordinates in the camera coordinate system.The camera and world coordinate systems are identical if thecamera center is at the origin, the X and Y axes coincidewith the sensor coordinate system and the Z axes coincidewith the principal ray.
Z
X
Y
O
X ′Y ′
Z ′
Fundamentals of Photogrammetry
Mathematical models
The camera model
The camera position and orientation (2)
In the general case, the transformation between thecoordinate systems is usually described asX ′
Y ′
Z ′
= R
XYZ
−X0
Y0
Z0
,
where C = (X0,Y0,Z0)T is the camera center in worldcoordinates and the rotation matrix R describes the rotationfrom world coordinates to camera coordinates.
Z
X
Y
O
CX ′
Y ′
Z ′
Fundamentals of Photogrammetry
Mathematical models
The camera model
The camera position and orientation (3)
In homogenous coordinates, this transformation becomes
Q ′ =
(R 00 1
)(I −C0 1
)XYZ1
=
(R −RC0 1
)Q.
The full projection is given by
q = KR(I | − C
)Q.
The equation
q = PQ = KR(I | − C
)Q,
is sometimes referred to as the camera equation.
The 3× 4 matrix P is known as the camera matrix.
Fundamentals of Photogrammetry
Mathematical models
The camera model
The camera position and orientation (4)
If the transformation from the world to the camera is writtenas
Q ′ =
X ′
Y ′
Z ′
= R
XYZ
−X0
Y0
Z0
,
how does the transformation from the camera to the worldlook like?
Z
X
Y
O
CX ′
Y ′
Z ′
Fundamentals of Photogrammetry
Mathematical models
The camera model
Camera coordinates
What are the (Z) coordinates of points in front of the camera?
Z
X
Y
O
CX ′
Y ′
Z ′
Z
X
Y
C
Q
c
ZX
Y
Z0
X0
Y0
xy q
qp
Fundamentals of Photogrammetry
Mathematical models
The camera model
The collinearity equations (revisited)
Given
K =
−c xq−c yp
1
,
the camera equationxy1
= q = K R(I | − C
)Q = K R
(I | − C
)XYZ1
becomes
x = xp − cr11(X − X0) + r12(Y − Y0) + r13(Z − Z0)
r31(X − X0) + r32(Y − Y0) + r33(Z − Z0),
y = yp − cr21(X − X0) + r22(Y − Y0) + r23(Z − Z0)
r31(X − X0) + r32(Y − Y0) + r33(Z − Z0).
Fundamentals of Photogrammetry
Mathematical models
The camera model
Internal and external parameters
The camera equation
q = K R(I | − C
)Q
that describes the general projection for a pinhole camera has9 degrees of freedom: 3 in K (the elements c , px , py ), 3 in R(rotation angles) and 3 for C .The elements of K describes properties internal to the camerawhile the parameters of R and C describe the relationbetween the camera and the world.The parameters are therefore called one of
K R,C
internal parameters external parametersinternal orientation external orientationintrinsic parameters extrinsic parameterssensor model platform model
Fundamentals of Photogrammetry
Mathematical models
The camera model
Aspect ratio
If we have different scale in the x and y directions, i.e. thepixels are not square, we have to include that deformationinto the equation.Let mx and my be the number of pixels per unit in the x andy direction of the image. Then the camera calibration matrixbecomes
K =
mx
my
1
c pxc py
1
=
mxc mxpxmy c mypy
1
=
αx x0
αy y0
1
,
where αx = fmx and αy = fmy is the camera constant inpixels in the x and y directions and(x0, y0)T = (mxpx ,mypy )T is the principal point in pixels.
A camera with unknown aspect ratio has 10 degrees offreedom.
Fundamentals of Photogrammetry
Mathematical models
The camera model
Skew
For an even more general camera model we can add a skewparameter s to describe any non-orthogonality between theimage axis. Then the camera calibration matrix becomes
K =
αx s x0
αy y0
1
.
The complete 3× 4 camera matrix
P = KR(I | − C
)has 11 degrees of freedom, the same as a 3× 4 homogenousmatrix.
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Rotations in <3
A rotation in <3 is usually described as a sequence of 3elementary rotations, by the so called Euler angles.
Warning: There are many different Euler angles and Eulerrotations!
Each elementary rotation takes place about a cardinal axis, x ,y , or z .
The sequence of axis determines the actual rotation.
A common example is the ω − ϕ− κ (omega-phi-kappa orx-y-z) convention that correspond to sequential rotationsabout the x , y , and z axes, respectively.
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Elementary rotations (1)
The first elementary rotation (ω, omega) is about the x-axis.The rotation matrix is defined as
R1(ω) =
1 0 00 cosω − sinω0 sinω cosω
.
Z ′
X ′
Y ′
Z
X
Y
ω
ω
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Elementary rotations (2)
The second elementary rotation (ϕ, phi) is about the y -axis.The rotation matrix is defined as
R2(ϕ) =
cosϕ 0 sinϕ0 1 0
− sinϕ 0 cosϕ
.
Z ′
X ′
Y ′
Z
X
Y
ϕ
ϕ
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Elementary rotations (3)
The third elementary rotation (κ, kappa) is about the z-axis.The rotation matrix is defined as
R3(κ) =
cosκ − sinκ 0sinκ cosκ 0
0 0 1
.
Z ′
X ′
Y ′
Z
X
Y
κ
κ
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Combined rotations
The axes follow the rotated object, so the second rotation isabout a once-rotated axis, the third about a twice-rotatedaxis.
A sequential rotation of 20 degrees about each of the axisis. . .
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Combined rotations (2)
Z
X
Y
ω
ω
Z ′
X ′
Y ′. . . first a rotation about the x-axis. . .
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Combined rotations (3)
Z ′
X ′
Y ′
ϕ
ϕ Z ′′
X ′′
Y ′′. . . followed by a rotation aboutthe once-rotated y -axis. . .
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Combined rotations (4)
Z ′′
X ′′
Y ′′
κ
κ
Z ′′′
X ′′′
Y ′′′. . . followed by a final rotation aboutthe twice-rotated z-axis. . .
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Combined rotations (5)
Z
X
Y
Z ′′′
X ′′′
Y ′′′. . . resulting in a total rotation lookinglike this.
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Rotations in <3 (2)
The inverse rotation is about the same axes in reversesequence with angles of opposite sign.
This is sometimes called roll-pitch-yaw, where the κ angle iscalled the roll angle.
Other rotations: azimuth-tilt-swing (z-x-z), axis-and-angle,etc.
Every 3-parameter-description of a rotation has some rotationwithout a unique representation.
x-y -z if the middle rotation is 90 degrees,z-x-z if the middle rotation is 0 degrees,axis-and-angle when the rotation is zero (axis undefined).
However, the rotation is always well defined.
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Lens distortion
A lens is designed to bend rays of light to construct a sharpimage.
A side effect is that the collinearity between incoming andoutgoing rays is destroyed.
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Lens distortion (2)
Positive radial distortion(pin-cushion)
Negative radial distortion(barrel)
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Lens distortion (3)
The effect of lens distortion is that the projected point ismoved toward or away from a point of symmetry.
The most common distortion model is due to Brown (1966,1971).
The distortion is separated into a symmetric (radial) andasymmetric (tangential) about the principal point:(
xcyc
)=
(xmym
)+
( (xryr
)+
(xtyt
) ).
corrected measured radial tangential
Warning: Someone’s positive distortion is someone else’snegative!
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Lens distortion (4)
The radial distortion is formulated as(xryr
)= (K1r 2 + K2r 4 + . . .)
(xmym
),
for any number of coefficients (usually 1–2), where r is afunction of the distance to the principal point
r 2 = ∆x2 + ∆y 2, and
(∆x∆y
)=
(xm − xpym − yp
).
The tangential distortion is formulated as follows(xtyt
)=
(2P1∆x∆y + P2(r 2 + 2∆x2)2P2∆x∆y + P1(r 2 + 2∆y 2)
),
Fundamentals of Photogrammetry
Mathematical models
Rotations in <3
Lens distortion (5)
The radial distortion follows from that the lens bends rays oflight. It is neglectable only for large focal lengths.
Any tangential distortion is due to de-centering of the opticalaxis for the various lens components. It is neglectable exceptfor high precision measurements.
The lens distortion parameters are usually determined atcamera calibration.
The lens distortion varies with the focal length. To use acalibrated camera, the focal length (and hence any zoom)must be the same as during calibration.
Warning: Some internal parameters are strongly correlated,e.g. the tangential coefficients P1,P2 and the principal point.Any calibration including P1,P2 must have multiple images atroll angles 0 and 90 degrees.
Fundamentals of Photogrammetry
Processing
Overview
Principles
History
Mathematical models
Processing
Applications
Fundamentals of Photogrammetry
Processing
Processing
1 Camera calibration
2 Image acquisition
3 Measurements
4 Spatial resection
5 Forward intersection
6 (Bundle adjustment)
1 Camera calibration
2 Image acquisition
3 Measurements
4 Relative orientation
5 Forward intersection
6 (Bundle adjustment)
7 (Absolute orientation)
Fundamentals of Photogrammetry
Processing
Camera calibration
Camera calibration
Special cameras may be calibrated by measuring deviationbetween input/output rays.
Most of the time, camera calibration is performed by imaginga calibration object or scene.
A 3D scene is preferable, but may be expensive.
A 2D object is easier to manufacture and transport.
Fundamentals of Photogrammetry
Processing
Camera calibration
Camera calibration (2)
Ideally, the calibration situation should mimic the actual scene.
With a 2D object, multiple images must be taken.
Remember: use the same focal setting during calibration andimage acquisition!
If possible, include rolled images of the calibration object.
Fundamentals of Photogrammetry
Processing
Image acquisition
Image acquisition
Camera networks
Parallel (stereo)ConvergentAerialOther
Fundamentals of Photogrammetry
Processing
Image acquisition
Stereo images
Simplified measurements.
Simplified automation.
May be viewed in “3D”.
C1
q1
C2
q2
Q
Fundamentals of Photogrammetry
Processing
Image acquisition
Convergent networks
Stronger geometry.
More than 2measurements per objectpoint.
Should ideally surroundthe object.
C1
q1
C2
q2
C3
q3
Q
Fundamentals of Photogrammetry
Processing
Image acquisition
Aerial networks
Highly structurized.Typically around 60% overlap (along-track) and 30% sidelap(cross-track).
-� -�
Fundamentals of Photogrammetry
Processing
Measurements
Measurements
Fundamentals of Photogrammetry
Processing
Spatial resection
Spatial resection
Determine the external orientation C ,R of the camera frommeasurements and (ground) control points.
Direct method from 3 points — solve 4th order polynomial(Grunert 1841, Haralick 1991, 1994). May have multiplesolutions.
Q1
C
q1
Q2
q2
Q3q3
Fundamentals of Photogrammetry
Processing
Forward intersection
Forward intersection
If the camera external orientations are known, an object pointmay be estimated from measurements in (at least) twoimages.Requires at least two observations.Linear estimation, robust.
C1
q1
C2
q2
Q
Fundamentals of Photogrammetry
Processing
Forward intersection
Forward intersection (2)
From the left camera we know that
Q = C1 + (v1 − C1)α1,
for some value of α1, where v1 are the 3D coordinates of q1.
Similarly, for the right camera
Q = C2 + (v2 − C2)α2.
We have 3+3 equations and 5 un-knowns (Q, α1, α2).
In theory, the point Q is at the in-tersection of the two lines, so wedrop 1 equation and solve the re-maining 5 to get Q.
C1
q1
C2
q2
Q
Fundamentals of Photogrammetry
Processing
Forward intersection
Forward intersection (3)
C1
q1
C2
q2
Q
Q
In reality, the lines may not intersect.
In that case, we may choose to findthe point that is closest to both linesat the same time, i.e. that solves thefollowing minimization problem
minQ,α1,α2
‖Q − l1(α1)‖2 + ‖Q − l2(α2)‖2 ,
or
minQ,α1,α2
∥∥∥∥[Q − (C1 + t1α1)Q − (C2 + t2α2)
]∥∥∥∥2
,
where ti = xi − Ci .
Fundamentals of Photogrammetry
Processing
Forward intersection
Forward intersection (4)
This problem is linear in the unknowns and may be rewritten
minx‖[
I3 −t1 0I3 0 −t2
]︸ ︷︷ ︸
A
Qα1
α2
︸ ︷︷ ︸
x
−[
C1
C2
]︸ ︷︷ ︸
b
‖2.
The solution is given by the normal equations
ATAx = ATb.
Fundamentals of Photogrammetry
Processing
Forward intersection
Forward intersection (5)
C1
q1
C2
q2
C3
q3
Q
Given one more camera, we extend the equation system
minx‖
I3 −t1 0 0I3 0 −t2 0I3 0 0 −t3
︸ ︷︷ ︸
A
Qα1
α2
α3
︸ ︷︷ ︸
x
−
C1
C2
C3
︸ ︷︷ ︸
b
‖2,
with the solution againgiven by the normal equa-tions
ATAx = ATb.
Fundamentals of Photogrammetry
Processing
Forward intersection
Forward intersection (6)
Stereorestitution (“normal case”)
O1 O2X
Z
P = (PX ,PZ )
x xc
B
P1 P2
x1 x2
In photo 1: In photo 2:
X = Zx1
−cY = Z
y1
−cX = B + Z
x2
−cY = Z
y2
−c
Fundamentals of Photogrammetry
Processing
Forward intersection
Forward intersection (7)
If we have non-zero y parallax, i.e. py = y1 − y2 6= 0, we mustapproximate.
Otherwise,
−Zx1
−c= B − Z
x2
−c,
−Z =Bc
x1 − x2=
Bc
px.
Error propagation (first order)
σZ =Bc
p2x
σx =Z
c
Z
Bσx .
The ratio B/Z is the base/object distance.
The ratio Z/c is the scale factor.
Fundamentals of Photogrammetry
Processing
Relative orientation
Relative orientation
C1
p1
p2
p3
p4
p5
C2
q1
q2
q3
q4
q5
One camera fixed, determineposition and orientation ofsecond camera.
Need 5 point pairs measuredin both images.
No 3D information isnecessary.
Direct method (Nister2004). Solve 10th orderpolynomial. May havemultiple solutions.
Fundamentals of Photogrammetry
Processing
Absolute orientation
Absolute orientation
A 3D similarity transformation.
7 degrees of freedom (3 translations, 3 rotations, 1 scale).
Direct method based on singular value decomposition (Arun1987) for isotropic errors.
Fundamentals of Photogrammetry
Processing
Bundle adjustment
Bundle adjustment
Simultaneous estimation of camera external orientation andobject points.
Iterative method, needs initial values.
May diverge.
Fundamentals of Photogrammetry
Applications
Applications
Architecture
Forensics
Maps
Industrial
Motion analysis
Movie industry
Orthopaedics
Space science
Microscopy
GIS
Fundamentals of Photogrammetry
Epipolar geometry
Epipolar geometry
Let Q be an object point and q1 and q2 its projections in twoimages through the camera centers C1 and C2.
The point Q, the camera centers C1 and C2 and the (3Dpoints corresponding to) the projected points q1 and q2 willlie in the same plane.
This plane is called the epipolar plane for C1, C2 and Q.
C1
q1
C2
q2
Q
Fundamentals of Photogrammetry
Epipolar geometry
Epipolar lines
Given a point q1 in image 1, the epipolar plane is defined bythe ray through q1 and C1 and the baseline through C1 andC2.A corresponding point q2 thus has to lie on the intersectingline l2 between the epipolar plane and image plane 2.The line l2 is the projection of the ray through q1 and C1 inimage 2 and is called the epipolar line to q1.
q1q2
e1 e2C1
C2
Q
Fundamentals of Photogrammetry
Epipolar geometry
Epipoles
The intersection points between the base line and the imageplanes are called epipoles.
The epipole e2 in image 2 is the mapping of the cameracenter C1.
The epipole e1 in image 1 is the mapping of the cameracenter C2.
q1q2
e1 e2C1
C2
Q
Fundamentals of Photogrammetry
Epipolar geometry
Examples
Fundamentals of Photogrammetry
Epipolar geometry
RANSAC
Robust estimation — RANSAC
The Random Sample Consensus (RANSAC) algorithm (Fishlerand Bolles, 1981) is an algorithm for handling observationswith large errors (outliers).
Given a model and a data set S containing outliers:
Pick randomly s data points from the set S and calculate themodel from these points. For a line, pick 2 points.Determine the consensus set Si of s, i.e. the set of pointsbeing within t units from the model. The set Si define theinliers in S .If the number of inliers are larger than a threshold T ,recalculate the model based on all points in Si and terminate.Otherwise repeat with a new random subset.After N tries, choose the largest consensus set Si , recalculatethe model based on all points in Si and terminate.
Fundamentals of Photogrammetry
Epipolar geometry
RANSAC
b
da
c
C DB
A
C DB
A