Prof. Feng Liu Winter 2020 http://www.cs.pdx.edu/~fliu/courses/cs410/ 01/30/2020
Prof. Feng Liu
Winter 2020
http://www.cs.pdx.edu/~fliu/courses/cs410/
01/30/2020
Last Time
Hough transform
2
Today
Image alignment
3The following slides are from Prof. S. Lazebnik
Image alignment
Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/
A look into the past
http://blog.flickr.net/en/2010/01/27/a-look-into-the-past/
A look into the past
Leningrad during the blockade
http://komen-dant.livejournal.com/345684.html
Bing streetside images
http://www.bing.com/community/blogs/maps/archive/2010/01/12/new-bing-maps-
application-streetside-photos.aspx
Image alignment: Applications
Panorama stitching
Recognition
of object
instances
Image alignment: Applications
Small degree of overlap, intensity changes
Occlusion,
clutter
Image alignment
◼ Direct (pixel-based) alignment
Search for alignment where most pixels agree
◼ Feature-based alignment
Search for alignment where extracted features agree
Can be verified using pixel-based alignment
Alignment as fitting
• Previous lectures: fitting a model to features in one
image
i
i Mx ),(residual
Find model M that minimizes
M
xi
Alignment as fitting
• Previous lectures: fitting a model to features in one
image
• Alignment: fitting a model to a transformation between
pairs of features (matches) in two images
i
i Mx ),(residual
i
ii xxT )),((residual
Find model M that minimizes
Find transformation T
that minimizes
M
xi
T
xixi'
2D transformation models
• Similarity
(translation,
scale, rotation)
• Affine
• Projective
(homography)
Let’s start with affine transformations
• Simple fitting procedure (linear least squares)
• Approximates viewpoint changes for roughly planar
objects and roughly orthographic cameras
• Can be used to initialize fitting for more complex
models
Fitting an affine transformation
• Assume we know the correspondences, how do we
get the transformation?
),( ii yx ),( ii yx
+
=
2
1
43
21
t
t
y
x
mm
mm
y
x
i
i
i
i
=
i
i
ii
ii
y
x
t
t
m
m
m
m
yx
yx
2
1
4
3
2
1
1000
0100
Fitting an affine transformation
• Assume we know the correspondences, how do we
get the transformation?
),( ii yx ),( ii yx
+
=
2
1
43
21
t
t
y
x
mm
mm
y
x
i
i
i
i
=
i
i
ii
ii
y
x
t
t
m
m
m
m
yx
yx
2
1
4
3
2
1
1000
0100
affine model parameters
Fitting an affine transformation
• Linear system with six unknowns
• Each match gives us two linearly
independent equations: need at least three
to solve for the transformation parameters
=
i
i
ii
ii
y
x
t
t
m
m
m
m
yx
yx
2
1
4
3
2
1
1000
0100
Fitting a plane projective transformation
• Homography: plane projective transformation
(transformation taking a quad to another
arbitrary quad)
Homography
• The transformation between two views of a planar surface
• The transformation between images from two cameras
that share the same center
Application: Panorama stitching
Source: Hartley & Zisserman
Fitting a homography
• Homogeneous coordinates
Converting to homogeneous
image coordinates
Converting from homogeneous
image coordinates
Fitting a homography
• Homogeneous coordinates
• Equation for homography:
Converting to homogeneous
image coordinates
Converting from homogeneous
image coordinates
=
11 333231
232221
131211
y
x
hhh
hhh
hhh
y
x
Fitting a homography
• Equation for homography:
ii xHx =
=
11 333231
232221
131211
i
i
i
i
y
x
hhh
hhh
hhh
y
x
0= ii xHx
−
−
−
=
i
T
ii
T
i
i
T
ii
T
i
T
i
T
i
i
T
i
T
i
T
i
i
yx
x
y
y
x
xhxh
xhxh
xhxh
xh
xh
xh
3
2
1
12
31
23
1
0
0
0
0
3
2
1
=
−
−
−
h
h
h
xx
xx
xx
TT
ii
T
ii
T
ii
TT
i
T
ii
T
i
T
xy
x
y3 equations,
only 2 linearly
independent
Direct linear transform
• H has 8 degrees of freedom (9 parameters, but scale is arbitrary)
• One match gives us two linearly independent equations
• Four matches needed for a minimal solution (null space of 8x9 matrix)
• More than four: homogeneous least squares
0
0
0
0
0
3
2
1111
111
=
−
−
−
−
h
h
h
xx
xx
xx
xx
T
nn
TT
n
T
nn
T
n
T
TTT
TTT
x
y
x
y
0=hA
Geometric approach
25
𝐻= arg min𝐻
σ𝑖=1𝑁 𝑑2 𝑥𝑖 , 𝐻𝑥𝑖
′ + 𝑑2 𝑥𝑖′, 𝐻−1𝑥𝑖
𝐻𝑥𝑖′: The location to which 𝑥𝑖
′ in Image I’ is mapped by H in Image I
Symmetric transfer error to minimize the
geometric errors
𝑑2 𝑥𝑖 , 𝐻𝑥𝑖′ : The error between the projected point and the target point
Robust and large-scale alignment
Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/
RANSAC
❑ The set of putative matches contains a very high
percentage of outliers
❑ RANSAC loop:
✓ Randomly select a seed group of matches
✓ Compute transformation from seed group
✓ Find inliers to this transformation
✓ If the number of inliers is sufficiently large, re-compute least-
squares estimate of transformation on all of the inliers
❑ Keep the transformation with the largest number of
inliers
Homework 2: Image Stitching
Feature detection using SIFT
Feature matching
Homography estimation
◼ Use RANSAC to remove feature outliers
Image warping and blending
28
Image warping
29
Image 1 Image 2
Image warping
30
Image 1
Image warping
31
Image 1
Image warping
32
Image 1
I_pano(x,y) = blend(I1(x, y), I2(H-1(x, y))
Backward warping
Next Time
Warping
33