Fitting a transformation: feature-based alignment Thursday, September 24 th 2015 Devi Parikh Virginia Tech 1 Slide credit: Kristen Grauman Disclaimer: Many slides have been borrowed from Kristen Grauman, who may have borrowed some of them from others. Any time a slide did not already have a credit on it, I have credited it to Kristen. So there is a chance some of these credits are inaccurate.
81
Embed
Fitting a transformation: feature-based alignment Thursday, September 24 th 2015 Devi Parikh Virginia Tech 1 Slide credit: Kristen Grauman Disclaimer:
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Fitting a transformation:feature-based alignment
Thursday, September 24th 2015Devi Parikh
Virginia Tech
1Slide credit: Kristen Grauman
Disclaimer: Many slides have been borrowed from Kristen Grauman, who may have borrowed some of them from others. Any time a slide did not already have a credit on it, I have credited it to Kristen. So there is a chance some of these credits are inaccurate.
Announcements
• Project proposals– Due Tuesday– Teams of > 2– Look at class webpage for guidelines
• PS2 out– Due October 5th
• PS1 graded– Grades will be released soon
2Slide credit: Adapted by Devi Parikh from Kristen Grauman
Given: initial contour (model) near desired object
a.k.a. active contours, snakes
Figure credit: Yuri Boykov
Goal: evolve the contour to fit exact object boundary
[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]
Main idea: elastic band is iteratively adjusted so as to• be near image positions with
high gradients, and• satisfy shape “preferences” or
contour priors
Last time: Deformable contours
3
Slide credit: Kristen Grauman
Last time: Deformable contours
Image from http://www.healthline.com/blogs/exercise_fitness/uploaded_images/HandBand2-795868.JPG Kristen Grauman
4
Recap: deformable contour
• A simple elastic snake is defined by:– A set of n points,– An internal energy term (tension,
bending, plus optional shape prior)– An external energy term (gradient-based)
• To use to segment an object:– Initialize in the vicinity of the object– Modify the points to minimize the total
energy
Kristen Grauman
5
),( 44 nvvE),( 433 vvE
)3(3E
)(3 mE )(4 mE
)3(4E
)2(4E
)1(4E
)(mEn
)3(nE
)2(nE
)1(nE
)2(3E
)1(3E
)(2 mE
)3(2E
),(...),(),( 11322211 nnntotal vvEvvEvvEE
),( 322 vvE
)1(2E
)2(2E
),( 211 vvE
0)1(1 E
0)2(1 E
0)3(1 E
0)(1 mE
Main idea: determine optimal position (state) of predecessor, for each possible position of self. Then backtrack from best state for last vertex.
states
1
2
…
m
vert
ices
1v 2v 3v 4v nv
)( 2nmOComplexity: vs. brute force search ____?
Viterbi algorithm
Example adapted from Y. Boykov
6
Slide credit: Kristen Grauman
),(...),(),( 11322211 nnn vvEvvEvvE DP can be applied to optimize an open ended snake
For a closed snake, a “loop” is introduced into the total energy.
1n
),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn
1n
2
1n
3 4
Work around:
1) Fix v1 and solve for rest .
2) Fix an intermediate node at its position found in (1), solve for rest.
Energy minimization: dynamic programming
8Slide credit: Kristen Grauman
Aspects we need to consider
• Representation of the contours• Defining the energy functions
– External– Internal
• Minimizing the energy function• Extensions:
– Tracking– Interactive segmentation
9
Slide credit: Kristen Grauman
Tracking via deformable contours
1. Use final contour/model extracted at frame t as an initial solution for frame t+1
2. Evolve initial contour to fit exact object boundary at frame t+1
3. Repeat, initializing with most recent frame.
Tracking Heart Ventricles (multiple frames)
Kristen Grauman
10
Visual Dynamics Group, Dept. Engineering Science, University of Oxford.
Traffic monitoringHuman-computer interactionAnimationSurveillanceComputer assisted diagnosis in medical imaging
Limitations• External energy: snake does not really “see” object
boundaries in the image unless it gets very close to it.
image gradientsare large only directly on the boundary
I
14
Slide credit: Kristen Grauman
Distance transform• External image can instead be taken from the distance
transform of the edge image.
original -gradient distance transform
edges
Value at (x,y) tells how far that position is from the nearest edge point (or other binary mage structure) >> help bwdist
Kristen Grauman
15
Deformable contours: pros and cons
Pros:• Useful to track and fit non-rigid shapes• Contour remains connected• Possible to fill in “subjective” contours• Flexibility in how energy function is defined, weighted.
Cons:• Must have decent initialization near true boundary, may
get stuck in local minimum• Parameters of energy function must be set well based on
prior information
Kristen Grauman
16
Summary
• Deformable shapes and active contours are useful for
– Segmentation: fit or “snap” to boundary in image– Tracking: previous frame’s estimate serves to initialize the next
• Fitting active contours:
– Define terms to encourage certain shapes, smoothness, low curvature, push/pulls, …
– Use weights to control relative influence of each component cost – Can optimize 2d snakes with Viterbi algorithm.
• Image structure (esp. gradients) can act as attraction force for interactive segmentation methods.
• We have previously considered how to fit a model to image evidence– e.g., a line to edge points, or a snake to a deforming contour
• In alignment, we will fit the parameters of some transformation according to a set of matching feature pairs (“correspondences”).
T
xixi
'
Kristen Grauman
56
Image alignment
• Two broad approaches:– Direct (pixel-based) alignment
• Search for alignment where most pixels agree
– Feature-based alignment• Search for alignment where extracted features agree
• Can be verified using pixel-based alignment 57
Slide credit: Kristen Grauman
Fitting an affine transformation• Assuming we know the correspondences, how do we
get the transformation?
),( ii yx ),( ii yx
2
1
43
21
t
t
y
x
mm
mm
y
x
i
i
i
i
58
Slide credit: Kristen Grauman
An aside: Least Squares ExampleSay we have a set of data points (X1,X1’), (X2,X2’),
(X3,X3’), etc. (e.g. person’s height vs. weight)
We want a nice compact formula (a line) to predict X’s from Xs: Xa + b = X’
We want to find a and b
How many (X,X’) pairs do we need?
What if the data is noisy?
'22
'11
XbaX
XbaX
'2
'1
2
1
1
1
X
X
b
a
X
XAx=B
.........
1
1
1
'3
'2
'1
3
2
1
X
X
X
b
a
X
X
X
overconstrained
2min BAx
Source: Alyosha Efros
59
Fitting an affine transformation• Assuming we know the correspondences, how do we
get the transformation?
),( ii yx ),( ii yx
2
1
43
21
t
t
y
x
mm
mm
y
x
i
i
i
i
i
i
ii
ii
y
x
t
t
m
m
m
m
yx
yx
2
1
4
3
2
1
1000
0100
60
Slide credit: Kristen Grauman
Fitting an affine transformation
• How many matches (correspondence pairs) do we need to solve for the transformation parameters?
• Once we have solved for the parameters, how do we compute the coordinates of the corresponding point for ?
• Where do the matches come from?
i
i
ii
ii
y
x
t
t
m
m
m
m
yx
yx
2
1
4
3
2
1
1000
0100
),( newnew yx
Kristen Grauman
61
What are the correspondences?
?
• Compare content in local patches, find best matches.e.g., simplest approach: scan with template, and compute SSD or correlation between list of pixel intensities in the patch
• Later in the course: how to select regions according to the geometric changes, and more robust descriptors.
Kristen Grauman
62
Fitting an affine transformation
Figures from David Lowe, ICCV 1999
Affine model approximates perspective projection of planar objects.