Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision
Post on 26-Jan-2020
0 Views
Preview:
Transcript
The University of
Ontario
How to fit a surface to a point cloud?or
optimization of surface functionals in computer vision
Yuri Boykov
TRICS seminarComputer Science Department
The University of
Ontario
Optimization of surface
functionals in computer vision
Computer vs. human vision
Model fitting in computer vision• templates, pictorial structures, trees, deformable models, contours/snakes, meshes,
surfaces, complexes, graphs, weak-membrane model, Mumford-Shah, Potts
model,……
Optimization in computer vision• dynamic programming, gradient descent, PDEs, shortest paths, min. spanning
trees, linear and quadratic programming, primal-dual schema, network flow
algorithms, QPBO, ...
Applications
• segmentation, stereo, multi-view reconstruction, optical flows
• surface fitting
The University of
Ontario+Texture
The New Yorker Album of Drawings, The Viking Press, NY, 1975
recognizing 3D perspective
The University of
OntarioWhat do humans get by „looking‟?
J. VermeerThe Guitar Player
Contours
Shading
Color
Texture
…
basic image cues:
The University of
OntarioWhat do humans get by „looking‟?
Contours
Shading
Color
Texture
…
basic image cues:
Segmentation
Motion
3D shape perception
3D scene geometry
Detection/Recognition
…
higher-level perception:
The University of
OntarioWhat do computers get by „looking‟?
x
y
3D plot of image
intensity I(x,y)x
y
I(x,y)
x
y
The University of
OntarioWhat do computers get by „looking‟?
P. PicassoThe Guitar Player
Intensity discontinuities(contours)
Intensity gradients (shading)
Multi-valued intensities (color)
Filtering (e.g. texture)
…
basic image cues: higher-levelgrouping?
The University of
Ontario
model
Bayesian approach
Prior + Data Low-level cues
(local info)high-level knowledge
(global picture)
Fit some prior model into data
The University of
OntarioRigid Template Matching
• In matching we estimate “position” of a rigid template in the image • “Position” includes global location parameters of a rigid template:
- translation, rotation, scale,…
Face template
image
image
translation,rotation,scaling
The University of
OntarioNon-rigid (parametric) matching
1. Pick one image (red)
2. Warp the other images to match it (homographic transform)
3. Blend
panorama mosaicing
The University of
Ontarioe.g…. using flexible templates
• In flexible template matching we estimate “position” of each rigid component of a template• For tree-structured models, efficient global optimization is possible via DP
(Felzenswalb&Huttenlocher 2002)
The University of
Ontariotracking parameters => activity recognition
Bottom-up tracker
The University of
Ontario
5-18
deformable contours (“snakes”)
2D curve which matches to image data
Initialized near target, iteratively refined
Can restore missing data
initial intermediate final
Optimization gets harder when a loop is introduced. DP does not apply.
One solution: gradient descent
Kass, Witkin, Terzopoulos 1987
The University of
Ontario
6-19Cremers, Tischhäuser, Weickert, Schnörr, “Diffusion Snakes”, IJCV '02
local minima, fixed contour topology
The University of
Ontario
6-20
A contour may be approximated from
u(x,y) with near sub-pixel accuracy
C
-0.8 0.2
0.5
0.70.30.6-0.2
-1.7
-0.6
-0.8
-0.4 -0.5
),( ppp yxuu
• Level set function u(x,y) is normally discretized/stored over image pixels
• Values of u(p) can be interpreted as distances or heights of image pixels
Implicit representation of contours
Osher&Sethian 1989
The University of
Ontario
6-21
[Visualization is courtesy of O. Juan]
Simple evolution
Morphological Operation:Erosion
1p
ppp NCd
|| ppp udu
The University of
Ontario
6-22
Visualization is courtesy of O. Juan
Example of gradient descent evolution
2
2
2
2
)(y
u
x
up dtu
Gradient descent
w.r.t.
Euclidean length
The University of
Ontario
6-23
Example of gradient descent evolution
Laplacian
Osher&Sethian 1989
Gradient descent
w.r.t.
Euclidean length
2
2
2
2
)(y
u
x
up dtu
The University of
Ontario
6-24
[example from Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE TIP ‟01]
Geodesic Active Contours via Level-sets
y
ug
x
ug
p
yxdtu)()(
)(
C
dsgCE )()(
The University of
Ontario
6-25
Other geometric energy functionals
besides length [courtesy of Ron Kimmel]
weighted length
C
dsgCE )()(
Functional E( C ) gradient descent evolution
,~ Ngg
weighted area dafCE )( ~ f
alignment(flux)
C
dsNCE ,)(
v )div(~ v
Geometric measures commonly used in segmentation
NdC
The University of
Ontario
in 3D…
deformable meshes, level-sets, …
Estimation of positionfor mesh points
Many loops.optimization - gradient
descent
GOALS: global optima (?)“right” functional (?)
Typical problems:- local minima (clutter, outliers)
-over-smoothing
The University of
Ontario
More generally...
Estimate labels for graph nodes
I
p
L
along one scan line in the image
observed noisy image I image labeling L(restored intensities)
NOTE:similar to robust regression modelestimation
The University of
Ontario
(simple example)
Piece-wise smooth restoration
Markov Random Fields (MRF) approach
weak membrane model (Geman&Geman‟84, Blake&Zisserman‟83,87)
pL qL
Nqp
qp
p
pp LLVILLE),(
2 ),()()(
discontinuity preserving prior
optimizing E(L) is NP hard!
T T
(Continuous analogue: Mamford-Shah functional, 1989)
The University of
Ontario
I
p
L
observed noisy image I image labeling L(restored intensities)
(simple example)
Piece-wise constant restoration
along one scan line in the image
The University of
Ontario
(simple example)
Piece-wise constant restoration
Potts model Boykov Veksler Zabih ’01
Greig et al.’89 (for 2 labels)
Nqp
qp
p
p
i
i
LLILLEi ),(
2 )()()(
global optimization is still NP hard, but there are fast provably good
combinatorial approximation algorithms, linear and quadratic programming,
QPBO, primal-dual schema
pL qL
0}2L:p{ p2
“perceptual grouping”
}1L:p{ p1
}0L:p{ p0
The University of
OntarioPerceptual grouping from stereo
(Birchfield &Tomasi‟99)
constant label = plane
The University of
Ontario
Binary labeling(binary image restoration)
)|Pr()()( 2
pppppp LIILLD
}1,0{pL
original binary image I optimal binary labeling L
Nqp
qp
p
pp LLLDLE),(
)()()(
Greig Porteous Seheult ‟89
Globally optimal solution is possible using combinatorial graph cut algorithms• pseudo-boolean optimization Hammer‟65, Picard&Ratlif’75
The University of
Ontario
Binary labeling(object extraction)
1pL
0pL
object segmentation
left ventricle of heart
}1,0{pL
The University of
Ontario
Binary labeling(object extraction)
1
0
C
Globally optimal solution is possible using graph cut algorithms• pseudo-boolean optimization (Hammer‟65, Picard&Ratlif‟75)
surface extraction
Nqp
qppq
p
pp LLwLDLE),(
)()()(
Boykov&Jolly‟01
left ventricle of heart
}1,0{pL
The University of
OntarioImplicit surface representation via graph-cuts
• Any contour (or surface in 3D) satisfying labeling of exterior/interiorpoints (pixel centers) is acceptable if some explicit surface has to be output.
0 1
1
1110
0
0
0
0 0
The University of
Ontario
)int(
)(,)()(CC
x
C
dpxfdsdsgCE vN
Geometric lengthany convex,
symmetric metric (e.g. Riemannian)
Flux
any vector field v
Regional bias
any scalar function f
“edge alignment”
Tight characterization for geometric functionals of contour C that can
be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05)
disclaimer: for pairwise interactions only
Global optimization of
geometric surface functionals
N)q,p(
qp
p
pp )L,L(V)L(D)L(E
The University of
OntarioGlobally optimal surfaces in 3D
Volumetric segmentation(BJ01,BK’03,KB’05)
The University of
Ontario
Binary labeling(object extraction)
Blake et al.‟04, Rother et al.‟04
}1,0{pL
iteratively re-estimate color models e.g. using mixture of Gaussians
The University of
OntarioOptimal surfaces in 3D
3D reconstruction
Vogiatzis, Torr, Cippola‟05
Local cues: voxel’s photoconsistency
Prior:smoothness,
projective geometry constraints
The University of
OntarioGlobally optimal surfaces in 3D
Lempitsky&Boykov, 2006
from a cheap digital camera
The University of
Ontario
3D model
multi-view reconstruction set up
Furukawa&Ponce 2006
Globally optimal surfaces in 3D
(texture mapped)
The University of
OntarioSurface fitting to point cloud
a cloud of 3D points (e.g. from a laser scanner)
3D model:
surface fitting:
The University of
OntarioSurface fitting functional
CC
dsdsCE 1,)( vN
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
ip
in
S
data fit prior
The University of
OntarioOptimal surfaces in 3D
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
From 10 views
No initialization is needed
CC
dsdsCE 1,)( vN
The University of
OntarioGlobal vs. local optimization
regional potentials
)(vdivf
CC
dsdpdivCE 1)()int(
v
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
initial solution local minima global minima
The University of
OntarioSummary
Global optimization-Your solution is as good as your functional
-No need to worry about initial guess or convergence issues
-Polynomial algorithms, but many practical
issues (efficient data structures, memory limitations,
parallelization, dynamic applications,…)
- Many useful functionals are NP hard (lots of approximation methods are developed)
- New approaches allowing global optimization
are introduced (including new version of level-sets)
top related