This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Yuri Boykov, UWO
1: Basics of optimization-based segmentation
- continuous and discrete approaches
2 : Exact and approximate techniques
- non-submodular and high-order problems
3: Multi-region segmentation (Milan)
- high-dimensional applications
Tutorial on Medical Image Segmentation: Beyond Level-SetsMICCAI, 2014 Western University
Canada
www.csd.uwo.ca/faculty/yuri/miccai14_MIS
Yuri Boykov, UWO
Introduction to Image Segmentation implicit/explicit representation of boundaries
• active contours, level-sets, graph cut, etc. Basic low-order objective functions (energies)
• physics, geometry, statistics, information theory
Set functions, submodularity• Exact methods
Approximation methods• Higher-order and non-submodular objectives• Comparison to gradient descent (level-sets)
Part
1Pa
rt 2
Yuri Boykov, UWO
Thresholding
T
Yuri Boykov, UWO
Thresholding
S={ p : Ip < T }
T
Yuri Boykov, UWO
Background Subtraction
?
Thresholding
- =I= Iobj - Ibkg
Threshold intensities above T
better segmentation?
Yuri Boykov, UWO
Good segmentation S ?
Objective function must be specified
Quality function
Cost function
Loss function E(S) : 2P “Energy”
Regularization functional
Segmentation becomes an optimization problem: S = arg min E(S)
Yuri Boykov, UWO
Good segmentation S ?
combining different constraints
e.g. on region and boundary
Objective function must be specified
Quality function
Cost function
Loss function E(S) = E1(S)+…+ En(S)“Energy”
Regularization functional
Segmentation becomes an optimization problem: S = arg min E(S)
Yuri Boykov, UWO
Beyond linear combination of terms
Ratios are also used• Normalized cuts [Shi, Malik, 2000]
• Minimum Ratio cycles [Jarmin Ishkawa, 2001]
• Ratio regions [Cox et al, 1996]
• Parametric max-flow applications [Kolmogorov et al 2007]
)()(
)(SE
SESE
2
1
Not in this tutorial
Yuri Boykov, UWO
Segmentation principles
Boundary seeds• Livewire (intelligent scissors)
Region seeds• Graph cuts (intelligent paint)• Distance (Voronoi-like cells)
Push-relabel [Goldberg-Tarjan, 1986] - good choice for denser grids, e.g. in 3D
Preflow [Hochbaum, 2003] - also competitive
Yuri Boykov, UWO
Optimal boundary in 2D
“max-flow = min-cut”
Yuri Boykov, UWO
Optimal boundary in 3D
3D bone segmentation (real time screen capture, year 2000)
Yuri Boykov, UWO
‘Smoothness’ of segmentation boundary
- snakes (physics-based contours)
- geodesic contours (geometry)
- graph cuts
NOTE: many distance-to-seed methods optimize segmentation boundary only indirectly,
they compute some analogue of optimum Voronoi cells[fuzzy connectivity, random walker, geodesic Voronoi cells, etc.]
(discrete geometry)
Yuri Boykov, UWO
Discrete vs. continuous boundary cost
Geodesic contours
S
s dswSE )(
Both incorporate segmentation boundary smoothness and
alignment to image edges
qp
qppq SSwSE,
][)(
Graph cuts
}10{ ,S p
[Caselles, Kimmel, Sapiro, 1997] (level-sets)[Boykov and Jolly 2001]
C
[Chan, Esidoglu, Nikolova, 2006] (convex) [Boykov and Kolmogorov 2003]
Yuri Boykov, UWO
Graph cuts on a grid and boundary of S
Severed n-links can approximate geometric length of contour C [Boykov&Kolmogorov, ICCV 2003]
This result fundamentally relies on ideas of Integral Geometry (also known as Probabilistic Geometry) originally developed in 1930’s.• e.g. Blaschke, Santalo, Gelfand
}10{ ,S p
Ce
|e|SB )(
Yuri Boykov, UWO
Integral geometry approach to length
C 2
0
a set of all lines L
CL
a subset of lines L intersecting contour C
ddnC L21||||Euclidean length of C :
the number of times line L intersects C
Cauchy-Crofton formula
probability that a “randomly drown” line intersects C
Yuri Boykov, UWO
Graph cuts and integral geometry
C
k
kkknC 21||||
Euclidean length
2kk
kw
gcC ||||graph cut cost
for edge weights:the number of edges of family k intersecting C
Edges of any regular neighborhood system
generate families of lines
{ , , , }
Graph nodes are imbeddedin R2 in a grid-like fashion
Length can be estimated without computing any derivatives
Yuri Boykov, UWO
Metrication errors
“standard” 4-neighborhoods
(Manhattan metric)
larger-neighborhoods8-neighborhoods
Euclidean metric
Riemannianmetric
Yuri Boykov, UWO
Metrication errors
4-neighborhood 8-neighborhood
Yuri Boykov, UWO
implicit (region-based) representation of contours
Differential vs. integral approach to geometric boundary length
ddnC L21||||
Cauchy-Crofton formula
1
0|||| dtCC t
Inte
gra
l g
eom
etr
yD
iffere
nti
al
geom
etr
y
Parametric(explicit)contour
representation
dxuC
|||||| Level-setfunction
representation
Yuri Boykov, UWO
A contour may be approximated from u(x,y) with
sub-pixel accuracy
C
-0.8 0.2
0.5
0.70.30.6-0.2
-1.7
-0.6
-0.8
-0.4 -0.5
)()( y,xupu
• Level set function u(p) is normally stored on image pixels• Values of u(p) can be interpreted as distances or heights of image pixels
Implicit (region-based) surface representation via level-sets
Yuri Boykov, UWO
Implicit (region-based) surface representation via graph-cuts
)( y,xSS p
• Graph cuts represent surfaces via binary labeling Sp of each graph node• Binary values of Sp indicate interior or exterior points (e.g. pixel
centers)
There are many contours satisfying
interior/exterior labeling of points
Question: Is this a contour to be reconstructed from binary labeling Sp ? Answer: NO
0 1
1
1 1 1 0
0
0
0
0 0
C
Yuri Boykov, UWO
Contour/surface representations(summary)
Implicit (area-based) Explicit (boundary-based)
Level sets (geodesic active contours)
Graph cuts(minimum cost cuts)
Live-wire(shortest paths on graphs)
Snakes (physics-based band model)
What else besides boundary length |∂S| ?
Yuri Boykov, UWO
From seeds to more general region constraints
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
assume are known
“expected” intensities of object and background
ts II and
|II|tD tpp )(
|II|sD spp )(
SS
segmentation
[Boykov and Jolly 2001]
cost of severed t-links
Sp
tp
Sp
sp IIII ||||E(S) = +
cost of severed n-links|| S
Yuri Boykov, UWO
From seeds to more general region constraints
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
could be unknown intensities
of object and background
ts II and
|II|tD tpp )(
|II|sD spp )(
SS
segmentation
[Boykov and Jolly 2001]
cost of severed t-links
Sp
tp
Sp
sp IIII ||||E(S, Is,It) = +
cost of severed n-links|| S
Chan-Vese model
re-estimatets II and
Block-Coord.Descent
Yuri Boykov, UWO
Block-coordinate descent for
Minimize over labeling S for fixed I 0, I 1
Minimize over I 0, I 1 for fixed labeling S
1:
21
0:
2010 )()(),,(pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][
),,( 10 IISE
1:
21
0:
2010 )()(),,(pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][fixed for S=const
optimal L can be computed using graph cuts
optimal I 1, I 0 can be computed by minimizing squared errors inside object and background
segments
0:
||10ˆ
pSppS
II
1:
||11ˆ
pSppS II
Yuri Boykov, UWO
Chan-Vese segmentation(binary case )
1:
21
0:
2010 )()(),,(pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][
}10{ ,S p
Yuri Boykov, UWO
Chan-Vese segmentation(could be used for more than 2 labels )
...)()(,...),,(1:
21
0:
2010
pp Sp
pSp
p IIIIIISE
Npq
qppq SSw}{
][
can be used for segmentation, to reduce color-depth,or to create a cartoon
},...2,1,0{pS
Yuri Boykov, UWO
without the smoothing term, this is like “K-means” clustering in the color space
Chan-Vese segmentation(could be used for more than 2 labels )
...)()(,...),,(1:
21
0:
2010
pp Sp
pSp
p IIIIIISE
can be used for segmentation, to reduce color-depth,or to create a cartoon
},...2,1,0{pS
joint optimization
over S and I0, I1,… is NP-hard
Yuri Boykov, UWO
From fixed intensity segmentsto general intensity distributions
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
Appearance models can be
given by intensity distributions
of object and background
)()( t|IPrlntD pp
)()( s|IPrlnsD pp
SS
segmentation
[Boykov and Jolly 2001]
cost of severed t-links
Sp
pSp
p tIsI )|Pr(ln)|Pr(lnE(S) = +cost of severed n-links
|| S
Yuri Boykov, UWO
Graph cut (region + boundary)
Yuri Boykov, UWO
Graph cut as energy optimization for S
pqw
n-links
s
t a cut)(sDp
t-lin
k
)(tDp
t-link
segmentation cut
SS
cost of severed t-links
Sp
pSp
p DD )0()1(cost(cut) = +E(S)
}10{ ,S p
unary terms pair-wise terms
cost of severed n-links
Npq
pq SSw ][ qpp
pp SD )(
regional properties of S boundary smoothness for S
[Boykov and Jolly 2001]
Yuri Boykov, UWO
Unary potentials as linear term wrt.
p
pp SD )(unary terms
Sp
pSp
p DD )0()1(
)( pf
Ssfp
pp f, =
pp
pp SDDconst )0()1(
}10{ ,S p
p
ppp SDSD )-(1)0()1( p
Linear region term analogous to
geodesic active contours
S
fSE )(
Yuri Boykov, UWO
p
pp sfSf ,
Examples of potential functions f
unary terms(linear)
2)( cIf pp • Chan-Vese
1pf• Volume Ballooning
patresponsefilterf p • Attention
)( pp If Prln• Log-likelihoods
Unary potentials as linear term wrt.
Yuri Boykov, UWO
In general,…
pS
Npq
pq SSw ][ qp
pair-wise terms
Npq
pq SSSSw qpqp )1()1(
quadratic polynomial wrt.
k-arity potentials are k-th order polynomial
Quadratic term analogous to boundary
length ingeodesic active
contours
S
sw dswSSE ||)(
Yuri Boykov, UWO
{0,1}ps
Examples of discontinuity penalties w
• Euclidean boundary length
second-order terms
][|| qpNpq
pqw sswS
(quadratic)
• contrast-weighted boundary length
2)( qppq IIw exp~
[Boykov&Jolly, 2001]
Basic (quadratic) boundary regularization
||~
pqwpq
1
[Boykov&Kolmogorov, 2003], via integral geometry
Yuri Boykov, UWO
w|S|Sf,E(S)
Basic second-order segmentation energy
Sp
pf
Npq
pq SSw ][ qp
includes linear and quadratic terms
Yuri Boykov, UWO
w|S|Sf,E(S)
Basic second-order segmentation energy
includes linear and quadratic terms
MAIN ADVANTAGE: guaranteed global optimum (t.e. best segmentation w.r.t. objective) (discrete case) via graph cuts
[Boykov&Jolly’01; Boykov&Kolmogorov’03] public code [BK’2004], fast on CPU
(continuous case) via convex TV formulations[Chen,Esidoglu,Nikolova’06; Chambolle,Pock,Cremers’08] public code [C. Nieuwenhuis’2014], comparably fast on GPU
NOTE: this formulation is different from basic level-sets [Osher&Sethian’89]
Yuri Boykov, UWO
Sf,E(S) B(S)Sf,E(S)
Optimization vs Thresholding
I
Fg)|Pr(I Bg)|Pr(I
Sp
pfE(S)
bg)|Pr(I(p)
fg)|Pr(I(p)lnf(p)
S
thresholding e.g. graph cut [BJ, 2001]
Yuri Boykov, UWO
Other examples of usefulglobally optimizable segmentation objectives
Flux [Boykov&Kolmogorov 2005]
Color consistency [“One Cut”, Tang et al. 2014]
Distance ||S-S0|| from template shape• Hamming, L2,… [e.g. Boykov,Cremers,Kolmogorov, 2006]
Star-shape prior [Veksler 2008]
NOT ALLOWED
Yuri Boykov, UWO
Many more example of usefulhard-to-optimize segmentation objectives
Continuous case• Non-convexity
Discrete case• Non-submodularity (more later)• High-order• Density (too many terms)
Typical Problems: Typical Solutions:
gradient descent (linearization)
+ level sets
to be discussed
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
psE(S) 00 VSV ||
E(S)
|| S0V
can not be represented as a sum of simpler (unary or quadratic)
terms
high-orderpotential
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
E(S)
|| S0V
can be represented as a sum of unary and quadratic terms
NOTE: 2nd-orderpotential
still difficult to optimize(completely connected graph)
2
02
0 psVSV ||E(S)
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
m
pm sVSV 00 ||E(S)
E(S)
|| S0V
can not be represented as a sum of simpler (unary or quadratic)
terms
high-orderpotential
Yuri Boykov, UWO
Examples of useful higher-order energies
Cardinality potentials (constraints on segment size)
Curvature of the boundary Shape convexity Segment connectivity Appearance entropy, color consistency Distribution consistency High-order shape moments …
Yuri Boykov, UWO
Implicit surface representationGlobal optimization is possible
Summary
Thresholding, region growing Snakes, active contours Geodesic contours Graph cuts (binary labeling, MRF)
Covered basics of: Not-Covered: Ratio functionals Normalized cuts Watersheds Random walker Many others…