Snakes with Some Math. Acknowledgements: University of Western Ontario, University of Manchester, demos from Visual Dynamics Group (University of Oxford),

Snakes with Some Math

Acknowledgements: University of Western Ontario, University of Manchester, demos from Visual Dynamics Group

(University of Oxford),

Snakes

Given: initial contour (model) near desirable object

•Snakes, active contours [Kass, Witkin, Terzopoulos 1987]

•In general, deformable models are widely used

Snakes

•Snakes, active contours [Kass, Witkin, Terzopoulos 1987]

•In general, deformable models are widely used

Given: initial contour (model) near desirable object

Goal: evolve the contour to fit exact object boundary

Tracking via deformable models

1. Use final contour/model extracted at frame t as an initial solution for frame t+1

2. Evolve initial contour to fit exact object boundary at frame t+1

3. Repeat steps 1 and 2 for t ‘= t+1

Tracking via deformable models

Tracking Heart Ventricles

“Snakes”

• A smooth 2D curve which matches to image data• Initialized near target, iteratively refined• Can restore missing data

initial intermediate final

Parametric Curve Representation

(continuous case)• A curve can be represented parametrically

open curve closed curve

10))(),(()( ssysxs

Note: in computer vision and medical image analysis communities the term “snake” is normally associated with such parametric representation of contours.

Internal Energy

• The bending energy of a continuous curve is

The more the curve bends, the larger this value

Elasticity Stiffness

sd

ddsd

sssEin 2

2

)()())((

22

1

0

))(( dssEE inin

External energy

• The external energy describes how well the curve matches the image data locally

• Numerous forms can be used, attracting the curve toward different image features

Simple Edge Strength

• Suppose we have an image I(x,y)• Generate gradient images &• External energy based on edge strength at

a point is then

• An external energy term for a snake is thus

),( yxGx),( yxGy

)|))((||))((|())(( 22 sGsGsE yxex (Negative in order that minimizing it forces the curve toward large edges)

1

0

))(( dssEE exex

Snake Energy (continuous form)

• The total energy of the snake is defined as

e.g. bending energy

e.g. total edge strength under curve

exintotal EEE

1

0

))(( dssEE inin

1

0

))(( dssEE exex

Discrete approach

discrete imagediscrete snake representation

discrete optimization(dynamic programming)

Discrete and Continuous approaches

ContinuousModels in

Image Analysis

DiscreteModels in

Image Analysis

Continuous Model of LightElectro-Magnetic Waves

Discrete Model of LightPhotons

The relationship is somewhat analogous to relation between continuous and discrete models in physics,

e.g.

Parametric Curve Representation

(discrete case)• Represent the curve with a set of n points

10),( niyx iii

Discrete Representation

• If the curve is represented by n points

Elasticity Stiffness

10),( niyx iii

211

iiv

ds

d 11112

2

2)()( iiiiiiids

d

1

0

211

21 |2|||

n

iiiiiiinE

Simple Elastic Curve (example)

• For a curve represented as a set of points a simple elastic energy term is

This encourages the closed curve to shrink to a

point (like a very small elastic band)

1

0

2n

iiin LE

21

1

0

21 )()( ii

n

iii yyxx

Encouraging point spacing

• To stop the curve from shrinking to a point, add a term such as [v-v’]^2 where v’ is average distance between snaxels.

Simple Edge Strength

• An external energy term for a (discrete) snake based on image edge

(Negative in order that minimizing it forces the curve toward large edges)

21

0

2 |),(||),(| iiy

n

iiixex yxGyxGE

Simple Elastic Snake

• A simple elastic snake is thus defined by– A set of n points,– An internal elastic energy term– An external edge based energy term

• To use this to locate the outline of an object– Initialize in the vicinity of the object– Modify the points to minimize the total energy

Simple Elastic Snake

• In this case we have

• where

• Optimization problem (2n variables)– note that we can easily compute the derivatives, which

allows efficient optimization (via gradient descent) converging to a local minima

• more robust option: Dynamic Programming

)()()( xExExE exintotal

Tnn yyxxx ),,,,,( 1010

Synthetic example

(1) (2)

(3) (4)

Dealing with missing data

• The smoothness constraint can deal with missing data (sometimes maybe wrong!):

Relative weighting

• Notice that the strength of the internal elastic component can be controlled by a parameter,

• Increasing this increases stiffness of curve

large small medium

1

0

2n

iiin LE

Note: values of energy are normalized from 0 to 1, so if 0.1 times 10 becomes .01, etc.,Then re-scale it. Basically this weight makes it more sensitive to internal energy.

Simple shape prior

• If object is some smooth variation on a known shape, use

• where give points of the basic shape

1

0

2)ˆ(n

iiiinE

}ˆ{ i

Alternative External (image) Energies

• Directed gradient measures

– Where is the unit normal to the boundary at contour point

– This gives a good response when the boundary has the same direction as the edge, but weaker responses when it does not

1

0,, )()(

n

iiyiyixixex GuGuE

),( ,, iyixi uuu i

Open and Closed Curves

• When using an open curve we can impose constraints on the end points (e.g. end points may have fixed position)

open curveclosed curve

n 0 0

1n

Additional Constraints

• Snakes originally developed for interactive image segmentation

• Initial snake result can be nudged where it goes wrong

(correct the incorrect ones)

• Simply modify the external energy term to

– Pull nearby points toward cursor, or

– Push nearby points away from cursor

Interactive (external) forces

• Pull points towards cursor:

Nearby points get pulled hardest

Negative sign gives better energy for

positions near p

1

02

2

||

n

i ipull p

rE

Interactive (external) forces

• Push points from cursor:

Nearby points get pushed

hardest

Positive sign gives better energy for

positions far from p

1

02

2

||

n

i ipush p

rE

Dynamic snakes

• Adding motion parameters into hidden variables (for each snake node)

• Introduce energy terms for motion consistency

• Example: use optic flow to get candidate

points for the snaxels in the next frame.

Discrete Snakes Optimization• At each iteration we compute a new snake position within proximity to the previous snake• New snake energy should be smaller than the previous one• Stop when the energy can not be decreased within local neighborhood of the snake (local energy minima)

Optimization Methods

1. Gradient Descent

2. Dynamic Programming

Gradient Descent

i

i

yE

xE

E

negative gradient at point (x,y) gives direction of the steepest descent towards lower values of

function E

• Example: minimization of functions of 2 variables

),( yxE

),( 00 yx

Gradient Descent

Etpp

• Example: minimization of functions of 2 variables

),( yxE

),( 00 yx

yE

xE

ty

x

y

x

Stop at a local minima where 0

E

Gradient Descent

),( yxE

• Example: minimization of functions of 2 variables

High sensitivity wrt. the initialisation !!

Gradient Descent for Snakessimple elastic snake energy

21

0

21010 |),(||),(|),,,,,( iiy

n

iiixnntotal yxGyxGyyxxE

21

1

0

21 )()( ii

n

iii yyxx

i

i

yE

xE

iF

iF EF

Force is a negative gradient of energy function (2n-

dimentional vector)

energy as a function 2n variables

1

1

0

...

nF

F

F

We can break force/gradient into components

corresponding to individual snake nodes

Discrete Snakes:“Gradient Flow” evolution

1,,0 nitFiii

Update equation for each node

i

Stopping criteria:

iallforEF ii 0|

at a local minima of energy

i i

EdtdC Contour evolution via

“Gradient flow”

Difficulties with Gradient Descent

• Very difficult to obtain accurate estimates of high-order derivatives on images (discretization errors)– E.g., estimating requires computation of

• Gradient descent is not trivial even for one-dimensional functions. Robust numerical performance for 2n-dimensional function is problematic. – Choice of parameter is non-trivial

• Small , the algorithm may be too slow• Large , the algorithm may never converge

– Even if “converged” to a good local minima, the snake is likely to oscillate near it t

xE

2

2

xI

xGx

tt

Alternative solution for 2D snakes:

Dynamic Programming

1

0110 ),(),,(

n

iiiintotal EE

• In most cases, snake energy can be rewritten as a sum of pair-wise interaction potential

More generally, it can be written as a sum of triple-interaction potentials.

1

01110 ),,(),,(

n

iiiiintotal EE

Snake energy: pair-wise interactions

21

0

21010 |),(||),(|),,,,,( iiy

n

iiixnntotal yxGyxGyyxxE

21

1

0

21 )()( ii

n

iii yyxx

Example: simple elastic snake energy

1

0

210 ||)(||),,(

n

iintotal GE

1

0

21 ||||

n

iii

1

0110 ),(),,(

n

iiiintotal EE

21

21 ||||||)(||),( iiiiii GE where

Q: give an example of snake with triple-interaction potentials?

DP Snakes

1v2v

3v

4v6v

5v

control points

Energy E is minimized via Dynamic Programming

[Amini, Weymouth, Jain, 1990]

),(...),(),(),...,,( 1132221121 nnnn vvEvvEvvEvvvE First-order interactions

DP Snakes

1v2v

3v

4v6v

5v

control points

[Amini, Weymouth, Jain, 1990]

Iterate until optimal position for each point is the center of the box,

i.e. the snake is optimal in the local search space constrained by boxes

Energy E is minimized via Dynamic Programming

),(...),(),(),...,,( 1132221121 nnnn vvEvvEvvEvvvE First-order interactions

),( 44 nvvE),( 433 vvE

)3(3E

)4(3E )4(4E

)3(4E

)2(4E

)1(4E

)4(nE

)3(nE

)2(nE

)1(nE

)2(3E

)1(3E

)4(2E

)3(2E

Dynamic Programming (DP)Viterbi Algorithm

),(...),(),( 11322211 nnn vvEvvEvvE

),( 322 vvE

)1(2E

)2(2E

),( 211 vvE

)( 2nmOComplexity:

0)1(1 E

0)2(1 E

0)3(1 E

0)4(1 E

Here we will concentrate on first-order interactions

states

1

2

…

m

site

s

1v 2v 3v 4v nv

This is iterative, as search nbd changeseach time.

Dynamic Programming for a closed snake?

),(...),(),( 11322211 nnn vvEvvEvvE Clearly, DP can be applied to optimize an open ended snake

Can we use DP for a “looped” energy in case of a closed snake?

1n

),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn

1n

2

1n

3 4

Dynamic Programming for a closed snake

),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn

Unfortunately, DP can not be directly applied in case of a “loop”.However, some (approximation) tricks are often used in practice…

1. Use DP to optimize snake energy with fixed (according to a given initial snake position).

2. Use DP to optimize snake energy again. This time fix position of an intermediate node where is an optimal position obtained in step 1.

1

2/2/ ˆnn ̂

Problems with snakes• Depends on number and spacing of control

points

• Snake may oversmooth the boundary

• Not trivial to prevent curve self intersecting

• Can not follow topological changes of objects

(having a model helps!)

Problems with snakes• May be sensitive to initialization

• may get stuck in a local energy minimum near initial contour

• Numerical stability can be an issue for gradient descent and variational methods (continuous formulation)– E.g. requires computing second order derivatives

• The general concept of snakes (deformable models) does generalize to 3D (deformable mesh), but many robust optimization methods suitable for 2D snakes do not apply in 3D– E.g.: dynamic programming only works for 2D snakes

Problems with snakes• External energy: may need to diffuse image gradients,

otherwise the snake does not really “see” object boundaries in the image unless it gets very close to it.

image gradientsare large only directly on the boundary

I

Diffusing Image GradientsIimage gradients diffused via

Gradient Vector Flow (GVF)

Chenyang Xu and Jerry Prince, 98

http://iacl.ece.jhu.edu/projects/gvf/

Alternative Way to Improve External Energy

• Use instead of where D() is– Distance Transform (for detected binary image features, e.g.

edges)

1

0

)(n

iiex vDE

1

0

)(||n

iiex vIE

binary image features(edges)

Distance Transform ),( yxD

Distance Transform can be visualized as a gray-scale image

• Generalized Distance Transform (directly for image gradients)

Distance Transform

34

23

23

5 4 4

223

112

2 1 1 2 11 0 0 1 2 1

0001

2321011 0 1 2 3 3 2

101110 1

2

1 0 1 2 3 4 3 210122

Distance Transform Image features (2D)

Distance Transform is a function that for each image pixel p assigns a non-negative number corresponding to

distance from p to the nearest feature in the image I

)(D)( pD

Some more Variations..

Internal Energy

• Controls the contour smoothness and continuity

dvvvvvv

vvvvvE ii

iiii

iiiiiint

11

11

111 )1()(

External Energy

• Limitations of the constraint of intensity homogeneity – Cannot be applied to open contour– Does not work for band-shaped object

Ultrasound image

Key-chain ring image

Band Energy

• For the contour element , we can define its tangent as:

• Two regions and can be defined for contour element

iv

iviR iR'

Band Energy

• Normalized mean intensity difference

• Band energy (given ring example is opposite)

• Combination of intensity and gradient:

Penalty

Performance

Without band energy

With band energy

Image Initialization Without band energy With band energy

3D reconstruction Experiments

Input

Results