S I E M E N S C O R P O R A T E R E S E A R C H 1 1 General Purpose Image Segmentation with Random Walks Leo Grady Department of Imaging and Visualization.

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

1

General Purpose Image Segmentation with Random Walks

Leo Grady Department of Imaging and Visualization

Siemens Corporate Research

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

2

Outline

• Overview of Siemens Corporate Research (SCR)

• General purpose segmentation

• Random walker algorithm– Concept– Properties– Theory– Numerics– Results– New

• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

3

• About 200 full time research staff• 75+ people working on medical imaging• Basic research clinical products• 1/3 mid/long term research - 2/3 applied projects

SCR

Siemens Medical Solutions

Clinical & UniversityPartners

Princeton, USA

Overview of SCR

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

4

Overview of SCR

Clinical Imaging

• Goals of clinical application software

• Measures something that could not be measured practically before

• Makes diagnosis more accurate or treatment more effective

• Enables therapy that was not possible before

• Increases patient control

• Saves time

• Reduces cost

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

5

Overview of SCRCore interests – Segmentation, registration, visualization

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

6

Offline & Online: Intervention• So far: diagnostic radiology: offline problem

• Interventional imaging: online problem

• Continuous imaging, constant human input

• Rich source of new problems

Overview of SCR

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

7

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

8

General Purpose Segmentation

Goal: Input an image and output the desired segmentation

Why?• Image editing, etc. (e.g., Magic Wand)

• Do not want to reinvent segmentation for each new product

Problem: Two users might want different objects from same image

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

9

General Purpose SegmentationRequires user interaction

NCuts, watershed, mean shift

Snakes, level sets,intelligent scissors

Graph cuts, magic wand region growing

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

10

General Purpose SegmentationAtomic methods:

• Semi-automatic

• Atoms must be subsets of true segmentation

Boundary methods:• Easier to incorporate shape prior• Can be used to improve segmentation of another algorithm• Iterative – local minima• Difficult to initialize automatically and in higher dimension• Harder to generalize to point sets, surfaces, nonuniform sampling, etc.

Seeding methods:• Leads naturally to steady-state and graph-based algorithms

• Easy to seed (even automatically) in arbitrary dimensions

• Generalizes easily to other data modalities

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

11


Popular seeding algorithmsRegion growing: Grow segment from initial seed until distance/contrast/etc. requirement is met

• Simple• Fast• Leaks through weak boundaries• Killed by noise

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

12


Popular seeding algorithmsGraph cuts: Max-flow/min-cut found between seeds

• Fast• Probabilistic interpretation• Requires lots of seeds to avoid “small cut” problem• Metrication artifacts• True minimum only for two objects (i.e., foreground/background)

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

13

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

14

Given labeled voxels, for each voxel ask: What is the probability that a random walker starting from this voxel

first reaches each set of labels?

Random Walker - Concept

Do not despair – Can be computed analytically!

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

15

Green Red Yellow Blue

Partially labeled image Segmented image

Probabilities


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

16


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

17

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

18

Random Walker - Properties

Naturally respects weak object boundaries

Solid border Weak border

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

19

Naturally respects weak object boundariesRandom Walker - Properties

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

20

Provably robust to identically distributed noise

No texture or filtering used – Based purely on intensity weighting


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

21

1. Segmented regions are connected to a seed

2. The probabilities for a blank image (e.g., all black) yield a Voronoi-like segmentation

3. The expected segmentation for an image of pure noise (identical r.v.s) is equal to the Voronoi-like segmentation obtained from a blank image


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

22

Random walker

Graph cuts


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

23


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

24

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

25

How to compute?

Solution to random walk problem equivalent to minimization of the Dirichlet integral

with appropriate boundary conditions.

The solution is given by a harmonic function, i.e., a functionsatisfying

D[u]=12

Z

(gr u(x;y))2

r ¢gr u = 0

Random Walker - Theory

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

26

Discrete or continuous space?Discrete• Finite

• Exact

• Generalizable

Continuous• Euclidean

• Approximate

• Convergence, etc.

4-connected 8-connected


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

27

• Mean value theorem:

• Maximum/Minimum principle:

Attractive numerical properties of a harmonic function

xi =

Pwi j (xi ¡ xj )P

wi j

min(xBoundary) · xInterior · max(xBoundary)


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

28

Need to represent Laplacian on a graph:

In the notation of algebraic topology, the Laplacian is given by

0-coboundary operator (since we operate on nodes) is theincidence matrix:

With the constituitive matrix Ceij eij=wij playing the role of the

metric tensor, the combinatorial Laplace-Beltrami operatoris given as


r 2u = @±

Aei j vk =

8><

>:

+1 if i = k;¡ 1 if j = k;0 otherwise

L = AT CA

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

29

D[u]=12

Z

(gr u(x;y))2

Subject to boundary conditions at seed locations

xF = 1; xB = 0

Energy functional:

r ¢gr u = 0 Lx = 0Euler-Lagrange:

D[x]=12

¡xT AT

¢C (Ax) =

12xT Lx


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

30


Laplacian matrix defined by graph as:

Decompose Laplacian matrix into labeled (marked) and unlabeled blocksand define an indicator vector for the marked nodes:

Must solve a sparse, SPD, system of linear equations for probabilities

Since probabilities must sum to unity, for K labels, only K-1 systems must be solved

L vi vj =

8><

>:

dviif i = j ;

¡ wi j if vi andvj areadjacent nodes;0 otherwise

L =

·LM BB T LU

¸ms

j =

(1 if Q(vj ) = s;0 if Q(vj ) 6= s:

LUxs = ¡ Bms

xK = 1¡X

i<K

xi

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

31

Input image Overlaid graph (lattice) Edge strength (line width) encodes image gradient

Random walk formulated on a lattice (graph) that represents the image


wi j = e¡ ¯ (I i ¡ I j )2

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

32

Therefore, we can formulate a combinatorial Dirichlet integral:

Represents minimum power distribution of an electrical circuit

We can analytically solve the equivalent circuit problem for the random walker probabilities


D[x]=12

¡xT AT

¢C (Ax) =

12xT Lx

AT z = f (Kirchho®'s Current Law);

Cp = z (Ohm's Law);

p = Ax (Kirchho®'s VoltageLaw);

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

33

Situation exactly analogous to DC circuit steady-state

Labels – Unit voltage sources or groundsWeights – Branch conductancesProbabilities – Steady-state potentials

Label 1 prob.

Initial labeling

Label 2 prob. Label 3 prob.


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

34

Algorithm summary:

1. Generate weights based on image intensities

2. Build Laplacian matrix

3. Solve system of equations for each label

4. Assign pixel (voxel) to label for which it has the highest probability


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

35

Equally valid interpretations of algorithm:1. What is the steady-state temperature distribution in the

inhomogeneous domain, given fixed temperatures at the seeds?

2. What is the probability that a random walker leaving this node first reaches a label of each color?

3. What is the electrical potential at this node when the labeled nodes are fixed to unity voltage (w.r.t. ground)?

4. What is the (normalized) effective resistance between this node and the labeled nodes?


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

36

Equally valid interpretations of algorithm:

5. If a 2-tree (tree with a missing edge) is drawn randomly, what is the probability that this node is connected to each label?

Interpretation used to prove noise robustness


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

37

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

38

Main computational burden is solving the system of linear equations

Fortunately, system is sparse, symmetric, positive definite For a lattice (or any regular graph), the sparsity structure of the matrix

is circulant

Random Walker - Numerics

LUxs = ¡ Bms

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

39

• Structure of the Laplacian matrix allows for efficient storage and operations – Off diagonals may be packed into RGBA

• Progressive visualization of solution possible

• Z-buffer allows masking out of seeds

Advantages of a GPU implementation

Random Walker - Numerics

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

40

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

41

Random Walker - Results

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

42


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

43


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

44


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

45

Cardiac segmentation across modalities


S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

46

Segmentation of objects with varying size, shape and textureRandom Walker - Results

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

47

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

48

Original image Priors only

Random walkeronly

Combined

Possible to incorporate other terms – Intensity priors

Useful for multiple, disconnected objects

Random Walker - New

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

49

Random Walker - New

wi j = e¡ ¯ (I i ¡ I j )2Gaussian weighting

Systematic study of weighting function

Reciprocal weighting

wi j =1

1+¯ (I i ¡ I j )2

Run on 62 CT datasets with seeds and manual segmentations

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

50

Random Walker - New

Systematic study of edge topology

6-connected

10-connected

26-connected

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

51

Random Walker - New

Formulate as special case of general segmentation approach -Compare with other instances of algorithm

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

52

Random Walker - New

5 eigs 20 eigs 40 eigs 60 eigs 80 eigs 100 eigs Exact

Precomputation

• Precompute eigenvectors of Laplacian• Input seeds• Instant result (approximation)

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

53

Outline




• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

54

Conclusion

1. General-purpose2. Robust to noise and weak boundaries3. Has a single parameter (not adjusted for these results)4. Stable5. Accurate6. Available

Random walker algorithm is:

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

55

Conclusion – More Information

My webpage:http://cns.bu.edu/~lgrady

Random walkers paper:http://cns.bu.edu/~lgrady/grady2006random.pdf

Random walker demo page:http://cns.bu.edu/~lgrady/Random_Walker_Image_Segmentation.html

CVPR Short Course: Fundamentals linking discrete and continuous approaches to computer vision - A topological view

http://cns.bu.edu/~lgrady/Short_Course.html

MATLAB toolbox for graph theoretic image processing at:http://eslab.bu.edu/software/graphanalysis/

Writings and code

Random walkers MATLAB code:http://cns.bu.edu/~lgrady/random_walker_matlab_code.zip

S I E M E N S C O R P O R A T E R E S E A R C H 1 1 General Purpose Image Segmentation with Random Walks Leo Grady Department of Imaging and Visualization.

Documents