Fast, Multiscale Image Segmentation: From Pixels to Semantics Ronen Basri The Weizmann Institute of Science Joint work with Achi Brandt, Meirav Galun,

Fast, Multiscale Image Fast, Multiscale Image Segmentation: From Pixels to Segmentation: From Pixels to

SemanticsSemantics

Ronen BasriThe Weizmann Institute of Science

Joint work with

Achi Brandt, Meirav Galun, Eitan Sharon

CamouflageCamouflage

CamouflageCamouflage

Malik et al.’s“Normalized cuts”

Our ResultsOur Results

SSegmentation by egmentation by WWeighted eighted AAggregationggregation

A multiscale algorithm:• Optimizes a global measure• Returns a full hierarchy of segments• Linear complexity• Combines multiscale measurements:

– Texture– Boundary integrity

The Pixel GraphThe Pixel GraphCouplings (weights)

reflect intensity

similarity

Low contrast –strong couplingHigh contrast –weak coupling

i jI I

ijW e

Normalized-cut MeasureNormalized-cut Measure

( )( )

( )

Cut SE S

Int S

Minimize:

2( ) ( )ij i ji j

Cut S w u u

( ) ij i ji j

Int S w u u

Si

Siui 0

1

Saliency MeasureSaliency Measure

2( ) ( )

1( )

2

( ) 2

Tij i j

i j

Tij i j

i j

T

T

Cut S w u u u Lu

Int S w u u u Wu

u LuE S

u Wu

Lu WuMinimize:

Multiscale Computation Multiscale Computation of Ncutsof Ncuts

• Our objective is to rapidly find the segments (0-1 partitions) that optimize

• For single-node cuts we simply evaluate • For multiple-node cuts we perform “soft

contraction” using coarsening procedures from algebraic multigrid solvers of PDEs.

Coarsening the GraphCoarsening the Graph

• Suppose we can define a sparse mappingsuch that for all minimal states

: , ( / 2)N nP N n R R

11

22

.

..

Nn

Uu

P

uU

uU

Coarse EnergyCoarse Energy

• Then

• PTWP, PTLP define a new (smaller)

graph

( ) 2 2T T T

T T T

u Lu U P LPUE S

u Wu U P WPU

Recursive CoarseningRecursive Coarsening

iu ju

Recursive CoarseningRecursive Coarsening

lUkURepresentative subset

1 2( , ,..., )NU U U U

Recursive CoarseningRecursive Coarsening

11

22

.

..

Nn

Uu

P

uU

uU

For a salient segment :

( )n NP ,sparse interpolation matrix

iu julUkU

Weighted AggregationWeighted Aggregation

ijwi

jjlp

aggregate k aggregate l

[[ 1] ]s T sWW P P

klWikp

HierarchicHierarchical Graphal Graph

Pyramid of graphs Soft relations between levels Segments emerge as salient nodes at some level of the pyramid

Importance of Soft Importance of Soft RelationsRelations

Physical MotivationPhysical Motivation

• Our algorithm is motivated by algebraic multigrid solutions to heat or electric networks

• u - temperature/potential• a(x, y) – conductivity• At steady state largest temperature

differences are along the cuts• AMG coarsening is independent of f

( , ) ( , ) ( , )u u

a x y a x y f x yx x y y

Determine the Determine the BoundariesBoundaries

1

001,0,0,…,0

P

HierarchHierarchyy

in in SWASWA

Texture ExamplesTexture Examples

FiltersFilters(From Malik and Perona)

Oriented filtersCenter-

surround

A Chicken and Egg A Chicken and Egg Problem Problem

Problem:Coarse measurements mix neighboring statistics

Solution: Support of measurements is determined as the segmentation process proceeds

Hey, I was here first

Texture AggregationTexture Aggregation

• Aggregates assumed to capture texture elements

• Compare neighboring aggregates according to the following statistics:– Multiscale brightness measures– Multiscale shape measures– Filter responses

• Use statistics to modify couplings

Recursive Computation of Recursive Computation of MeasuresMeasures

• Given some measure of aggregates at a certain level (e.g., orientation)

• At every coarser level we take a weighted sum of this measure from previous level

• The result can be used to compute the average, variance or histogram of the measure

• Complexity is linear

Use Averages to Modify the Use Averages to Modify the GraphGraph

Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements

Averaging

Our algorithm - SWA

Original

Geometric

Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements

Interpolation Geometric

Original

Our algorithm - SWA

Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements

Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements

Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements

Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements

Adaptive vs. Rigid Adaptive vs. Rigid MeasurementsMeasurements

Texture AggregationTexture Aggregation

Fine (homogeneous) Coarse (heterogeneous)

Multiscale Variance Multiscale Variance VectorVector

Multiscale Variance Multiscale Variance VectorVector

Variance: Avoid MixingVariance: Avoid Mixing

aggregation Sliding window

LeopardLeopard

More Leopards…More Leopards…

And More…And More…

BirdsBirds

More AnimalsMore Animals

BoatBoat

Malik’s NcutsMalik’s Ncuts

Key DifferencesKey Differences

• Optimize a global measure(like Malik’s Ncuts)

• Hierarchy with soft relations(unlike agglomerative/graph contraction)

• Combine texture measurements while avoiding the “chicken and egg problem”

ComplexityComplexity

• Every level contains about half the nodes of the previous level:

Total #nodes 2 X #pixels• All connections are local, cleaning small

weights• Top-down sharpening: constant number

of levels• Linear complexity• Implementation: 5 seconds for 400x400

•Average intensity•Texture•Shape

RepresentationRepresentation

MatchingMatching(with Chen Brestel)

More…More…

Data: FilippiData: Filippi

30 slices, 180x220 in 3 minutes

MRI DataMRI Data

MS Lesion DetectionMS Lesion Detection

TaggedTagged Our resultsOur results

Data: FilippiData: Filippi

Data: FilippiData: Filippi

TaggedTagged Our resultsOur results

Data: FilippiData: Filippi

TaggedTagged Our resultsOur results

Data: FilippiData: Filippi

TaggedTagged Our resultsOur results

2D Segmentation2D Segmentation

Data: FilippiData: Filippi

Data: FilippiData: Filippi

3D Segmentation3D Segmentation

Cell MovementCell Movement

SummarySummary

image

segments

Shape propertiesLeopard