Graph Cuts vs. Level Sets - University of Western Ontario ·  · 2006-05-29Graph Cuts vs. Level Sets part IV ... • Non-convex formulation? – Graph cuts (e.g. geo-cuts) • Same

ECCV 2006 tutorial on

Graph Cuts vs. Level Sets

part IV

Global vs. local optimisation algorithms

Vladimir Kolmogorov

University College London

Yuri Boykov

University of Western Ontario

Daniel Cremers

University of Bonn

Global vs. local minima

(Geodesic active contours)

• Geodesic active contours

– Variational approach (e.g. level sets)

• Gradient descent in the space of contours

• Local minimum

• Non-convex formulation?

– Graph cuts (e.g. geo-cuts)

• Same problem, global minimum

• Convex formulation?

[Anonymous attendee of CVPR’05]:

How is it possible?

Global vs. local minima

(Geodesic active contours)

• Geodesic active contours

– Variational approach (e.g. level sets)

• Gradient descent in the space of contours

• Local minimum

• Non-convex formulation?

– Graph cuts (e.g. geo-cuts)

• Same problem, global minimum

• Convex formulation?

[Anonymous attendee of CVPR’05]:

How is it possible?

Graph cuts

• Function E(x) of discrete variables: convexity not defined

• Extend the space of solutions: xp∈{0,1} ⇒ xp∈[0,1]

– Allow fractional segmentations

• Extend energy: linear programming (LP) relaxation

– Now convex problem!

Energy function

with discrete variables

E

submodular function ⇒ integer solution

LP relaxation

E tight

Graph cuts

• Function E(x) of discrete variables: convexity not defined

• Extend the space of solutions: xp∈{0,1} ⇒ xp∈[0,1]

– Allow fractional segmentations

• Extend energy: linear programming (LP) relaxation

– Now convex problem!

Energy function

with discrete variables

E Enot tight

non-submodular function ⇒ fractional solution (in general)

LP relaxation

Solving LP relaxation

• Too large for general purpose LP solvers (e.g. interior point methods)

• Solve dual problem instead of primal:

– Formulate lower bound on the energy

– Maximize this bound

– When done, solves primal problem (LP relaxation)

• Two different ways to formulate lower bound

– Part A: Via posiforms => maxflow algorithm (for binary variables)

– Part B: Via convex combination of trees => tree-reweighted message passing

Lower bound on

the energy function

E

Energy function

with discrete variables

E E

LP relaxation

Notation and Preliminaries

Energy function

∑∑ ++=qp

qppq

p

ppconst xxxE,

),()()|( θθθθx

p

qunary terms

(data)pairwise terms

(coherence)

- xp are discrete variables (for example, xp∈{0,1})

- θp(•) are unary potentials

- θpq(•,•) are pairwise potentials

LP relaxation

• [Schlesinger’76,Koster et al.’98,Chekuri et al.’00,Wainwright et al.’03]

• Introduce indicator variables xp;i , xpq;ij

=

=

→+

∑

∑

∑∑

i

ip

j

ipijpq

jiqp

ijpqpq

ip

ipp

x

xx

xjixi

1

min),()(

;

;;

,,,

;

,

; θθ

xpq;ij∈{0,1}relaxation

xpq;ij∈[0,1]

Energy function - visualisation

0

4

0

1

3

02

5

node p edge (p,q) node q

label 0

label 1

)0(pθ

)1,0(pqθ

0

constθ

∑∑ ++=qp

qppq

p

ppconst xxxE,

),()()|( θθθθx

0

4

0

1

3

02

5

node p edge (p,q) node q

label 0

label 1

Energy function - visualisation

∑∑ ++=qp

qppq

p

ppconst xxxE,

),()()|( θθθθx

0

vector of

all parameters=θ

0 0 4

4

1 12

5

0

-1

-1

0 + 1

Reparameterisation

Reparameterisation

0 0 3

4

1 02

5

• Definition. is a reparameterisation of

if they define the same energy:

xxx any for )|()|( θθ EE =′

( )θθ ≡′θ ′ θ

4 -1

1 -1 0 +1

• Maxflow, BP and TRW perform reparameterisations

1

Part A: Lower bound

via posiforms

(⇒⇒⇒⇒ maxflow algorithm)

non-negative

constθ - lower bound on the energy:

xx ∀≥ constE θθ )|(

maximize

Lower bound via posiforms[Hammer, Hansen, Simeone’84]

∑∑ ++=qp

qppq

p

ppconst xxxE,

),()()|( θθθθx

• Posiform maximisation: algorithm?

• Binary variables, submodular functions

– Reduction to maxflow

– Global minimum of the energy

• Binary variables, non-submodular functions

– Reduction to maxflow

• More complicated graph

– Part of optimal solution

Outline of part A

Posiform maximisation

Binary variables,

submodular functions

• Definition: E is submodular if every pairwise term satisfies

• Can be converted to “canonical form”:

Submodularity and canonical form

)0,1()1,0()1,1()0,0( pqpqpqpq θθθθ +≤+

2

1 2 3 4

10

0 05

zero

cost

Overview of min cut/max flow

Min Cut problem

source

sink

2 1

1

2

3

4

5

Directed weighted graph

Min Cut problem

sink

2 1

1

2

3

4

5

S = {source, node 1}

T = {sink, node 2, node 3}

Cut:

source

Min Cut problem

sink

2 1

1

2

3

4

5

S = {source, node 1}

T = {sink, node 2, node 3}

Cut:

• Task: Compute cut with minimum cost

Cost(S,T) = 1 + 1 = 2

source

sink

2 1

1

2

3

4

5

source

Maxflow algorithm

value(flow)=0

Maxflow algorithm

sink

2 1

1

2

3

4

5

value(flow)=0

source

Maxflow algorithm

sink

1 1

1

2

3

4

5

value(flow)=0

source

Maxflow algorithm

sink

1 1

0

2

3

4

5

value(flow)=0

source

Maxflow algorithm

sink

1 1

0

2

3

4

4

value(flow)=1

source

Maxflow algorithm

sink

1 1

0

3

3

4

4

value(flow)=1

source

Maxflow algorithm

sink

1 1

0

3

3

4

4

value(flow)=1

source

Maxflow algorithm

sink

1 1

0

3

3

4

4

value(flow)=1

source

Maxflow algorithm

sink

1 0

0

3

4

3

3

value(flow)=2

source

Maxflow algorithm

sink

1 0

0

3

4

3

3

value(flow)=2

source

value(flow)=2

sink

1 0

0

3

4

3

3

source

Maxflow algorithm

Posiform maximisation

Binary variables,

non-submodular functions

Reduction to maxflow

2

1 2 3 4

10

0 05

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

2

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

2

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

2

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

2

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

2

3

4

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2

1

2

3

4

5

source

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2 1

1

2

3

4

5

source

0

Maxflow algorithm

and reparameterisation

2

1 2 3 4

10

0 05

sink

2 1

1

2

3

4

5

source

value(flow)=0

0

Maxflow algorithm

and reparameterisation

sink

2 1

1

2

3

4

5

value(flow)=0

2

1 2 3 4

10

0 05

0

source

Maxflow algorithm

and reparameterisation

sink

1 1

0

3

3

4

4

value(flow)=1

1

0 3 3 4

10

0 04

1

source

Maxflow algorithm

and reparameterisation

sink

1 1

0

3

3

4

4

value(flow)=1

1

0 3 3 4

10

0 04

1

source

Maxflow algorithm

and reparameterisation

sink

1 0

0

3

4

3

3

value(flow)=2

1

0 3 4 3

00

0 03

2

source

Maxflow algorithm

and reparameterisation

sink

1 0

0

3

4

3

3

value(flow)=2

1

0 3 4 3

00

0 03

2

source

Maxflow algorithm

and reparameterisation

value(flow)=2

0

00

0

)1,1,0(=x

minimum of the energy:

2

0

sink

1 0

0

3

4

3

3

source

Maxflow algorithm

and reparameterisation

Posiform maximisation

Binary variables,

non-submodular functions

Arbitrary functions of binary variables

• Can be solved via maxflow[Hammer,Hansen,Simeone’84][Boros,Hammer,Sun’91]

– Specially constructed graph

• Gives solution to LP relaxation: for each node

xp∈{0, 1/2, 1}

E

LP relaxation

non-negativemaximize

∑∑ ++=qp

qppq

p

ppconst xxxE,

),()()|( θθθθx

Arbitrary functions of binary variables

0

1

0

1

1 1/2 1/2

1/2

1/2

Part of optimal solution

[Hammer, Hansen, Simeone’84]

Graph construction - Main idea

∑

∑

∑

+

+

=

),(~

),(

)(})({

qppq

qppq

ppp

xxE

xxE

xExEunary

pairwise submodular

pairwise non-submodular

• Double # of variables:

– Ideally,

• Write E as a function of both old and new variables

– New function is submodular!

ppp xxx ,→

pp xx −=1

∑

∑

∑

−+−+

−−++

−+=

2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({

qppqppq

qpppppq

pppp

pp

xxExxE

xxExxE

xExExxE

Graph construction - Main idea

∑

∑

∑

+

+

=

),(~

),(

)(})({

qppq

qppq

ppp

xxE

xxE

xExE

• Double # of variables:

– Ideally,

• Write E as a function of both old and new variables

– New function is submodular!

ppp xxx ,→

pp xx −=1

∑

∑

∑

−+−+

−−++

−+=

2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({

qppqppq

qpppppq

pppp

pp

xxExxE

xxExxE

xExExxE

Graph construction - Main idea

∑

∑

∑

+

+

=

),(~

),(

)(})({

qppq

qppq

ppp

xxE

xxE

xExE

• Double # of variables:

– Ideally,

• Write E as a function of both old and new variables

– New function is submodular!

ppp xxx ,→

pp xx −=1

∑

∑

∑

−+−+

−−++

−+=

2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({

qppqppq

qpppppq

pppp

pp

xxExxE

xxExxE

xExExxE

Graph construction - Main idea

∑

∑

∑

+

+

=

),(~

),(

)(})({

qppq

qppq

ppp

xxE

xxE

xExE

• Double # of variables:

– Ideally,

• Write E as a function of both old and new variables

– New function is submodular!

ppp xxx ,→

pp xx −=1

∑

∑

∑

−+−+

−−++

−+=

2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({

qppqppq

qpppppq

pppp

pp

xxExxE

xxExxE

xExExxE

Graph construction - Main idea

∑

∑

∑

+

+

=

),(~

),(

)(})({

qppq

qppq

ppp

xxE

xxE

xExE

• Double # of variables:

– Ideally,

• Write E as a function of both old and new variables

– New function is submodular!

ppp xxx ,→

pp xx −=1

∑

∑

∑

−+−+

−−++

−+=

2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({

qppqppq

qpppppq

pppp

pp

xxExxE

xxExxE

xExExxE

Graph construction - Main idea

∑

∑

∑

+

+

=

),(~

),(

)(})({

qppq

qppq

ppp

xxE

xxE

xExE

• Minimise new function

– Without constraint

}){},({ pp xxE

pp xx −=1

Graph construction

2 00

0 10

sink

source

px

px

Graph construction

2 00

0 10

sink

source

Graph construction

00

0 10

sink

source

2

Graph construction

2 00

0 10

sink

source

px

px

Assigning labels

0

1

=

=

p

p

x

x

• Assign labels based on minimum cut in auxiliary graph

node p is unlabeled

• To maximize # of labeled nodes, choose a particular minimum cut

1

0

=

=

p

p

x

x

Optimality

0 0

0

0

0

0=px

Theorem [Hammer,Hansen,Simeone’84].

Labeling x is part of optimal labeling x*.

1=px ?=px

labeled part unlabeled part

Part B: Lower bound via

convex combination of trees

(⇒⇒⇒⇒ tree-reweighted message passing)

• Goal: compute minimum of the energy for θ :

• In general, intractable!

• Obtaining lower bound:

– Split θ into several components: θ = θ1 + θ2 + ...

– Compute minimum for each component:

– Combine Φ(θ1), Φ(θ2), ... to get a bound on Φ(θ)

• Use trees!

)|(min)( θθ xx

E=Φ

)|(min)( iiE θθ x

x=Φ

Convex combination of trees

[Wainwright, Jaakkola, Willsky ’02]

θ ≡ '

2

1 TθTθ2

1+

graph tree T tree T’

)(θΦ )(2

1 TθΦ )(2

1 'TθΦ+≥

lower bound on the energymaximize

Convex combination of trees

(cont’d)

TRW algorithms

• Goal: find reparameterisation maximizing lower bound

• Apply sequence of different reparameterisation operations:– Node averaging

– Ordinary BP on trees

• Order of operations?– Affects performance dramatically

• Algorithms:– [Wainwright et al. ’02]: parallel schedule

• May not converge

– [Kolmogorov’05]: specific sequential schedule• Lower bound does not decrease, convergence guarantees

• Needs half the memory

Experimental results: stereo

left image ground truth

BP TRW-S20 40 60 80 100

3.6

3.8

4x 10

5

• Global minima for some instances with TRW [Meltzer,Yanover,Weiss’05]

Parts A and B: Summary

• MAP estimation algorithms are based on LP relaxation– Maximize lower bound

• Two ways to formulate lower bound

• Via posiforms: leads to maxflow algorithm (for binary variables)– Polynomial time solution

– Submodular functions: global minimum

– Non-submodular functions: part of optimal solution

• Via convex combination of trees: leads to TRW algorithm– Convergence in the limit (for TRW-S)

– Applicable to arbitrary energy function

Non-binary variables:

Other methods for solving LP

• No polynomial-time algorithm (except general purpose LP solvers)

• Iterative methods:– [Koval,Schlesinger’76]: augmenting DAG algorithm

– [Kovalevsky,Koval’75, Flach’98] (unpublished): max-sum diffusion• See tech. report [Werner’05]

• Not guaranteed to solve LP (only arc constistent solution) – same as TRW

• Special case: submodular functions– LP has integer optimal solution [Schlesinger,Flach’00]

– Reduction to maxflow [Ishikawa’03, D.Schlesinger’05]

EE E

Continuous mincut/maxflow

Continuous mincut/maxflow

• Primal problem:

min ))(( →∫C

dssCg

min|| →∇∫ gu

Alternatively:

subject to

∈=

∈=

tpu

spu

p

p

,1

,0

total variation

[Rudin,Osher,Fatemi’92] :

image restoration

[Amar,Belletini’94]:

definition for arbitrary metric

subject to

Ct

Cs

outside

inside

up = 0

up = 1

Continuous mincut/maxflow

• Primal problem:

min ))(( →∫C

dssCg

min|| →∇∫ gu

Alternatively:

subject to

∈=

∈=

tpu

spu

p

p

,1

,0

subject to

Ct

Cs

outside

inside

• up∈[0,1] – fractional segmentations

• Convex problem

• Integer optimal solution

up = 0

up = 1

Continuous mincut/maxflow

• Dual problem:

(capacity constraint)

(flow conservation)

gf p ≤||r

0div =pfr

subject to

for p ∉ s,t

max )(div →∫s

p dafr

Reparameterisation

• Any flow with

defines reparameterisation

(by the divergence theorem):

tspf p ,for 0div ∉=r

∫∫ ⋅−+=≡CC

dsNfgconstdsgCE )( )(rr

∫=s

dsfconst )(divr

where

Reparameterisation

∫∫ ⋅−+=≡CC

dsNfgconstdsgCE )( )(rr

distance

maps

⇒ non-negativegf ≤||r

lower bound on E(C)

Reparameterisation

∫∫ ⋅−+=≡CC

dsNfgconstdsgCE )( )(rr

Suppose flow saturates cut C*

( for p ∈ C* ):

⇒ C* = minimum cut

Ngf pp

rr=

zero for C*

Global vs. local optimisation algorithms:

Summary

• Geodesic active contours

– Variational approach (e.g. level sets)

• Gradient descent in the space of contours

• Local minimum

• Non-convex formulation

– Graph cuts (e.g. geo-cuts)

• Extended space (fractional segmentations)

• Convex formulation

• Integer solution (for submodular functions)

– `

Global vs. local optimisation algorithms:

Summary

• Geodesic active contours

– Variational approach (e.g. level sets)

• Gradient descent in the space of contours

• Local minimum

• Non-convex formulation

– Graph cuts (e.g. geo-cuts)

• Extended space (fractional segmentations)

• Convex formulation

• Integer solution (for submodular functions)

– `

Other relaxations/extensions

• Energy E(x) defined for integer configurations (xp∈ {0,1})

• How to define for fractional configurations (xp∈ [0,1])?

Other relaxations/extensions

• LP relaxation [Schlesinger’76,Koster et al.’98,Chekuri et al.’00, Wainwright et al.’03]

– Defined for multi-valued variables

– Convex

– E is submodular ⇒ integer solution

• Lovász extension [Lovász’83]

– Defined for binary variables

– Always integer solution

– E is submodular ⇔ extension is convex

– “Submodularity” – discrete analogue of convexity

• Sherali-Adams relaxation, semi-definite relaxation, SOCP relaxation, ...

(0,0)(1,0)

(1,1)

LP relaxation and Lovász extension

Submodular function: E(0,0) + E(1,1) ≤ E(0,1) + E(1,0)

LP relaxation and Lovász extension

(0,0)(1,0)

(1,1)(1,0)

(0,0)(1,0)

(1,1)(1,0)

LP Lovász

Non-submodular function: E(0,0) + E(1,1) ≥ E(0,1) + E(1,0)