Stereo & Iterative Graph-Cuts

Stereo &

Iterative Graph-Cuts

Alex Rav-AchaVision Course Hebrew University

The stereo problem

The stereo problem

•Both images are very similar (like images that you see with your two eyes)

• Most of the pixels in the left image are present in the right image (except for few occlusions)

After rectification: all correspondences are along the same horizontal scan lines

The stereo problem

(pixels in one image simply shift horizontally in the other image)

The stereo problem

•The horizontal shifts between the images are sometimes called: “disparities”

•The Disparities relates to depth: Closer objects have larger disparities.

The stereo problem: compute the disparity map between two images

Traditional Approaches• Matching rigid windows around each pixel• Each window is matched independently Modern approaches• Finding coherent correspondences for all pixels

- “Graph cuts”- “Belief Propagation”

Window-Based Approach

•Compute a cost for each location

•Location with the lowest cost wins

General Problem : Ambiguity

Left Right

scanline

Window-Based Approach

Small Window Large Window

noisy in low texture areas blurred boundaries

Results with best window size(still not good enough)

Window-based matching(best window size)

Ground truth

Graph Cuts

Ground truthGraph cuts

Maximum flow problem

• Max flow problem:– Each edge is a “pipe”– Find the largest flow F

of “water” that can be sent from the “source” to the “sink” along the pipes

– Edge weights give the pipe’s capacity

“source”

A graph with two terminals

S T

“sink”

a flow F

Minimum cut problem

• Min cut problem:– Find the cheapest way

to cut the edges so that the “source” is completely separated from the “sink”

– Edge weights now represent cutting “costs”

a cut C

“source”

A graph with two terminals

S T

“sink”

Max flow/Min cut theorem

• Max Flow = Min Cut:– Maximum flow saturates the

edges along the minimum cut.

– Ford and Fulkerson, 1962– Problem reduction!

• Ford and Fulkerson gave first polynomial time algorithm for globally optimal solution

“source”

A graph with two terminals

S T

“sink”

Min-Cut: Important Rule

No subset of the cut can also be a cut

This is not a minimal cut

Energy Minimization Using Iterative Graph cuts

Fast Approximate Energy Minimization via Graph Cuts

Yuri Boykov, Olga Veksler and Ramin Zabih

Pami 2001

To do better we need a better model of images

• We can make reasonable assumptions about the surfaces in the world

• Usually assume that the surfaces are smooth• Can pose the problem of finding the

corresponding points as an energy (or cost) minimization:

how well the pixels match up for different

disparities

neighboring pixels have similar disparities

)()()( fEfEfE smoothdata f-assignment

),(

),( ),()()(qp

qpqp

ppp ffVfDfE

To do better we need a better model of images

• We can make reasonable assumptions about the surfaces in the world

• Usually assume that the surfaces are smooth• Can pose the problem of finding the

corresponding points as an energy (or cost) minimization:

Data term is calculated for

each pixels

Smoothness is calculated on

neighbor pixels

f-assignment

p,q-pixels

Example for Smoothness terms

• Quadratic

• L1

• Truncated L1

• Potts model )(),( qpqp ffKffv

),min( Kff qp

qp ff

2)( qp ff

Constructing a Graph to Solve the Stereo Problem

Constructing a Graph to Solve the Stereo Problem

Constructing a Graph to Solve the Stereo Problem

The labels of each pixel are the possible disparity values

The labels of each pixel are the possible disparity values

Constructing a Graph to Solve the Stereo Problem

Relation between the Energy and the Graph labeling problem

),(

),( ),()()(qp

qpqp

ppp ffVfDfE

Data term Smoothness term

p q1

10

{fp=10}

{fq=2}

Relation between the Energy and the Graph labeling problem

),(

),( ),()()(qp

qpqp

ppp ffVfDfE

Data term Smoothness term

p q1

10

Dp(10)

V(p,q)(1, 10)

Iterative graph-cuts

• Use an iterative scheme to find a “good” local optimum of the energy function.

• In each iteration: convert the original multi-label problem to a binary one, and solve it by finding a minimal graph-cut (max-flow).

• The most popular scheme is the expansion move.

• -expansion: set the label of each pixel to be either or the current label.

Types of Moves

Problem: A lot of local minima

A Single Pixel Move

Types of Moves

Any pixel can change its label to alpha

Expansion Move

Types of Moves

Claim (without proof): The difference between the optimal solution and the solution from the iterative expansion moves is bounded

Expansion Move

Energy Minimization Algorithm

1. Start with arbitrary labeling f

2. Set success = 0

3. For each label – Find– If set and success =1

4. If success =1 goto (2)

5. Return f

expansion- one within among )(minargˆ ffEf L

)()( fEfE ff :

Conditions on the Smoothness for using expansion moves:

In other words: V should be a metric

Note : The Quadratic smoothness is not a metric

For each pair of vertices such that we add a ‘dummy’ vertex (together with the respective edges as shown in the table).

Nqp },{Nqp },{

The Relation between the cut and the Energy

• Given a cut C, we define a labelling fc by:

• The cost of a cut C is |C| = E(fC) (plus a constant)

oldp

p ff

If the cut C separates p and If the cut C separates p and

The Relation between the cut and the Energy

The Relation between the cut and the Energy

Conditions on the Smoothness for using expansion moves:

In other words: V should be a metric