Top Banner
OBJ CUT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD
88

O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Mar 28, 2015

Download

Documents

Justin Fletcher
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

OBJ CUT & Pose CutCVPR 05ECCV 06

Philip TorrM. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman

UNIVERSITYOF

OXFORD

Page 2: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Conclusion

• Combining pose inference and segmentation worth investigating. (tommorrow)

• Tracking = Detection

• Detection = Segmentation

• Tracking (pose estimation) = Segmentation.

Page 3: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Segmentation

• To distinguish cow and horse?

First segmentation problem

Page 4: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Aim• Given an image, to segment the object

Segmentation should (ideally) be• shaped like the object e.g. cow-like• obtained efficiently in an unsupervised manner• able to handle self-occlusion

Segmentation

ObjectCategory

Model

Cow Image Segmented Cow

Page 5: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Challenges

Self Occlusion

Intra-Class Shape Variability

Intra-Class Appearance Variability

Page 6: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Cow Image

Object Seed Pixels

Page 7: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Cow Image

Object Seed Pixels

Background Seed Pixels

Page 8: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Segmented Image

Page 9: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Cow Image

Object Seed Pixels

Background Seed Pixels

Page 10: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Segmented Image

Page 11: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Problem • Manually intensive

• Segmentation is not guaranteed to be ‘object-like’

Non Object-like Segmentation

Motivation

Page 12: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Our Method• Combine object detection with segmentation

– Borenstein and Ullman, ECCV ’02– Leibe and Schiele, BMVC ’03

• Incorporate global shape priors in MRF

• Detection provides– Object Localization– Global shape priors

• Automatically segments the object– Note our method completely generic– Applicable to any object category model

Page 13: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Outline

• Problem Formulation

• Form of Shape Prior

• Optimization

• Results

Page 14: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Problem• Labelling m over the set of pixels D• Shape prior provided by parameter Θ

• Energy E (m,Θ) = ∑Φx(D|mx)+Φx(mx|Θ) + ∑ Ψxy(mx,my)+ Φ(D|mx,my)

• Unary terms– Likelihood based on colour– Unary potential based on distance from Θ

• Pairwise terms– Prior– Contrast term

• Find best labelling m* = arg min ∑ wi E (m,Θi)– wi is the weight for sample Θi

Unary terms Pairwise terms

Page 15: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

MRF

Probability for a labelling consists of• Likelihood

• Unary potential based on colour of pixel• Prior which favours same labels for neighbours (pairwise potentials)

Prior Ψxy(mx,my)

Unary Potential Φx(D|mx)

D (pixels)

m (labels)

Image Plane

x

y

mx

my

Page 16: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior

x …

y …

x …

y …

Φx(D|obj)

Φx(D|bkg)Ψxy(mx,my)

Likelihood Ratio (Colour)

Page 17: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Example

Cow Image Object SeedPixels

Background SeedPixels

PriorLikelihood Ratio (Colour)

Page 18: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Contrast-Dependent MRF

Probability of labelling in addition has• Contrast term which favours boundaries to lie on image edges

D (pixels)

m (labels)

Image Plane

Contrast Term Φ(D|mx,my)

x

y

mx

my

Page 19: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + Contrast

x …

y …

x …

y …

Likelihood Ratio (Colour)

Ψxy(mx,my)+Φ(D|mx,my)

Φx(D|obj)

Φx(D|bkg)

Page 20: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastLikelihood Ratio (Colour)

Page 21: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Our Model

Probability of labelling in addition has• Unary potential which depend on distance from Θ (shape parameter)

D (pixels)

m (labels)

Θ (shape parameter)

Image Plane

Object CategorySpecific MRFx

y

mx

my

Unary PotentialΦx(mx|Θ)

Page 22: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastDistance from Θ

Shape Prior Θ

Page 23: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastLikelihood + Distance from Θ

Shape Prior Θ

Page 24: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastLikelihood + Distance from Θ

Shape Prior Θ

Page 25: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Outline

• Problem Formulation– E (m,Θ) = ∑Φx(D|mx)+Φx(mx|Θ) + ∑ Ψxy(mx,my)+ Φ(D|mx,my)

• Form of Shape Prior

• Optimization

• Results

Page 26: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Detection

• BMVC 2004

Page 27: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Layered Pictorial Structures (LPS)• Generative model

• Composition of parts + spatial layout

Layer 2

Layer 1

Parts in Layer 2 can occlude parts in Layer 1

Spatial Layout(Pairwise Configuration)

Page 28: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Layer 2

Layer 1

Transformations

Θ1

P(Θ1) = 0.9

Cow Instance

Layered Pictorial Structures (LPS)

Page 29: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Layer 2

Layer 1

Transformations

Θ2

P(Θ2) = 0.8

Cow Instance

Layered Pictorial Structures (LPS)

Page 30: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Layer 2

Layer 1

Transformations

Θ3

P(Θ3) = 0.01

Unlikely Instance

Layered Pictorial Structures (LPS)

Page 31: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

How to learn LPS

• From video via motion segmentation see Kumar Torr and Zisserman ICCV 2005.

Page 32: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

LPS for Detection• Learning

– Learnt automatically using a set of examples

• Detection– Matches LPS to image using Loopy Belief Propagation– Localizes object parts

Page 33: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Detection

• Like a proposal process.

Page 34: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Pictorial Structures (PS)

PS = 2D Parts + Configuration

Fischler and Eschlager. 1973

Aim: Learn pictorial structures in an unsupervised manner

• Identify parts• Learn configuration• Learn relative depth of parts

Parts +Configuration +Relative depth

LayeredPictorialStructures(LPS)

Page 35: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Pictorial Structures

• Each parts is a variable• States are image locations•AND affine deformation

Affine warp of parts

Page 36: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Pictorial Structures

• Each parts is a variable• States are image locations • MRF favours certain

configurations

Page 37: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Bayesian Formulation (MRF)

• D = image.

• Di = pixels Є pi , given li

• (PDF Projection Theorem. )

z = sufficient statistics

• ψ(li,lj) = const, if valid configuration

= 0, otherwise.

Pott’s model

Page 38: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Defining the likelihood

• We want a likelihood that can combine both the outline and the interior appearance of a part.

• Define features which will be sufficient statistics to discriminate foreground and background:

Page 39: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Features

• Outline: z1 Chamfer distance

• Interior: z2 Textons

• Model joint distribution of z1 z2 as a 2D Gaussian.

Page 40: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Chamfer Match Score

• Outline (z1) : minimum chamfer distances over multiple outline exemplars

• dcham= 1/n Σi min{ minj ||ui-vj ||, τ }

Image Edge Image Distance Transform

Page 41: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Texton Match Score

• Texture(z2) : MRF classifier – (Varma and Zisserman, CVPR ’03)

• Multiple texture exemplars x of class t

• Textons: 3 X 3 square neighbourhood• VQ in texton space• Descriptor: histogram of texton labelling• χ2 distance

Page 42: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Bag of Words/Histogram of Textons

• Having slagged off BoW’s I reveal we used it all along, no big deal.

• So this is like a spatially aware bag of words model…

• Using a spatially flexible set of templates to work out our bag of words.

Page 43: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

2. Fitting the Model

• Cascades of classifiers– Efficient likelihood evaluation

• Solving MRF– LBP, use fast algorithm– GBP if LBP doesn’t converge– Could use Semi Definite Programming (2003)– Recent work second order cone programming

method best CVPR 2006.

Page 44: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Efficient Detection of parts

• Cascade of classifiers

• Top level use chamfer and distance transform for efficient pre filtering

• At lower level use full texture model for verification, using efficient nearest neighbour speed ups.

Page 45: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Cascade of Classifiers-for each part

Y. Amit, and D. Geman, 97?; S. Baker, S. Nayer 95

Page 46: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

High Levels based on Outline

(x,y)

Page 47: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Side Note

• Chamfer like linear classifier on distance transform image Felzenszwalb.

• Tree is a set of linear classifiers.

• Pictorial structure is a parameterized family of linear classifiers.

Page 48: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Low levels on Texture

• The top levels of the tree use outline to eliminate patches of the image.

• Efficiency: Using chamfer distance and pre computed distance map.

• Remaining candidates evaluated using full texture model.

Page 49: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Efficient Nearest Neighbour

• Goldstein, Platt and Burges (MSR Tech Report, 2003)

Conversion from fixeddistance to rectangle search

• bitvectorij(Rk) = 1

= 0• Nearest neighbour of x• Find intervals in all dimensions• ‘AND’ appropriate bitvectors• Nearest neighbour search on pruned exemplars

Rk Є Iiin dimension j

Page 50: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Recently solve via Integer Programming

• SDP formulation (Torr 2001, AI stats)

• SOCP formulation (Kumar, Torr & Zisserman this conference)

• LBP (Huttenlocher, many)

Page 51: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Outline

• Problem Formulation

• Form of Shape Prior

• Optimization

• Results

Page 52: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Optimization

• Given image D, find best labelling as m* = arg max p(m|D)

• Treat LPS parameter Θ as a latent (hidden) variable

• EM framework– E : sample the distribution over Θ

– M : obtain the labelling m

Page 53: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

E-Step

• Given initial labelling m’, determine p(Θ|m’,D)

• Problem Efficiently sampling from p(Θ|m’,D)

• Solution• We develop efficient sum-product Loopy Belief

Propagation (LBP) for matching LPS.

• Similar to efficient max-product LBP for MAP estimate– Felzenszwalb and Huttenlocher, CVPR ‘04

Page 54: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Results

• Different samples localize different parts well.• We cannot use only the MAP estimate of the LPS.

Page 55: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

M-Step

• Given samples from p(Θ|m’,D), get new labelling mnew

• Sample Θi provides– Object localization to learn RGB distributions of object and background– Shape prior for segmentation

• Problem– Maximize expected log likelihood using all samples– To efficiently obtain the new labelling

Page 56: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

M-Step

Cow Image Shape Θ1

w1 = P(Θ1|m’,D)

RGB Histogram for Object RGB Histogram for Background

Page 57: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Cow Image Shape Θ1

M-Step

w1 = P(Θ1|m’,D)

Θ1

Image PlaneD (pixels)

m (labels)

• Best labelling found efficiently using a Single Graph Cut

Page 58: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Segmentation using Graph Cuts

x …

y … … …

z … …

Obj

Bkg

CutΦx(D|bkg) + Φx(bkg|Θ)

m

Φz(D|obj) + Φz(obj|Θ)

Ψxy(mx,my)+

Φ(D|mx,my)

Page 59: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Segmentation using Graph Cuts

x …

y … … …

z … …

Obj

Bkg

m

Page 60: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

M-Step

Cow Image Shape Θ2

w2 = P(Θ2|m’,D)

RGB Histogram for BackgroundRGB Histogram for Object

Page 61: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

M-Step

Cow Image Shape Θ2

w2 = P(Θ2|m’,D)

Θ2

Image PlaneD (pixels)

m (labels)

• Best labelling found efficiently using a Single Graph Cut

Page 62: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

M-Step

Θ2

Image Plane

Θ1

Image Plane

w1 + w2 + ….

• Best labelling found efficiently using a Single Graph Cut

m* = arg min ∑ wi E (m,Θi)

Page 63: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Outline

• Problem Formulation

• Form of Shape Prior

• Optimization

• Results

Page 64: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

SegmentationImage

ResultsUsing LPS Model for Cow

Page 65: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

In the absence of a clear boundary between object and background

SegmentationImage

ResultsUsing LPS Model for Cow

Page 66: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

SegmentationImage

ResultsUsing LPS Model for Cow

Page 67: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

SegmentationImage

ResultsUsing LPS Model for Cow

Page 68: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

SegmentationImage

ResultsUsing LPS Model for Horse

Page 69: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

SegmentationImage

ResultsUsing LPS Model for Horse

Page 70: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Our Method Leibe and SchieleImage

Results

Page 71: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

AppearanceShape Shape+Appearance

Results

Without Φx(D|mx) Without Φx(mx|Θ)

Page 72: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Face Detector and ObjCut

Page 73: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Do we really need accurate models?

• Segmentation boundary can be extracted from edges

• Rough 3D Shape-prior enough for region disambiguation

Page 74: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Energy of the Pose-specific MRF

Energy to be minimized

Unary term

Shape prior

Pairwise potential

Potts model

But what should be the value of θ?

Page 75: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

The different terms of the MRF

Original image

Likelihood of being foreground given a

foreground histogram

Grimson-Stauffer

segmentation

Shape prior model

Shape prior (distance transform)

Likelihood of being foreground

given all the terms

Resulting Graph-Cuts

segmentation

Page 76: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Can segment multiple views simultaneously

Page 77: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Solve via gradient descent

• Comparable to level set methods

• Could use other approaches (e.g. Objcut)

• Need a graph cut per function evaluation

Page 78: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Formulating the Pose Inference Problem

Page 79: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

But…But…

… to compute the MAP of E(x) w.r.t the pose, it means that the unary terms will be changed at EACHEACH iteration and the maxflow recomputed!

However…However… Kohli and Torr showed how dynamic graph cuts can

be used to efficiently find MAP solutions for MRFs that change minimally from one time instant to the next: Dynamic Graph Cuts (ICCV05).

Page 80: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Dynamic Graph Cuts

PB SB

cheaperoperation

computationally

expensive operation

Simplerproblem

PB*

differencesbetweenA and B

A and Bsimilar

PA SA

solve

Page 81: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Dynamic Image Segmentation Image

Flows in n-edges Segmentation Obtained

Page 82: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

First segmentation problem MAP solution

Ga

Our Algorithm

Gb

second segmentation problem

Maximum flow

residual graph (Gr)

G`

differencebetween

Ga and Gbupdated residual

graph

Page 83: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Dynamic Graph Cut vs Active Cuts

• Our method flow recycling

• AC cut recycling

• Both methods: Tree recycling

Page 84: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Experimental Analysis

MRF consisting of 2x105 latent variables connected in a 4-neighborhood.

Running time of the dynamic algorithm

Page 85: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Segmentation Comparison

Gri

mson

-G

rim

son

-S

tau

ffer

Sta

uff

er

Bath

ia0

Bath

ia0

44O

ur

Ou

r m

eth

od

meth

od

Page 86: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Segmentation

Page 87: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Segmentation

Page 88: O BJ C UT & Pose Cut CVPR 05 ECCV 06 Philip Torr M. Pawan Kumar, Pushmeet Kohli and Andrew Zisserman UNIVERSITY OF OXFORD.

Conclusion

• Combining pose inference and segmentation worth investigating.

• Tracking = Detection

• Detection = Segmentation

• Tracking = Segmentation.

• Segmentation = SFM ??