Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision

The University of

Ontario

How to fit a surface to a point cloud?or

optimization of surface functionals in computer vision

Yuri Boykov

TRICS seminarComputer Science Department

The University of

Ontario

Optimization of surface

functionals in computer vision

Computer vs. human vision

Model fitting in computer vision• templates, pictorial structures, trees, deformable models, contours/snakes, meshes,

surfaces, complexes, graphs, weak-membrane model, Mumford-Shah, Potts

model,……

Optimization in computer vision• dynamic programming, gradient descent, PDEs, shortest paths, min. spanning

trees, linear and quadratic programming, primal-dual schema, network flow

algorithms, QPBO, ...

Applications

• segmentation, stereo, multi-view reconstruction, optical flows

• surface fitting

The University of

OntarioContours

The University of

Ontario+Shading

M.C. EscherDrawing hands

3D shapeunderstanding

The University of

Ontario+Color

Da VinciMadonna Litta

Recognition

The University of

Ontario+Texture

MagritteSouvenir de Voyage

recognizingmaterial

The University of

Ontario+Texture

The New Yorker Album of Drawings, The Viking Press, NY, 1975

recognizing 3D perspective

The University of

OntarioWhat do humans get by „looking‟?

J. VermeerThe Guitar Player

Contours

Shading

Color

Texture

…

basic image cues:

The University of

OntarioWhat do humans get by „looking‟?

Contours

Shading

Color

Texture

…

basic image cues:

Segmentation

Motion

3D shape perception

3D scene geometry

Detection/Recognition

…

higher-level perception:

The University of

OntarioWhat do computers get by „looking‟?

x

y

3D plot of image

intensity I(x,y)x

y

I(x,y)

x

y

The University of

OntarioWhat do computers get by „looking‟?

P. PicassoThe Guitar Player

Intensity discontinuities(contours)

Intensity gradients (shading)

Multi-valued intensities (color)

Filtering (e.g. texture)

…

basic image cues: higher-levelgrouping?

The University of

Ontario

model

Bayesian approach

Prior + Data Low-level cues

(local info)high-level knowledge

(global picture)

Fit some prior model into data

The University of

OntarioRigid Template Matching

• In matching we estimate “position” of a rigid template in the image • “Position” includes global location parameters of a rigid template:

- translation, rotation, scale,…

Face template

image

image

translation,rotation,scaling

The University of

OntarioNon-rigid (parametric) matching

1. Pick one image (red)

2. Warp the other images to match it (homographic transform)

3. Blend

panorama mosaicing

The University of

Ontarioe.g.… using homographies

The University of

Ontarioe.g…. using flexible templates

• In flexible template matching we estimate “position” of each rigid component of a template• For tree-structured models, efficient global optimization is possible via DP

(Felzenswalb&Huttenlocher 2002)

The University of

Ontariotracking parameters => activity recognition

Bottom-up tracker

The University of

Ontario

5-18

deformable contours (“snakes”)

2D curve which matches to image data

Initialized near target, iteratively refined

Can restore missing data

initial intermediate final

Optimization gets harder when a loop is introduced. DP does not apply.

One solution: gradient descent

Kass, Witkin, Terzopoulos 1987

The University of

Ontario

6-19Cremers, Tischhäuser, Weickert, Schnörr, “Diffusion Snakes”, IJCV '02

local minima, fixed contour topology

The University of

Ontario

6-20

A contour may be approximated from

u(x,y) with near sub-pixel accuracy

C

-0.8 0.2

0.5

0.70.30.6-0.2

-1.7

-0.6

-0.8

-0.4 -0.5

),( ppp yxuu

• Level set function u(x,y) is normally discretized/stored over image pixels

• Values of u(p) can be interpreted as distances or heights of image pixels

Implicit representation of contours

Osher&Sethian 1989

The University of

Ontario

6-21

[Visualization is courtesy of O. Juan]

Simple evolution

Morphological Operation:Erosion

1p

ppp NCd

|| ppp udu

The University of

Ontario

6-22

Visualization is courtesy of O. Juan

Example of gradient descent evolution

2

2

2

2

)(y

u

x

up dtu

Gradient descent

w.r.t.

Euclidean length

The University of

Ontario

6-23

Example of gradient descent evolution

Laplacian

Osher&Sethian 1989

Gradient descent

w.r.t.

Euclidean length

2

2

2

2

)(y

u

x

up dtu

The University of

Ontario

6-24

[example from Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE TIP ‟01]

Geodesic Active Contours via Level-sets

y

ug

x

ug

p

yxdtu)()(

)(

C

dsgCE )()(

The University of

Ontario

6-25

Other geometric energy functionals

besides length [courtesy of Ron Kimmel]

weighted length

C

dsgCE )()(

Functional E( C ) gradient descent evolution

,~ Ngg

weighted area dafCE )( ~ f

alignment(flux)

C

dsNCE ,)(

v )div(~ v

Geometric measures commonly used in segmentation

NdC

The University of

Ontario

in 3D…

deformable meshes, level-sets, …

Estimation of positionfor mesh points

Many loops.optimization - gradient

descent

GOALS: global optima (?)“right” functional (?)

Typical problems:- local minima (clutter, outliers)

-over-smoothing

The University of

Ontario

Global Optimization andSurface Functionals

The University of

Ontario

More generally...

Estimate labels for graph nodes

I

p

L

along one scan line in the image

observed noisy image I image labeling L(restored intensities)

NOTE:similar to robust regression modelestimation

The University of

Ontario

(simple example)

Piece-wise smooth restoration

Markov Random Fields (MRF) approach

weak membrane model (Geman&Geman‟84, Blake&Zisserman‟83,87)

pL qL

Nqp

qp

p

pp LLVILLE),(

2 ),()()(

discontinuity preserving prior

optimizing E(L) is NP hard!

T T

(Continuous analogue: Mamford-Shah functional, 1989)

The University of

Ontario

I

p

L

observed noisy image I image labeling L(restored intensities)

(simple example)

Piece-wise constant restoration

along one scan line in the image

The University of

Ontario

(simple example)

Piece-wise constant restoration

Potts model Boykov Veksler Zabih ’01

Greig et al.’89 (for 2 labels)

Nqp

qp

p

p

i

i

LLILLEi ),(

2 )()()(

global optimization is still NP hard, but there are fast provably good

combinatorial approximation algorithms, linear and quadratic programming,

QPBO, primal-dual schema

pL qL

0}2L:p{ p2

“perceptual grouping”

}1L:p{ p1

}0L:p{ p0

The University of

OntarioPerceptual grouping from stereo

(Birchfield &Tomasi‟99)

constant label = plane

The University of

Ontario

Binary labeling(binary image restoration)

)|Pr()()( 2

pppppp LIILLD

}1,0{pL

original binary image I optimal binary labeling L

Nqp

qp

p

pp LLLDLE),(

)()()(

Greig Porteous Seheult ‟89

Globally optimal solution is possible using combinatorial graph cut algorithms• pseudo-boolean optimization Hammer‟65, Picard&Ratlif’75

The University of

Ontario

Binary labeling(object extraction)

1pL

0pL

object segmentation

left ventricle of heart

}1,0{pL

The University of

Ontario

Binary labeling(object extraction)

1

0

C

Globally optimal solution is possible using graph cut algorithms• pseudo-boolean optimization (Hammer‟65, Picard&Ratlif‟75)

surface extraction

Nqp

qppq

p

pp LLwLDLE),(

)()()(

Boykov&Jolly‟01

left ventricle of heart

}1,0{pL

The University of

OntarioImplicit surface representation via graph-cuts

• Any contour (or surface in 3D) satisfying labeling of exterior/interiorpoints (pixel centers) is acceptable if some explicit surface has to be output.

0 1

1

1110

0

0

0

0 0

The University of

Ontario

)int(

)(,)()(CC

x

C

dpxfdsdsgCE vN

Geometric lengthany convex,

symmetric metric (e.g. Riemannian)

Flux

any vector field v

Regional bias

any scalar function f

“edge alignment”

Tight characterization for geometric functionals of contour C that can

be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05)

disclaimer: for pairwise interactions only

Global optimization of

geometric surface functionals

N)q,p(

qp

p

pp )L,L(V)L(D)L(E

The University of

OntarioGlobally optimal surfaces in 3D

Volumetric segmentation(BJ01,BK’03,KB’05)

The University of

Ontario

Binary labeling(object extraction)

Blake et al.‟04, Rother et al.‟04

}1,0{pL

iteratively re-estimate color models e.g. using mixture of Gaussians

The University of

OntarioSegmentation for Image Blending

The University of

OntarioSegmentation for Image Blending

The University of

OntarioOptimal surfaces in 3D

3D reconstruction

Vogiatzis, Torr, Cippola‟05

Local cues: voxel’s photoconsistency

Prior:smoothness,

projective geometry constraints

The University of

OntarioGlobally optimal surfaces in 3D

Lempitsky&Boykov, 2006

from a cheap digital camera

The University of

Ontario

3D model

multi-view reconstruction set up

Furukawa&Ponce 2006

Globally optimal surfaces in 3D

(texture mapped)

The University of

OntarioSurface fitting to point cloud

a cloud of 3D points (e.g. from a laser scanner)

3D model:

surface fitting:

The University of

OntarioSurface fitting to point cloud

The University of

OntarioSurface fitting functional

CC

dsdsCE 1,)( vN

Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)

ip

in

S

data fit prior

The University of

OntarioOptimal surfaces in 3D

Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)

From 10 views

No initialization is needed

CC

dsdsCE 1,)( vN

The University of

OntarioGlobal vs. local optimization

regional potentials

)(vdivf

CC

dsdpdivCE 1)()int(

v

Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)

initial solution local minima global minima

The University of

OntarioFitting to sparse data

The University of

OntarioFitting to sparse data

The University of

OntarioFitting to sparse data

The University of

OntarioSummary

Global optimization-Your solution is as good as your functional

-No need to worry about initial guess or convergence issues

-Polynomial algorithms, but many practical

issues (efficient data structures, memory limitations,

parallelization, dynamic applications,…)

- Many useful functionals are NP hard (lots of approximation methods are developed)

- New approaches allowing global optimization

are introduced (including new version of level-sets)