1 Deformation Invariant Shape and Image Matching Polikovsky Senya Advanced Topics in Computer Vision Seminar Faculty of Mathematics and Computer Science.

1

Deformation Invariant Shape and Image

Matching

Polikovsky SenyaAdvanced Topics in Computer Vision Seminar

Faculty of Mathematics and Computer Science

Weizmann Institute

May 2007

2

Based on…

Integral Invariants for Shape Matching

Siddharth Manay, Daniel Cremers, Member, Byung-Woo Hong,Anthony J. Yezzi Jr., and Stefano Soatto

Deformation Invariant Image Matching

Haibin Ling ,David W. Jacobs

Part I : Invariant Shape Matching

Can you guess what it is ?

3

4

Outline

Integral Shape Matching• Introduction

– Basic Definitions, Previous Work – Curvature

• Integral Invariant (II)– Relation of Local Area II to Curvature

– Shape Matching and Distance– Multi-scale Shape Matching

• Implementation and Experimental Results

5

Applications of shape matching

• Airport security :

• Industry quality control :

• Medical images analyses :

6

Ultimate Goal

Compare objects represented as closed

planar contours.

7

Basic Definition

• Object : – Closed planar contour.– No self-intersections.

• Shape : – Equivalence class of objects. Obtained under the action of a finite-dimensional group : Euclidean, similarity, affine, projective group.

8

Basic Definition (cont.)

In Addition:

- articulated

- occluded

- jagged (not obtained with standard additive, zero mean)

Two objects have the same shape if and only if

one can be generated by transformation group actions on other shape.

9

Summary Goal

• Define a distance so that shapes that vary by Euclidean transformations have zero distance.

• Shapes vary by scaling , articulation , occlusion have small distance.

• Resistant to : “small deformations” “high-frequency noise” “localized changes”

10

Summary Goal (cont.)

The babies should have low distance to each other,

but high distance to other classes of shapes.

Low distance High distance

11

Previous Work

• Statistical approach using moments. [35],[27]

High-order moments sensitive to noise.

• Normalized Fourier descriptors. [93],[52],[2]

High order Fourier coefficients are not stable with respect to noise.

• Local neighborhoods using Wavelet transform. [83],[34]

Previous Work (Cont.)

1212

• Global radial histogram of the relative coordinates of the rest of the shape at each point. [6]

• Differential invariants (Curvature)

[44], [36], [17], [58], [76],

[30], [48], [64], [91], [85], [37]

13

Curvature

• Intuitively, curvature is the amount by which a geometric object deviates from being flat.

Less general y = f(x) Plane curve c(t) = (x(t),y(t))

2/322 )()(

yx

xyyxtk

2/32 )1( y

yk

• Curvature can be also be geometrical understood as in terms of osculating (kiss) circle and radius curvature.

14

Curvature (cont.)

• Circle of radius r has curvature 1/r everywhere .• Straight line r = ∞ has curvature 1/ ∞ =0 everywhere

• Features:– Invariant to rotation, reflections of the original curve.

– NOT invariant to scaling.

15

Outline

Integral Shape Matching• Introduction

– Basic Definitions, Previous Work – Curvature

• Integral Invariant (II)– Relation of Local Area II to Curvature

– Shape Matching and Distance– Multi-scale Shape Matching

• Implementation and Experimental Results

16

• Closed planar contour. C:S1 → R2. ds - infinitesimal arclength.

• G - transformation group acting on R2. dx - area %on R2.

• μ curve C corresponding measure dμ(x).

Integral Invariant

17

Integral Invariant

• Function I: R2 → R G-invariant if satisfies:

• I(.) is associates to each point on contour a real number.

GgCgICI )()(

Cp

18

Step 1: Curvature

Curvature κ(C) of curve C is G-invariant.

19

General notion of Integral Invariant

Function IC(p) : R2 → R is and integral G-invariant

Kernel k : R2 X R2 → R.

K( ● , ● ) :

• p don’t necessarily lie on the curve C.

)(),()( xdxpkpIC

},|{

)( ),()( ),(

xGggxg

Ggxdxgpkxdxpkg

p

20

Step 2: Distance integral invariant

CpxdsxppICC )()(

Unlike curvature distance invariant is R+.

(Euclidean distance is always nonnegative)

• For global descriptor , local change of a shape affects the values of the distance integral invariant for the entire shape.

21

Step 2: Distance integral invariant

CpxdsxppICC )()(

22

Step 3: “Shape Context”

)(),()( xdxpkpIC ))(,())(,(),(

),(),(),(

sCpdsCpqxpk

xpdxpqxpk

Preserves locality can be obtained by weighting the integral with a kernel q(p, x).

))(,( sCpq ))(,( sCpd

Local radial histogram.

23

Step 3: “Shape Context” (cont.)• NOT discriminative, same value for different geometric features.

r – radius. p - center of the ball. C - interior of the region bounded by C.

24

Finale Step: Local Area Integral Invar.

• Define a ball Br(p), Br : R2 X R2 {0,1} ; ( R+ )

0

1),(

otherwith

rxpxpBr )(),()( xdxpBpI

C rrC

)(),(

)(),()(

2xdxpB

xdxpBpI

R r

C rrC

25

Local Area Integral Invar.(cont.)

rr

• Naturally forms a multi-scale invariant.

26

Relation of Local AII to Curvature

22)( rpI rC

))(2

1(2)( 12 prcorrpI r

C

R

r

2)cos(

r

C

r

R

θ

C

Assume that C is smooth curve, because of the curvature.

pRp )(

27

Outline

Integral Shape Matching• Introduction

– Basic Definitions, Previous Work – Curvature

• Integral Invariant (II)– Relation of Local Area II to Curvature

– Shape Matching and Distance– Multi-scale Shape Matching

• Implementation and Experimental Results

28

Shape Matching and Distance

• Shape distance is a scalar value that quantifies similarity of the two contours. D(C1,C2).

• Basing on group invariant , integral invariant :– invariant to G group action. – robust to noise and local deformations

• Corresponding points.

C1 = C2=

Local Are Integral Invariant : I1 , I2

,

29

Corresponding points

• Disparity function d(s).

Reparameterizes C1 ,I1 and C2 ,I2 .

• optimal point correspondence

);,,(minarg)(* 21)(

sdIIEsdsd

SssdsCsdsC ,))((~))(( *2

*1

( ~ denotes correspondence)

30

Energy Functional E(...,d)

E1 - measures the similarity of two curves.E2 - elastic energy. α - control parameter α > 0.

dssdsdsIsdsI

dEdIIEdIIE

2'21

0 21

'2,211,21

)())(())((

)(),(),(

• If d(s) = 0 , direct match.• If d’(s) = 0 , circular “shifts”.• Other d(s), “stretch” , “shrink”.

31

Control Parameter α

small α :

large α :

d*(s)

d*(s)

32

Multi-scale Shape Matching

Trace of local extrema across scales

Curvature Scale-space Integral Invariant Scale-space

Curvature scale-space is derived from Gaussian smoothing.

33

Multi-scale Shape Matching (cont.)

• Matching shapes of different sizes.

R R’

'

'

R

r

R

r

• Normalized kernel radius : r / R.

34

Multi-scale Shape Matching (cont.)

Control parameter α. Size of the kernel width r.

Correspondences between two signals influenced by:

fine scale intermediate scale coarse scale

35

Outline

Integral Shape Matching• Introduction

– Basic Definitions, Previous Work – Curvature

• Integral Invariant (II)– Relation of Local Area II to Curvature

– Shape Matching and Distance– Multi-scale Shape Matching

• Implementation and Experimental Results

36

Implementation

C1

C2

C2[j]C2[j+1]

C1[i]

C1[i+1]v[

i , j

]

v[i ,

j +

1]

v[i + 1, j]v[

I +

1, j

+ 1

]

Each point in each curve must have at least onecorresponding point in the other curve.

v[i , j]

v[I + 1, j + 1]

v[i +1, j]

v[i , j+1]

e e

e

e = v(i,j) → v(k,l)

Minimization of the energy functional E.

37

Implementation (cont.)

),()...0,0(0 MNvvvv L

Lvvvp ,..., 10

• Minimization of the energy functional E is equivalent to finding a shortest path that gives a minimum weight.

L

ttt vvwpw

01),()(

w(p) ← E(I1,I2,d)Graph used to compute the correspondence for two

curves with M = N = 5.

nodes = MNedges = 3MN

v[i , j]

v[I + 1, j + 1]

v[i +1, j]

v[i , j+1]

e e

e

M

N

0

0

1

1

C1

C2

38

Implementation (cont.)

• In previous implementations we choose start point in C1 and run all over C2.

• Alternately, observing strong features, in the invariant space.

39

Experimental Results

The gray levels indicate the dissimilarity between points lighter shade indicates higher dissimilarity.

dissimilarity

higher dissimilarity

40

Experimental Results (cont.)

“On Aligning Curves” [76]

• 100 sample on contours• r = 15 • α = 0.1

41

Experimental Results (cont.)

Int.Inv.

Curvature

Curvature

Histograms of shape distance between Shape 24 and1,000 perturbations of Shape 20 with noise at variance = 2.5.

Int.Inv.

Curvature

42

Experimental Results (cont.)

Noisy shapes (across top) and original shapes (along left side)vie differential invariant

dissimilarity

higher dissimilarity

43

Experimental Results (cont.)

Noisy shapes (across top) and original shapes (along left side)via integral invariant

dissimilarity

higher dissimilarity

44

Summary

Curvature.

Four Steps : – Curvature, sensitivity to noise.– Global descriptor, local change.– Local radial histogram, NOT discriminative.– Local Area Integral Invar.

Shape Matching ,Multi-scale Shape Matching.

Implementation.

Results.

M

N

0

0

1

1C1

C2

Part II :Deformation Invariant Image

Matching

45

Outline

Introduction

Deformation Invariant Framework

Experiments

Summary

General Deformation

• One-to-one, continuous mapping.• Intensity values are deformation invariant.

– (their positions may change)• Affine model for lighting change. (Out of the scope)

Solution

• A deformation invariant framework

– Embed images as surfaces in 3D

– Geodesic distance is made deformation invariant by adjusting an embedding parameter

– Build deformation invariant descriptors using geodesic distances

Outline

Introduction

Deformation Invariant Framework- Intuition through 1D images

- 2D images

Experiments

Summary

1D Image Embedding

1D Image I(x):

EMBEDDINGI(x) ( (1-α)x, αI )αI(1-α)x

Aspect weight α : measures the importance of the intensity

2D Surface :

Geodesic Distance

αI

(1-α)x

p qg(p,q)

• Length of the shortest path along surface

Geodesic Distance and α

I1 I2

Geodesic distance becomes deformation invariant

for α close to 1

Image Embedding & Curve Lengths

]1,0[:),( 2 RyxI

dtIyxl ttt 222222 )1()1(

))('),('),('()( tztytxt

Depends only on intensity I Deformation Invariant

IzyyxxI ',)1(',)1('),(

dtIl t

21

Image I:

Embedded Surface σ :

Curve γ on σ:

Length of γ:

Take limit

Geodesic distances for real images

55

Automatically adapts to deformation.

Almost like Euclidean distances

• Interest point p0 = (x0, y0).• Compute Geodesic distances from p0 to all other points on embedded surface σ(I; α).

α = 0

α = 0.98

Geodesic Distance for 2D Images

• Computation– Geodesic level curves – Fast marching [Sethian96]

is the marching speed 2/122222)1(

yx IIF

• Geodesic distance– Shortest path– Deformation invariant

F

T is the geodesic distance

T=1T=2T=3

T=4

p

q1|| FT

Deformation Invariant Sampling

Geodesic Sampling1. Fast marching: get

geodesic level curves with sampling interval Δ

2. Sampling along level curves with Δ

p

sparsedense

Δ

ΔΔ

Δ

Δ

Geodesic-Intensity Histogram(GIH)• Divide 2D intensity-geodesic distance space into K×M bins. K - number of intensity intervals. M - number of geodesic distance intervals.• Insert all points in Pp into Hp.• Normalize each column of Hp Normalize the whole Hp.

Deformation Invariant Descriptor

p qp q

geodesic distance M

Inte

nsit

y

K

geodesic distance M

Inte

nsit

y

K

Real Example

pq

Invariant vs. Descriminative

0

10

1

Deformation Invariant Framework

Image Embedding ( α close to 1)I(x,y) → σ(I, α)

Deformation Invariant SamplingGeodesic Sampling

Build Deformation Invariant Descriptors(GIH)

Practical Issues

• Interest-Point– No special interest-point is required.

Automatically locates the support region by

Geodesic Sampling.

But:– Points on constant region indistinguishable.– Corners may vary due to sampling.

• Extreme point , local intensity extremum,

is more reliable and effective.

Outline

Introduction

Deformation Invariant Framework- Intuition through 1D images

- 2D images

Experiments- Interest-point matching

Summary

Data Sets

Synthetic Deformation & Lighting Change (8 pairs) Real Deformation (3 pairs)

Interest-PointsInterest-point Matching

• Harris-affine points [Mikolajczyk&Schmid04] *

• Affine invariant support regions• Not required by GIH• 200 points per image

• Ground-truth labeling• Automatically for synthetic image pairs• Manually for real image pairs

• Correct match, three pixel distance.

Descriptors & Performance Evaluation

Descriptors• GIH compared with following descriptors:

Steerable filter [Freeman&Adelson91], SIFT [Lowe04], Moments [VanGool&etal96], Complex filter [Schaffalitzky&Zisserman02], Spin Image [Lazebnik&etal05]• α = 0.98 Performance Evaluation• Receiver Operating Characteristics (ROC) curve: detection

rate among top N matches. • Detection rate:

matches possible#

matchescorrect #r

Synthetic Image Pairs

N – top matches.

Real Image Pairs

N – top matches.

Outline

Introduction

Deformation Invariant Framework- Intuition through 1D images

- 2D images

Experiments- Interest-point matching

Summary

Summary

1D Image Intuition.

Geodesic Distance and α.

Geodesic Sampling.

Performance Evaluation.

p ΔΔ ΔΔ

Δ

Thank You!