Object Recognition Based on Shape Similarity Longin Jan Latecki Computer and Information Sciences Dept. Temple Univ., latecki@temple.edulatecki@temple.edu.

Object Recognition Based on Shape Similarity

Longin Jan LateckiComputer and Information Sciences Dept. Temple

Univ., latecki@temple.eduCollaborators:

Zygmunt Pizlo, Psychological Sciences Dept., Purdue Univ.,

Nagesh Adluru, Suzan Köknar-Tezel, Rolf Lakaemper, Thomas Young, Temple Univ.,

Xiang Bai, Huazhong Univ. of Sci. & Tech. Wuhan, China

2

Object Recognition Process:

Source:2D image of a 3D object

Matching: Correspondence of Visual Parts

Contour Segmentation

Contour Extraction

Object Segmentation

Contour Cleaning, e.g., Evolution

3

Motivation Once a significant visual part is

recognized the whole recognition process is strongly constrained in possible top-down object models.

(H1) object recognition is preceded by, and based on recognition of visual parts.

(H2): contour extraction is driven by shape similarity to a known shape.

4What do you see?

5With grouping constraints we can see (i.e., recognize the object).

6

Object contours Psychophysical and

neurophysiological studies provide an abundance of evidence that contours of objects are extracted in early processing stages of human visual perception.

Contours play a central role in the Gestalt-theory.

7

Salient visual parts can influence the object

recognition (Singh and Hoffman 2001)

8

Salient visual parts can influence the object

recognition (Singh and Hoffman 2001)

9

Salient visual parts can influence the object

recognition (Singh and Hoffman 2001)

10

Visual parts and shape similarity (H1) object recognition is preceded by, and

based on shape recognition of visual parts.

(H2): contour extraction is driven by shape similarity to a known shape. becomes:

(H2) Contour extraction is based on grouping of contour parts to larger contour parts with grouping assignments driven by shape familiarity.

11

Contour detection is a difficult inverse problem

A given image could be produced by infinitely many possible 3D scenes. In order to produce a unique, stable and accurate interpretation, the visual system must use a priori constraints (see Pizlo, 2001 for a review).

The solution is obtained by optimizing a cost

function which consists of two general terms: 1. how close the solution is to the visual data 2. how well the constraints are satisfied

12

Partial shape similarity

13

Partial shape similarity (1) length problem, (2) scale problem, (3) distortion problem

Query Shape Target Shape Target Shape

Given only a part (of a shape ), find similar shapes

14

Partial Shape Similarity We reduce partial shape similarity to

subsequence matching: This is done by computing a curvature like

value at every contour point. We do this for complete contours of

known objects in our database and for query contour parts extracted

from edge images

15

Subsequence Matching (shape similarity)

Databasecontours

Querycontours

16

Motivation for subsequence matching

OSB DTW

LCSS with threshold 0.50 LCSS with threshold 1.00

The top (red) and bottom (blue) sequences represent parts of contours of two different but very similar bone shapes

17

Motivation(2)Example sequences:a = {1, 2, 8, 6, 8}b = {1, 2, 9, 15, 3, 5, 9}

18

OSB Algorithm Goal: given two real-valued sequences a and b, find subsequences a’ of a and b’ of b such that a’ best matches b’ Possible to skip elements in both a and b

• The ability to exclude outliers Preserve the order of the elements A one-to-one correspondence

19

OSB Algorithm (2)

Create a dissimilarity matrix No restrictions on the distance function d

• We used d(ai,bj) = (ai – bj)2

To find the optimal correspondence, use a shortest path algorithm on a DAG

20

OSB Algorithm (3)

The nodes of the DAG are all the index pairs of the matrix: (i,j){1,…,m}{1,…,n}

The edge weights w are defined by

C is the jump cost (the penalty for skipping an element)

otherwise

if ),()2()1()),(),,((

22 ljkibadCjliklkjiw lk

21

OSB Algorithm (4) The edge cost may be extended to

impose a warping window Set a maximal value for k – i – 1 and l

– j - 1 This definition of the edge weights

is our main contribution

22

A Simple Examplea = {1, 2, 8, 6, 8}b = {1, 2, 9, 15, 3, 5, 9}

b

1 2 9 15 3 5 9

a

1 0 1 64 196 4 16 64

2 1 0 49 169 1 9 49

8 49 36 1 49 25 9 1

6 25 16 9 81 9 1 9

8 49 36 1 49 25 9 1

d(ai,bj) = (ai – bj)2

The

dis

sim

ilarit

y m

atrix

23

(indices)elementsdistance

Key

(1,1)1 - 1

0

(1,2)1 - 2

1

(1,3)1 - 964

(1,4)1 - 15196

(1,5)1 - 3

4

(1,6)1 - 516

(1,7)1 - 964

(2,1)2 - 1

1

(2,2)2 - 2

0

(2,3)2 - 949

(2,4)2 - 15169

(2,5)2 - 3

1

(2,6)2 - 5

9

(2,7)2 - 949

(3,1)8 - 149

(3,2)8 - 236

(3,3)8 - 9

1

(3,4)8 - 15

49

(3,5)8 - 325

(3,6)8 - 5

9

(3,7)8 - 9

1

(4,1)6 - 125

(4,2)6 - 216

(4,3)6 - 9

9

(4,4)6 - 15

81

(4,5)6 - 3

9

(4,6)6 - 5

1

(4,7)6 - 9

9

(5,1)8 - 149

(5,2)8 - 236

(5,3)8 - 9

1

(5,4)8 - 15

49

(5,5)8 - 325

(5,6)8 - 5

9

(5,7)8 - 9

1

. . .

. . .. . .

. . .

. . .

... ......

The

DA

G

24

Experimental results onMPEG 7 dataset,1400 targets in 70 classes

25

Humans group contours automatically An adaptive, probabilistic process to

perform grouping All shapes contain local symmetry

exploit local symmetry

How to find contour parts in images?

26

Shape model

27

28

29

Play movie

30

Rao-Blackwellized particle filter is an adaptive, probabilistic approach

Frequently utilized in SLAM approaches to Robot Mapping

Each particle’s successor is its most likely successor

Particles are resampled to eliminate poorly performing particles

Contour Grouping as Robot Mapping

31

Traversal space generated as discrete “center points”

32

Center Points

Center points act as center points for maximal radius disk between the two sample points

Full set of center points gives full set of maximal radius disks Entire set of potential skeletal points Want to generate a skeletal path that

best groups the segments for a given shape model

33

Center points and particle paths

34

Shape model

System needs to utilize reference model Some a priori knowledge to discover

the proper shape Model is a sequence of radii at sample

skeleton points Position in reference model determined

by triangulation

35

Contour Smoothness

Smoothness as a criterion for segment selection Smoothness is the measure of the amount of

turn and the distance between segments Use least sum of distance to determine both

distance and the segment pairing Smoothness as Gaussian mixture of distance

and angle

36

SLAM framework

Obtain particles by sampling from the maximum posterior probability x is the path traversal m is the contour grouping model z is the observations u is the reference model

37

Particle filter

1) Sampling: The next generation of particles x(i)t is obtained from

the current generation x(i)t-1 by sampling from a proposal distribution

for

38

2) Importance Weighting: An individual importance weight w(i) is assigned to each particle, according to

39

log(M(c2)) – log of pdf that a given pixel is a center point of radius 10

40

log(M(c2)) – log of pdf that a given pixel is a center point of radius 10

41

4) Contour Estimating: For each pose sample x(i)t, the corresponding

contour estimate m(i)t is computed based on the trajectory

and the history of observations according to

3) Resampling: Particles with a low importance weight w(i)

are typically replaced by samples with a high weight.Residual resampling was used

42

Results Grouping performed on several

pictures

Useful groupings on many images Little or no noise grouped Few structural particles missed

43

44

45

Future Work Integration of the shape similarity of

parts and the contour grouping Learning good contour parts Further improvements to the

contour grouping to make it more robust