Shape-Representation and Shape Similarity. Motivation WHY SHAPE ?

Shape-Representation

and

Shape Similarity

Motivation

WHY SHAPE ?

Motivation

We’ve seen this already in the introductionof this course:These objects are recognized by…

Motivation

These objects are recognized by…

Texture Color Context Shape

X X

X X

X

X

X

X X

Motivation

Shape is not the only, but a verypowerful descriptor of

image content

Why Shape ?

Several applications in computer vision use shape processing:

• Object recognition• Image retrieval

• Processing of pictorial information• Video compression (eg. MPEG-7)

…

(Reminder: this course focuses on object recognition and image retrieval)

Blobworld

Example 1: Blobworldhttp://elib.cs.berkeley.edu/photos/blobworld/start.html

BLOB = “Binary Large Object”, “an indistinct shapeless (really ?) form”

http://elib.cs.berkeley.edu/photos/blobworld/start.html











Blobworld

Description of Blobworld:

PROJECT !

ISS Database

Example 2: ISS-Databasehttp://knight.cis.temple.edu/~shape

http://knight.cis.temple.edu/~shape





The Interface (JAVA – Applet)

ISS Database

ISS: Query by Shape

Sketch of Shape

Query:

by Shape only

Result:

Satisfying ?

ISS Database

The ISS-Database will be topicof this tutorial

Overview

Overview

• Why shape ?• What is shape ?• Shape similarity• (Metrices)

• Classes of similarity measures• (Feature Based Coding)

• Examples for global similarity

Why Shape ?

Why Shape ?

• Shape is probably the most important property that is perceived about objects. It allows to predict more facts about an object than other features, e.g. color (Palmer 1999)

• Thus, recognizing shape is crucial for object recognition. In some applications it may be the only feature present, e.g. logo recognition

Why Shape ?

Shape is not only perceived by visual means:

• tactical sensors can also provide shape information that are processed in a similar way.

• robots’ range sensor provide shape information, too.

Shape

Typical problems:

• How to describe shape ? • What is the matching

transformation?• No one-to-one correspondence• Occlusion• Noise

Shape

• Partial match: only a part of the query appears in a part of the database shape

What is Shape ?

What is Shape ?

Plato, "Meno", 380 BC:

• "figure is the only existing thing that is found always following color“

• "figure is limit of solid"

What is Shape ?

… let’s start with some properties easier to agree on:

• Shape describes a spatial regionShape is a (the ?) specific part of spatial cognition

• Typically addresses 2D space

What is Shape ?

• 3D => 2D projection

What is Shape ?

• the original 3D (?) object

What is Shape ?

Moving on from the naive understanding, some questions arise:

• Is there a maximum size for a shape to be a shape?

• Can a shape have holes?• Does shape always describe a connected

region?• How to deal with/represent partial shapes

(occlusion / partial match) ?

What is Shape ?

Shape or Not ?

Continuous transformation from shape to two shapes: Is there a point when it stops being a single shape?

What is Shape ?

But there’s no doubt that

a single, connected region

is a shape.

Right ?

What is Shape ?

A single, connected region.But a shape ?

A question of scale !

What is Shape ?

• There’s no easy, single definition of shape• In difference to geometry, arbitrary shape is not

covered by an axiomatic system

• Different applications in object recognition focus on different shape related features• Special shapes can be handled

• Typically, applications in object recognition employ a similarity measure to determine a plausibility that two shapes correspond to each other

Similarity

So the new question is:

What is Shape Similarity ?

or

How to Define a Similarity Measure

Similarity

Again: it’s not so simple (sorry).

There’s nothing like

THE

similarity measure

Similarity

which similarity measure,

depends onwhich required properties,

depends onwhich particular matching problem,

depends onwhich application

Similarity

... robustness

... invariance to basic transformations

Simple Recognition (yes / no)

Common Rating (best of ...)

Analytical Rating (best of, but...)

…which application

Similarity

…which problem

• computation problem: d(A,B)

• decision problem: d(A,B) <e ?

• decision problem: is there g: d(g(A),B) <e ?

• optimization problem: find g: min d(g(A),B)

Similarity

…which properties:

We concentrate here on the computational problem d(A,B)

Similarity Measure

Requirements to a similarity measure

• Should not incorporate context knowledge (no AI), thus computes generic shape similarity

Similarity Measure


• Must be able to deal with noise• Must be invariant with respect to basic

transformations

Next:StrategyScaling (or resolution)

Rotation

Rigid / non-rigid deformation

Similarity Measure


• Must be able to deal with noise

• Must be invariant with respect to basic transformations

• Must be in accord with human perception

Similarity Measure

Some other aspects worth consideration:

• Similarity of structure• Similarity of area

Can all these aspects be expressed by a single number?

Similarity Measure

Desired Properties of a Similarity Function C(Basri et al. 1998)

• C should be a metric• C should be continous• C should be invariant (to…)

Properties

Metric Properties

S set of patternsMetric: d: S S R satisfying1. Self-identity: xS, d(x,x)=02. Positivity: x yS, d(x,y)>03. Symmetry: x, yS, d(x,y)= d(y,x)4. Triangle inequality: x, y, zS, d(x,z)d(x,y)

+d(y,z)

• Semi-metric: 1, 2, 3• Pseudo-metric: 1, 3, 4• S with fixed metric d is called metric space

Properties

1. Self-identity: xS, d(x,x)=02. Positivity: x yS, d(x,y)>0

…surely makes sense

Properties

Properties

Properties

In general:

• a similarity measure in accordance with human perception is NOT a metric. This leads to deep problems in further processing, e.g. clustering, since most of these algorithms need metric spaces !

Properties

Properties

Properties

Some more properties:

• One major difference should cause a greater dissimilarity than some minor ones.

• S must not diverge for curves that are not smooth (e.g. polygons).

Similarity Measures

Classes of Similarity Measures:

Similarity Measure depends on

• Shape Representation

• Boundary

• Area (discrete: = point set)

• Structural (e.g. Skeleton)

• Comparison Model

• feature vector

• direct

Similarity Measures

direct feature based

Boundary Spring model, Cum. Angular Function, Chaincode, Arc Decomposition (ASR-Algorithm)

Central Dist. Fourier

Distance histogram

…

Area (point set) Hausdorff

…

Moments

Zernike Moments

…

Structure Skeleton

…

---

Feature Based Coding

Feature Based Coding (again…)

This category defines all approaches that determine a feature-vector for a given shape.

Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors.

Representation Feature Extraction Vector Comparison

Vector Comparison

We’ve done this already with

• histograms,

• fourier spectra,

• shape features (centroid distance spectrum)

Vector Comparison

Another feature you should have heard of:

(Discrete) MomentsShape A,B given as

• Area (continous) or

• Point Sets (discrete)

Moments

Discrete Point Sets

Moments

Moments

Moments

Discrete Moments

Exercise:

Please compute all 7 moments for the following shapes, compare the vectors using different comparison techniques

Discrete Moments

Result: each shape is transformed to a 7-dimensional vector. To compare the shapes, compare the vectors (how ?).

3D Distance Histogram

Another Example


Shape A,B given as 3D point set


PROJECT !

Feature Based Coding

Again:

Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors.

We hence have TWO TIMES an information reduction of the basic representation, which by itself is already a mapping of the ‘reality’.

Representation Feature Extraction Vector Comparison

Direct Comparison

End of Feature Based Coding !

Next:

Direct Comparison

Vector ComparisonDirect Comparison

Example 1

Hausdorff Distance

Shape A,B given as point sets

A={a1,a2,…}

B={b1,b2,…}

Vector ComparisonFeature Based Coding

Vector ComparisonHausdorff Distance

Vector ComparisonBoundary Representation

Hausdorff:

Unstable with respect to noise(This is easy to fix ! How ?)

Problem: Invariance !Nevertheless: Hausdorff is the motor behind many applications in specific fields (e.g. character recognition)


Example 2

Chaincode Comparison

Shape A,B given as chaincode



A binary image can be converted into a ‘chain code’ representing the boundary. The boundary is traversed and a string representing the curvature is constructed.

0

123

4

5 6 7

C

5,6,6,3,3,4,3,2,3,4,5,3,…


Resulting strings are then compared using classical string-matching techniques.

Not very robust.


Digital curves suffer from effects caused by digitalization, e.g. rotation:

Vector ComparisonStructural Representation

Structural approaches capture the

structure of a shape, typically by

representing shape as a graph.

Typical example: skeletons


Skeletons

Shape A,B primarily given as area or boundary, structure is derived from

representation


The computation can be described as a medial axis transform, a kind of discrete generalized voronoi.


The graph is constructed mirroring the adjacency of the skeleton’s parts. Edges are labeled according to the qualitative classes.

Matching two shapes requires matching two usually different graphs against each other.


Problems of skeletons:

- Pruning


-Robustness

Vector ComparisonShape similarity

All similarity measures shown can not deal with occlusions or partial matching (except skeletons ?) !

They are useful (and used) for specific applications, but are not sufficient to deal with arbitrary shapes

Solution: Part – based similarity !

Shape-Representation

and

Shape SimilarityPART 2: PART BASED SIMILARITY

Motivation

WHY PARTS ?

Motivation

Motivation

Motivation

Motivation

Global similarity measures fail at:

• Occlusion• Global Deformation• Partial Match• (actually everything that occurs under ‘real’ conditions)

Parts

Requirements for a Part Based Shape Representation

(Siddiqi / Kimia ’96: ‘Parts of Visual Form: Computational Aspects’)

Parts

How should parts be defined / computed ?

Some approaches:

• Decomposition of interior• Skeletons• Maximally convex parts• Best combination of primitives

• Boundary Based• High Curvature Points• Constant Curvature Segments

Parts

Principal approach:

Hoffman/Richards (’85):

‘Part decomposition should precede part description’

=> No primitives, but general principles

Parts

No primitives, but general principals

“When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)

Parts

“When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)

Divide a plane curve into parts at negative minima of curvature

Parts

Different notions of parts:

• Parts: object is composed of rigid parts

• Protrusions: object arises from object by deformation due to a (growth) process (morphology)

• Bends: Parts are result of bending the base object

Parts

The Shape Triangle

Parts

This lecture focuses on parts, i.e. on partitioning a shape

Framework

A Framework for a Partitioning Scheme

Scheme must be invariant to 2 classes of changes:

• Global changes : translations, rotations & scaling of 2D shape, viewpoint,…

• Local changes: occlusions, movement of parts (rigid/non-rigid deformation)

Framework

A general decomposition of a shape should be based on the

interaction between two parts rather than on their shapes.

-> Partitioning by Part Lines

Framework

Definition 1:

A part line is a curve whose end points rest on the boundary of the shape, which is entirely embedded in it, and which divides it into two connected components.

Definition 2:

A partitioning scheme is a mapping of a connected region in the image to a finite set of connected regions separated by part-lines.

Framework

Definition 3:

A partitioning scheme is invariant if the part lines of a shape that is transformed by a combination of translations, rotations and scalings are transformed in exactly the same manner.

Framework

Definition 4:

A partitioning scheme is robust if for any two shapes A and B, which are exactly the same in some neighborhood N, the part lines contained in N for A and B are exactly equivalent.

Framework

Definition 5:

A partitioning scheme is stable if slight deformations of the boundary of a shape cause only slight changes in its part lines

Framework

Definition 6:

A partitioning scheme is scale-tuned if when moving from coarse to fine scale, part lines are only added, not removed, leading to a hierarchy of parts.

Framework

A general purpose partitioning scheme that is consistent with

these requirements is the partitioning by

limbs and necks

Framework

Definition :

A limb is a part-line going through a pair of negative curvature minima with co-circular boundary tangents on (at least) one side of the part-line

Limbs and Necks

Motivation: co-circularity

Limbs and Necks

The decomposition of the right figure is no longer intuitive: absence of ‘good continuation’

Smooth continuation: an example for

form from function

• Shape of object is given by natural function

• Different parts having different functions show sharp changes in the 3d surface of the connection

• Projection to 2d yields high curvature points

Limbs and Necks

Examples of limb based parts

Limbs and Necks

Definition :A neck is a part-line which is also a

local minimum of the diameter of an inscribed circle

Limbs and Necks

Motivation for necks: Form From Function

• Natural requirements (e.g. space for articulation and economy of mass at the connection) lead to a narrowing of the joint between two parts

Limbs and Necks

The Limb and Neck partitioning scheme is consistent with the

previously defined requirements

• Invariance• Robustness• Stability• Scale tuning

Limbs and Necks

Examples:

Limbs and Necks

The scheme presented does NOT include a similarity measure !

Limbs and Necks

Part Respecting Similarity Measures

Algorithms

Curvature Scale Space(Mokhtarian/Abbasi/Kittler)

A similarity measure implicitely respecting parts

CSS

CSS

Creation of reflection-point based feature-vector which implicitely contains part – information

CSS

Properties:

• Boundary Based• Continous Model (!) • Computes Feature Vector

• compact representation of shape• Performs well !

CSS

PROJECT !

CSS

Some results (Database: 450 marine animals)

CSS

The main problem:

CSS is continous, the computer vision world is discrete.

How to measure curvature in discrete boundaries ?

Dominant Points

Local curvature = average curvature in ‘region of support’

To define regions of support, ‘dominant points’ are needed !

Dominant Points

Dominant Points(“Things should be expressed as simple as possible, but not simpler”,

A. Einstein)

Idea: given a discrete boundary S compute polygonal boundary S’ with minimum number of vertices which is

visually similar to S.

Dominant Points

Example Algorithms( 3 of billions…)

• Ramer• Line Fitting

• Discrete Curve Evolution

DCE

Discrete Curve Evolution(Latecki / Lakaemper ’99)

Idea:

Detect subset of visually significant points

Curve Evolution

Target: reduce data by elimination of irrelevant features, preserve relevant features

... noise reduction

... shape simplification:

Curve Evolution: Tangent Space

Transformation from image-space to tangent-space

Tangent Space: Properties

In tangent space...

... the height of a step shows the turn-angle

... monotonic increasing intervals represent convex arcs

... height-shifting corresponds to rotation

... the resulting curve can be interpreted as 1 – dimensional signal => idea: filter signal in tangent space (demo: 'fishapplet')

Curve Evolution: Step Compensation

(Nonlinear) filter: merging of 2 steps with area – difference F given by:

pq p + q

F

F

F

q

p


Interpretation in image – space:

... Polygon – linearization

... removal of visual irrelevant vertices

p q

removed vertex


Interpretation in image – space:

... Polygon – linearization

... removal of visual irrelevant vertices

next:Iterative SC

Curve Evolution: Iterative Step Compensation

Keep it simple: repeated step compensation !

Remark: there are of course some traps ...

(demo: EvoApplet)

Curve Evolution: Iterative Step Compensation

Remark: there are of course some traps:

Self intersection / Topology preservation

Stop parameter

Edge movement

The evolution...

... reduces the shape-complexity

... is robust to noise

... is invariant to translation, scaling and rotation

... preserves the position of important vertices

... extracts line segments

... is in accord with visual perception

... offers noise-reduction and shape abstraction

... is parameter free

Curve Evolution: Properties

... is translatable to higher dimensions


Robustness (demo: noiseApplet)


Preservation of position, no blurring !

Strong relation to digital lines and segments


Noise reduction as well as shape abstraction


Parameter free (?)


Extendable to higher dimensions








Result: The DCE creates a polygonal shape representation in different levels of granularity: Scale Space Curvature can be defined as the turning angle at the vertices Regions of support are defined by vertices Easy traceable Scale Space is created, since no points are relocated


Scale Space

Ordered set of representations on different information levels

The polygonal representation achieved by the DCE has a huge

advantage:

It allows easy boundary partitioning using convex / concave

parts (remember the limbs !)

(MATLAB Demo MatchingDemo)

Polygonal Representation

Some results of part line decomposition:

DCE

The ASR (Advanced Shape Recognition) Algorithm uses the boundary parts achieved by the

polygonal representation for a part based similarity measure !

(Note: this is NOT the area partitioning shown in the previous slide)

ASR

The ASR is used in the ISS Database

ASR / ISS

Behind The Scenes of the ISS - Database:

Modern Techniques of ShapeRecognition and Database Retrieval

How does it work ?

The 2nd Step First: Shape Comparison

Developed by Hamburg University in cooperation withSiemens AG, Munich, for industrial applications in...

... robotics

... multimedia (MPEG – 7)

ISS implements the ASR (Advanced Shape Recognition) Algorithm

Reticent Proudness…

MPEG-7: ASR outperformes classical approaches !

Similarity test (70 basic shapes, 20 different deformations):

Wavelet Contour Heinrich Hertz Institute Berlin 67.67 %

Multilayer Eigenvector Hyundai 70.33 %

Curvature Scale Space Mitsubishi ITE-VIL 75.44 %

ASR Hamburg Univ./Siemens AG 76.45 %

DAG Ordered Trees Mitsubishi/Princeton University 60.00 %

Zernicke Moments Hanyang University 70.22 %

(Capitulation :-) IBM --.-- %

Wide range of applications...

... recognition of complex and arbitrary patterns


... results which are in accord with human perception

... parameter-free operation

Requirements

Robust automatic recognition of arbitrary shaped objectswhich is in accord with human visual perception

Industrial requirements...

... robustness

... low processing time

... applicable to three main tasks of recognition






Requirements



... robustness


Next:StrategyScaling (or resolution)

Rotation

Rigid / non-rigid deformation







Requirements


... robustness


... robustness




Simple Recognition (yes / no)

Common Rating (best of ...)

Analytical Rating (best of, but...)

Different Approaches

... Correlation

Pattern Matching...

Geometrical description...

... Hough – Transformation

Feature – Vectors...

... (Zernicke - ) Moments

Based on Visual Parts...

... Mokhtarian

... ASR

ASR: StrategyASR: Strategy

Source: 2D - Image

Arc – Matching

Contour – Segmentation

Contour Extraction

Object - Segmentation

Evolution

ASR: StrategyASR: Strategy

Arc – Matching

DCE

Contour – Segmentation

Contour Segmentation

Correspondence ?

Similarity of parts ?

Part Similarity

Similarity of parts ?

= Boundary Similarity Measure

= Similarity of polygons

Tangent Space

Transformation from image-space to tangent-space

Shape Comparison: Measure

Tangent space offers an intuitive measure:

T g0 sT g1s a2 ds 12 max l g0, l g1max

l g0l g1

,l g1l g 0

Shape Comparison: Measure

Drawback: not adaptive to unequally distributed noise if used globally !

…but works for single parts

Shape Comparison: Contour Segmentation

Shape Comparison: Correspondence

Optimal arc-correspondence:

find one to many (many to one) correspondence, that

minimizes the arc-measure !

next:Corr. -example

Graph of Correspondence

a0 a1 a2 a3

b0 b1 b2 b3

a0

b0

a1

a2a3

b1

b2b3

Graph:

... edge represents correspondence

... node represents matched arcs

arc

correspondence


Example:

a0 a1 a2 a3

b0 b1 b2 b3

a0

b0

a1

a2a3

b1

b2b3


Result:Optimal correspondence is given by cheapest way

next:Corr. - Results

Correspondence: Results

(MATLAB Demo)

Correspondence: Results

Correspondence and arc-measure allow...

... the identification of visual parts as well as

... the identification of the entire object

... a robust recognition of defective parts

... a shape matching which is in accord with human perception

ASR Results

Correspondence and arc-measure meet the requirements stated by Kimia et al.

Discrete

Easy computable

ASR Results

Problem:

Time Consuming Algorithm (~10ms on 2GHz Pentium)

No Metric (-> no clustering) !

How to build a database with this approach ?

Shape-Representation and Shape Similarity. Motivation WHY SHAPE ?

Documents