Scale Saliency: Applications in Visual Matching,Tracking and View-Based Object Recognition

Scale Saliency: Applications in Visual Matching, Tracking and

View-Based Object Recognition

Jonathon S. Hare and Paul H. Lewis

Intelligence, Agents, Multimedia GroupDepartment of Electronics and Computer Science

University of Southampton{jsh02r, phl}@ecs.soton.ac.uk

Topics of Discussion

• Introduction

• An overview of the scale-saliency algorithm

• Image Matching and View-Based Recognition

• Tracking

• Future Work

• Salient points or ‘interest-points’ have often been used for various vision tasks

• We have modified and extended an existing saliency algorithm and investigated its application to the tasks of matching, recognition and tracking

Introduction

Scale-Saliency

• Algorithm developed by Kadir & Brady:

• Based on previous work by Gilles

• Defines saliency in terms of local signal complexity or unpredictability, weighted by local self-similarity

• Keypoints are selected from peaks in entropy scale-space, thus the algorithm picks regions in (x, y, scale)

Scale-SaliencyLena R^3 Space

0 100

200 300

400 500

x location 0 50

100 150

200 250

300 350

400 450

y location

0

5

10

15

20

25

30

scale

Salient regions in the Lena picture The corresponding ℜ3 space

Scale-Saliency

• The original Thesis showed that the algorithm could be used for efficient tracking and matching using intensity pixel values as a descriptor

• We built on this using higher-order descriptors

• Histograms (intensity, RGB, HSI, etc)

• Possibilities for other feature invariants

Matching and View-Based Recognition

• Simple matching algorithm:

• Feature vectors compared using euclidean distance

• Distance(s) of closest features summed for each image to get overall distance

• Image with lowest overall distance is selected as the winning match

Matching and View-Based Recognition

Matching and View-Based Recognition

Matching and View-Based Recognition

Matching Example

• The tracking problem can be posed as a sub-image matching problem

• i.e. we want to track single salient regions across multiple frames

• This can be accomplished using the matching algorithm across consecutive frames

Tracking

• Tracking Examples

• Advantage of this method is that it directly captures changes in scale

Tracking

Tracking

• We have demonstrated a number of practical uses for the scale-saliency algorithm

• Using the scale-saliency algorithm we show how sparse representations of the image can be created and represented by features that are invariant to translation, rotation and scale change

Conclusions

• We have shown that this sparse representation can be used for image matching tasks

• The matching approach can be extended to enable view-based recognition and Content-Based Retrieval

• It is also possible to use the matching technique for tracking by applying it across consecutive video frames

Conclusions

• Future Work

• We are currently investigating how this work can be extended into a facility for recognition of art objects

• We are also researching methods for ensuring spatial consistency of salient points, without the use of graph-matching

Conclusions

Any Questions?