Saliency Detection based on a Sparsity of Segments Modelavik/PPTFiles/icvgip.pdf · 2017-01-25 · Vision and Image Processing Lab, Indian Institute of Technology Bombay, India Avik

Vision and Image Processing Lab, Indian Institute of Technology Bombay, India

Avik Hati, Subhasis Chaudhuri, Rajbabu [email protected], [email protected], [email protected]

Saliency Detection based on a Sparsity of Segments Model

Introduction

• Saliency: measure of importance of objects in images

• Salient object

captures attention

distinctive in features

• Applications

Object segmentation

Video summarization

Content based image compression

Image and video quality assessment

• Only the important parts are processed

• Reduction in complexity

Typical Methods

• Local approach (center-surround window)

Pixel based [Harel et al., Itti et al.]

Boundary of large smooth salient object is detected

• Global approach (global comparison)

Superpixel based [Perazzi et al.]

Patch based [Margolin et al., Yan et al.]

Salient objects are not uniformly highlighted

• Frequency domain [Guo et al., Hou et al.]

Problem Definition

• Object based approach• Obtain a saliency map of an image such that

pixels within an object to have equal saliency salient objects to be extracted completely retain exact boundary of the salient object number of object segmentation to be small

Texture Removal

Gaussian Mixture Model

• EM algorithm assuming GMM• Minimize to find optimum number of Gaussians, K

: Gaussian pmfs: mixing proportions

• Number of segments is K

Input image Histogram Texture-removedimage

Histogram

TV : Total Variation: Weights: Texture-removed piecewise constant image: Regularization parameter

i

g

ke

i

( )if x

255

0 1

( ) ( )

arg min

k

k i i

x i

kk

e hist x f x

K e

Segmentation

• Obtain thresholds by minimizing probability of misclassification

, ,

Saliency Computation

• Color saliency

: area of region

• Spatial saliency

: spatial std. dev. of region

Salient Object Extraction

• Retain exact object boundaries

• Image Matting [Levin et al.]

• Eroded saliency output as foreground scribble

• Dilated saliency output as background scribble

Saliency Results

Challenges

• Choice of regularizer constant Large value removes small objects

• Texture-removed image may not always facilitate meaningful segmentation

Over smoothing Combination of Y, Cb, Cr

components

Proposed saliency maps in (e) are complete and without textures, holes, unlike (b-d). Comparison of precision, recall and F-measure of SF, RC, CA with the proposed method (Our) on the MSRA dataset. (see paper for details)

i

Texture-removed image Segmented regions (K=3)GMM (K=3)

1

01

( ) ( )i K

i

K

e i i i

i

P f x dx f x dx

0 0 255K

iA rir

Segmentationresult

Color saliencyoutput (thresholded)

Spatial saliency

Input image-2 Segmentation

Color saliency

1,2,...,

( ) 1max

j

s

j j K

s j

j

Inputimage

Saliencyoutput

Salient object

Inputimage

(a)

Chenget al.(b)

Perazziet al.(d)

Proposed(e)

Gofermanet al.

(c)

Multiple salient objects

Salient region that can be overlooked by presence of known objects

Very small salient objects

2( )

, ,

min i

i TVFgi Y Cb Cr

g I g

1

( ) ( ) ( , )K

c i c j i

i

s j r d r r

Precision-Recall

Salient object: Butterfly

Salient object: Green apple

jr