Elastic Edge Boxes for Object Proposal on RGB-D Images202.119.32.195/cache/3/03/software.nju.edu.cn/a4e1c5724645298f07b6b21... · NANJING UNIVERSITY Multimedia AnalyzinG and UnderStanding

Elastic Edge Boxes for Object

Proposal on RGB-D Images

Jing Liu, Tongwei Ren, Jia Bei

Nanjing University

January 5, 2016

MMM 2016, Paper ID: 86

Multimedia AnalyzinG

and UnderStanding

MAGUS

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSOutline

• Motivation

• Elastic Edge Boxes Method

• Experiments

• Conclusion

2

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSObject Proposal

3

Object detection Image segmentation Image retrieval

• Aims to detect bounding box which possibly contains

class-independent objects in an image

hit

miss

• Applications

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUS

• High recall

• High efficiency

• High accuracy

• Low intersection over union (IoU) is not enough

Object Proposal is Challenging

4

IoU = 0.5

IoU = 0.8

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSCurrent Methods

5

• Generate a pool of boxes and score the boxes

• Efficient but not accurate enough

• Over-segment images and merge the segments

• Accurate but not efficient enough

[Uijlings et. al, IJCV13][Cheng et. al, CVPR14]

How to combine these two strategies to obtain good performancein both efficiency and accuracy?

Window scoring Grouping

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSOutline

• Motivation

• Elastic Edge Boxes Method

• Experiments

• Conclusion

6

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSOverview

Initial boxes generation

Elastic range search

Bounding box adjustment

7

Elastic edge box for RGB-D object proposal

Step 1

resultRGB channels and depth channel

Step 2

Step 3

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSInitial Boxes Generation

8

• Perform sliding window to sample boxes• Calculate score by contours wholly enclosed

in a box• Utilize edge boxes method [Dolla ́r et. al, ECCV 14]

Initial boxes generation

Edge detection result Initial boxes

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUS

Initial boxes generation

Elastic Range Search

9

• Super-pixels straddling the box are elastic

range

• Use Super-pixels wholly included in the box to

represent object (cyan)

• Use super-pixels adjacent to elastic range of

similar sum as object part to represent

background (blue)

Elastic range (yellow super-pixels)

Elastic range search

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSBounding Box Adjustment

• Compute color distance, spatial distance

and depth distance as similar

measurement

• Only super-pixels more similar to object

than background in both RGB and depth

channels will be assigned to object

10

Bounding box adjustment

Adjusted bounding box (red box)Decision

Initial boxes generation

Elastic range search

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSOutline

• Motivation

• Elastic Edge Boxes Method

• Experiments

• Conclusion

11

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSDataset

• Improvement

• More balanced

• 300 images in each group (2, 3, 4,

5, 5+ objects, respectively)

• Higher average object number

• PASCAL VOC 2012: 2.38

• Stereo objectness: 2.98

• NJU1500: 4.22

• NJU1500: 1,500 stereo images for object proposal

• Extend from stereo objectness dataset [Xu et. al, ICME15]

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSResult

• Suitable for various images under high IoU

• Challenging situations

13

0.843

0.878

0.875

0.817

0.852

0.7860.876

Obscure(sword)

Small(dustbin)

Occluded(Papa Smurf)

0.845

0.903 0.796 0.825

0.83

0.79

0.738

0.876

0.829

0.8540.802

0.825

0.860.889

0.854

0.694

0.795

0.928

0.826

0.846

0.67

0.76

0.832

0.876

0.791

0.857

0.852

0.821

0.846

0.831

0.95

0.845

0.903 0.796 0.825

0.83

0.79

0.794

0.789

0.941

0.8440.708

0.893

0.802

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUS

• Compare with eight state-of-the-art methods

• Including AIDC, BING, EB, OBJ, GOP, MCG, SS and MEB

• Under IoU = 0.5 and IoU = 0.8, respectively

Comparison

14

IoU = 0.5 IoU = 0.8

Comparable to other methods when IoU = 0.5

Better than other methods when IoU = 0.8

Ours5.78s per image

MCG60.12s per image

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSOutline

• Motivation

• Elastic Edge Boxes Method

• Experiments

• Conclusion

15

NANJING UNIVERSITY

Multimedia AnalyzinG

and UnderStanding

MAGUSConclusion

• Contribution

• First attempt to integrate window scoring and

grouping strategies for RGB-D object proposal

• Provide an RGB-D image dataset NJU1500 for object

proposal

• Future work

• Object proposal for video analysis

• Usage of object proposal in multimedia applications

16

17

Thank You

Email: [email protected]

Multimedia AnalyzinG

and UnderStanding

MAGUS