The impact of visual saliency prediction in image classification

The Impact of Visual Saliency Prediction in Image Classification

1Eric Arazo Sánchez Kevin McGuinness Eva Mohedano Xavier Giró-i-Nieto

Advisors:

Introduction - Computer vision

2

ClassifierHandcrafted descriptors “guitar”

ClassifierLearned descriptors

Trainable

Trainable

Classical computer

vision

Deep Learning “guitar”

Introduction - Imagenet

3

Russakovsky, Olga, et al. “Imagenet large scale visual recognition challenge”. International Journal of Computer Vision (2015).

http://arxiv.org/abs/1409.0575

http://arxiv.org/abs/1409.0575

Imagenet

4

Images:

● 1.2 M train

● 50,000 test

● 1,000 categories

Evaluation dataset unpublished before the

competition

Imagenet

5

Metrics:

● Top-1 accuracy

● Top-5 accuracy

Imagenet

6

Metrics:

● Top-1 accuracy

● Top-5 accuracy


7

ILSVRC - Evolution since 2010

Slide credit: Kaiming He (FAIR)

http://kaiminghe.com/


8


Slide credit: Kaiming He (FAIR)

Some models have already reached

human-level performance.

Still the olympic games of computer

vision?



9Slide credit: Kaiming He (FAIR)

-9.4%2012

Introduction of the Convolutional Neural

Networks (CNN) in the competition with AlexNet



Introduction - AlexNet

10

Ref: Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. NIPS 2012.

http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks



Introduction - AlexNet

11

5 Convolutional

Layers

3 Fully Connected

Layers

1000 softmax

Object class

Introduction - CNN

12LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324.

http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf

Introduction - CNN

13LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324.

CNN are very useful in computer vision:

● Reduction of parameters (shared filters)

● Spatial coherence

http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf

Introduction - CNN

14

Image captioning Image segmentation

Introduction - CNN

15

Saliency prediction

Introduction - Saliency prediction

16

CNN model

Images

Saliency maps

Introduction - Saliency prediction

17

CNN for image classification

Objective

18

● Explore if saliency maps could improve other computer vision tasks

Objective

19

● Explore if saliency maps could improve computer vision tasks

Objective

20

● Explore if saliency maps could improve computer vision tasks

Outline● Introduction● Objective● State-of-the-art ● Methodology● Conclusions● Future work

21

State-of-the-art - Saliency prediction

22

SalNet

Pan, Junting and McGuinness, Kevin and Sayrol, Elisa and Giro-i-Nieto, Xavier and O'Connor, Noel E. Shallow and Deep Convolutional Networks for Saliency Prediction. CVPR 2016.

Trained on SALICON

https://github.com/imatge-upc/saliency-2016-cvpr

Saliency prediction

23

Application of saliency:

Saliency prediction

24


● In image retrieval

○ Finding the last appearance of an object.

Ref: Reyes, Cristian et al. Where is my Phone? Personal Object Retrieval from Egocentric Images (2016)

https://arxiv.org/pdf/1608.08139.pdf

Saliency prediction

25


● In image retrieval

○ Finding the last appearance of an object.

● Object recognition

○ Health care

Ref: Reyes, Cristian et al. Where is my Phone? Personal Object Retrieval from Egocentric Images (2016)

Ref: Pérez de San Roman, Philippe et al. Saliency Driven Object recognition in egocentric videos with deep CNN. 2016





Saliency prediction - our approach

26

Saliency prediction - our approach

27

AlexNet*SalNet


28

Methodology

29

RGB images

30

RGB images

RGB - The Baseline

31

RGB images

RGB - The Baseline

● 1.2 M images

● 227 x 227

● 1.2 M images

● 227 x 227

32

RGB images

RGB - The Baseline

9 days to train on computation

cluster

RGB - The Baseline

33

RGB - The Baseline

34

9 days

5 days

RGB - The Baseline

35

9 days

5 days

1.5 days

How to introduce saliency predictions?

36

Multiplication

Fan-in Network

Concatenation

37

AlexnetMultiplication

Fan-in Network

Concatenation

Alexnet


38

Multiplication

Fan-in Network

Concatenation

Alexnet

Alexnet


39

Multiplication

Fan-in Network

Concatenation

Alexnet

Alexnet

Alexnet

CNN


40

Multiplication

Fan-in Network

ConcatenationWhere?

Alexnet

Alexnet

Alexnet

CNN


41

Multiplication

Fan-in Network

Concatenation

Alexnet

Alexnet

Alexnet

CNN


42

Alexnet

Alexnet

Alexnet

CNN

Makes sense to use the baseline, which is already trained

Multiplication

Fan-in Network

Concatenation


43

Alexnet

Alexnet

Alexnet

CNN

Makes sense to use the baseline, which is already trained

Multiplication

Fan-in Network

Concatenation

Pre-trained CNN


Multiplication vs. Concatenation

44

Three strategies for each of them:


45


RGBS

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


46


RGB-1S-2SRGBS

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


47


RGBS RGB-1S-2S RGBS-1S-2S

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


48

RGBSRGBS

RGBS

RGB-1S-2S

RGBS-1S-2S

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


49

RGBSRGBS

RGBS

RGB-1S-2S

RGBS-1S-2S

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


50

RGB-1S-2S

RGBS

RGB-1S-2S

RGBS-1S-2S

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


51

RGB-1S-2S

RGBS

RGB-1S-2S

RGBS-1S-2S

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


52

RGBS-1S-2S

RGBS

RGB-1S-2S

RGBS-1S-2S

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


53

RGBS-1S-2S

RGBS

RGB-1S-2S

RGBS-1S-2S

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


54

The best option is concatenation:

● RGBS

● RGB-1S-2S

55

Multiplication

Fan-in Network

Concatenation


56

Multiplication

Fan-in Network

Concatenation


57

RGBS

RGB-1S-2S

Multiplication

Fan-in Network

Concatenation


58

RGBS

RGB-1S-2S

Multiplication

Fan-in Network

Concatenation


59

Alexnet

CNN

RGBS

RGB-1S-2S

Multiplication

Fan-in Network

Concatenation


60

Alexnet

CNN

RGBS

RGB-1S-2S

Multiplication

Fan-in Network

Concatenation

Where?


Fan-in architecture

61

Three strategies:

Fan-in C1.1

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency

Conv 1Batch Norm.Max-Pooling

Fan-in architecture

62

Three strategies:

Fan-in C1.1 Fan-in C2.1

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency



Fan-in architecture

63

Three strategies:

Fan-in C1.1 Fan-in C2.1 Fan-in C2

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency



Conv 1

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.Max-Pooling

Max-Pooling

RGBSaliency


Fan-in architecture

64

Fan-in C1.1

Fan-in C1.1

Fan-in C2.1

Fan-in C2

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


Fan-in architecture

65

Fan-in C1.1

Fan-in C1.1

Fan-in C2.1

Fan-in C2

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


Fan-in architecture

66

Fan-in C1.1

Fan-in C2.1

Fan-in C2

Fan-in C2.1

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency



Fan-in architecture

67

Fan-in C1.1

Fan-in C2.1

Fan-in C2

Fan-in C2.1

Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency



Fan-in architecture

68

Fan-in C1.1

Fan-in C2.1

Fan-in C2

Fan-in C2

Conv 1

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out


Max-Pooling

RGBSaliency


Fan-in architecture

69

Fan-in C1.1

Fan-in C2.1

Fan-in C2

Fan-in C2

Conv 1

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out


Max-Pooling

RGBSaliency


Fan-in architecture

70


● Fan-in C2.1

● Fan-in C2

Fan-in architecture

71


● Fan-in C2.1

● Fan-in C2

Surprising result for Fan-in C2 since it

has less parameters than the baseline

More experiments

12.4%

RGB-C2 (128x128)

72

Fan-in C2Fan-in Network

RGB-C2 (128x128)

73


RGB-C2 (128x128)

74

RGB-C2RGB (baseline)


75

RGB-C2 (128x128)

RGB (baseline)

Fan-in Network

RGB-C2

Fan-in C2

76

Multiplication

Fan-in Network

ConcatenationRGBS

RGB-1S-2S


77

Multiplication

Fan-in Network

ConcatenationRGBS

RGB-1S-2S

Fan-in C2.1

Fan-in C2


Analysis of per-class improvements

78

Fan-in C2.1

Fan-in C2

RGBS

RGB-1S-2S

Multiplication

Fan-in Network

Concatenation


79

Fan-in C2.1

Fan-in C2

RGBS

RGB-1S-2S

Multiplication

Fan-in Network

Concatenation


80

Class Increase of accuracy

Acoustic guitar

25 %

Volleyball 23 %

81

Analysis of per-class improvementsClass Increase of accuracy

Wrecker, tow car

-23 %

Entertainment center

-18 %


82

● CNNs trained to predict saliency maps can be used to improve other computer vision tasks such as image classification

83

Conclusions


84

Conclusions

Fan-in Network


85

Conclusions

Fan-in Network

● The best way to introduce the saliency maps to a CNN is with a Fan-in architecture, that provides freedom to the network to decide how to introduce the saliency maps

86

Conclusions


87

Conclusions

Fan-in C2.1Conv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency



Fan-in NetworkConcatenation

RGBSConv 1

Conv 2

Conv 3Conv 4Conv 5

FC 1

FC 1

FC 3 - Output

Drop Out

Drop Out

Batch Norm.

Batch Norm.

Max-Pooling

Max-Pooling

Max-Pooling

RGBSaliency


88

Conclusions

● The methodology of downsampling the images provides accurate results on the improvements of the CNN in larger images

89

Conclusions

227 x 227

128 x 128


90

Future work

91

● Several experiments:○ Fan-in:

■ Fan-in C2 without saliency maps

■ Concatenating instead of multiplying

○ Concatenation only in the first convolutional layer

○ Multiplication and training from scratch

● Once we have a reasonable model try with other saliency models

Future work

92

● Several experiments:○ Fan-in:

■ Fan-in C2 without saliency maps

■ Concatenating instead of multiplying

○ Concatenation only in the first convolutional layer

○ Multiplication and training from scratch

● Once we have a reasonable model try with other saliency models

Thank you

The impact of visual saliency prediction in image classification

Data & Analytics