SalGAN: Visual Saliency Prediction with Generative Adversarial Networks

1

SalGAN: Visual Saliency Prediction with Generative Adversarial Networks

Junting Pan Cristian Canton K.McGuinness Noel E. O’Connor Jordi Torres Elisa Sayrol Xavier Giró

2

Saliency?

3

Saliency Prediction

4

MODEL ARCHITECTURE

ARCHITECTURE OF GENERATOR

5

The encoder is initialized with VGG-16, and we do fine tuning of the last two groups of Conv Layers

The decoder is initialized randomly, the last Conv Layer have tanh nonlinearities and the output layer consist in a Conv Layer of kernel size 1x1 with sigmoid activation.

Then according to the post about GAN model we applied the loss function with smaller saliency maps

6

SALGAN-GAN: Downsample saliency map

[Inspiration from this blog post]

Compare (BCE)

DownsampledGenerated

Saliency Map

DownsampledGround Truth Saliency Map

SALICON VALIDATION

7

SALGAN: Downsample saliency map

8

APPLYING GANGAN Training showing saliency + image

9

APPLYING GAN - Model Selection

SALICON validation set accuracy metrics for GAN+BCE vs BCE on varying numbers of epochs.

10

APPLYING GAN - Model Selection

SALICON validation set Information Gain for different hyper parameter α on varying numbers of epochs

11

RESULTS

Qualitative Results

12

GroundTruth BCE SALGAN

Qualitative Results

13

GroundTruth BCE SALGAN

Qualitative Results- Failure case

14

GroundTruth BCE SALGAN

15

Quantitative Results - SALICON TEST / MIT300