MediaEval 2016 - Ranking Images and Videos on Visual Interestingness by Visual Sentiment Features

1

Soheil  Rayatdoost,  Mohammad  SoleymaniSwiss  Center  for  Affective  Sciences

University  of  GenevaSwitzerland

Ranking  Images  and  Videos  on  Visual  Interestingness  by  Visual  Sentiment  Features

What  makes  an  video-­image  interesting• Novelty

• Uncertainty

• Conflict

• Complexity

• Comprehensibility

• Familiarity  

• Emotional  content

22/8

Automatic  detection-­ Features:• Visual• Visual sentiment adjective noun-­pairs• Deep learning (fc7)

• Audio:• The extended Geneva Minimalistic Acoustic Parameter(eGeMAPS)

-­ Regression models:• Linear regression• Support vector regression (SVR)• Sparse approximation weighted regression (SPARROW)

-­ A Principal Component Analysis (PCA) for dimensionality reduction

33/8

Our  Method  for  video  sub  challenge

4

Shots …

MVSO

Feature  Extraction

SVR-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐SPARROW

Regression

OpenSMILE

Video

Audio

Ranking

Frames

Visual  sentiment  descriptors

Deep  learning  features

eGeMAPS

4/8

Our  Method  for  image  sub  challenge

5

Key  frame

Linear  Regression

MVSO

Visual  sentiment  descriptors

Deep  learning  features

Feature  ExtractionRegression

Ranking

5/8

Results

6

Data Task Method Features MAP  ↑Image LR MVSO+fc7 0.1710Video SPARROW MVSO+fc7 0.2617Video SPARROW Baseline 0.2414Video SPARROW eGeMAPS 0.1987Image LR MVSO+fc7 0.1704Video SPARROW MVSO+fc7 0.1710Video SPARROW Baseline 0.1497Video SPARROW eGeMAPS 0.1367

Dev.  Set

Eval.  Set

6/8

Conclusions  and  future  works

• Mid-­level  visual  descriptors          Affective  nature  of  interestingness

• Our  features  are  all  static  and  frame-­based

• Temporal  information     The  small  size  of  the  dataset

• Larger  scale  datasets           Sophisticated  methods

• Audio  features

77/8

8

Thanks