EgoMon Gaze and Video Dataset for Visual Saliency Prediction

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon Gaze and Video Dataset for Visual Saliency

Prediction

Mònica Chertó Sarret Supervised by: Cathal Gurrin and Xavier Giró

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Outline

1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works

2

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

1. Introduction

3

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Introduction. Main goals and project planning

4

Goals February March April May June

Construct the Dataset

Run state of the art saliency estimator with a single image

Frames extraction

Run saliency estimator with the extracted frames

Compare Results

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Equipment and Software. Eye tracker, Tobii Glasses

5

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Equipment and Software. Tobii studio Software

6

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Equipment and Software.

7

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Equipment and Software.

8

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Publication

9

Repositori of Egocentric-saliency in GitHub [online] Available: https://github.com/imatge-upc/egocentric-saliencyEgoMon Dataset [online] Available: https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Outline

1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works

10

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

2. State of the art

11

GTEA Dataset UT Ego Dataset

GTEA (Georgia Tech Egocentric Activities) – Gaze Dataset [online] Available: http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/UT (University of Texas) Ego Dataset [online] Available: http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Outline

1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works

12

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Acquisition. Calibration process of the Tobii Glasses

13

Video tutorial uploaded on YouTube.

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Acquisition. Results of the calibration process of the Tobii Glasses

14

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon Gaze & Video Dataset

15

...

7 x text files (gaze data)

7 x RAW (videos)

7 x Gaze (videos with the gaze information plotted)

13428 x frames extracted

75 x narrative images

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon Gaze & Video Dataset

16

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon Gaze & Video Dataset

17

INDOOR OUTDOOR

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Oral Presentation

18

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. DCU and Albert College Park

19

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Spanish Omelette

20

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Playing cards

21

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Botanic Gardens

22

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Botanic Gardens (Narrative Clip)

23

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Bus Ride

24

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Walking to the Office

25

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Privacy

26

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Problems with the Gaze (Losses)

27

static

non-static

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Processing, Eye Gaze data

28

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon. Frame extraction

29

DURATION FRAMES EXTRACTED

TOTAL 3:43:41 13428

AVERAGE: 0:34:30 1918

1 fps

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Outline

1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works

30

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

4. Visual Saliency Predictor.

31

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Saliency Predictor. SalNet

32

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon Gaze & Video Dataset

33

...

7 x text files (gaze data)

7 x RAW (videos)

7 x Gaze (videos with the gaze information plotted)

13428 x frames extracted

75 x narrative images

...13428 x saliency models

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Results of the Dataset

34

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Quantitative Evaluation. Comparison Metric

35

Location-based Distribution-based

AUC-Judd, sAUC, NSS SIM, CC, EMD, KL

NORMALIZED SCANPATH SALIENCY

MIT Saliency Benchmark [online] Available: http://saliency.mit.edu/results_mit300.html

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Results. Quantitative Evaluation

36

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Results. Qualitative Evaluation

37

Example of GOOD results

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Results. Qualitative Evaluation

38

Example of BAD results

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Outline

1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works

39

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 40

ConclusionsDataset Amount of Data Recorded

DeviceEnvironment Number of

participants

GTEA 17 sequences Tobii eye-tracker Glasses

Indoor 14

UT Ego 4 videos of 4 hours (16 h)

Looxcie wearable camera

Indoor + Outdoor 4

EgoMon 7 clean videos (4 h)7 gaze videos13428 extracted frames13428 saliency maps7 files with eye gaze data75 Narrative images

Tobii eye tracker glasses + Narrative Cip

Indoor + Outdoor 3

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Future Works

Fine-tuning of saliency estimator based on the comparison metric

41

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

Publication

42

http://imatge-upc.github.io/egocentric-2016-saliency/

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 43