EgoMon Gaze and Video Dataset for Visual Saliency Prediction

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.

EgoMon Gaze and Video Dataset for Visual Saliency

Prediction

Mònica Chertó Sarret Supervised by: Cathal Gurrin and Xavier Giró


Outline

1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works

2


1. Introduction

3


Introduction. Main goals and project planning

4

Goals February March April May June

Construct the Dataset

Run state of the art saliency estimator with a single image

Frames extraction

Run saliency estimator with the extracted frames

Compare Results


Equipment and Software. Eye tracker, Tobii Glasses

5


Equipment and Software. Tobii studio Software

6


Equipment and Software.

7


Equipment and Software.

8


Publication

9

Repositori of Egocentric-saliency in GitHub [online] Available: https://github.com/imatge-upc/egocentric-saliencyEgoMon Dataset [online] Available: https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/

https://github.com/imatge-upc/egocentric-saliency

https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/

http://papers.nips.cc/paper/4824-imagenet-classification-w


Outline


10


2. State of the art

11

GTEA Dataset UT Ego Dataset

GTEA (Georgia Tech Egocentric Activities) – Gaze Dataset [online] Available: http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/UT (University of Texas) Ego Dataset [online] Available: http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html

http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/

http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html

http://papers.nips.cc/paper/4824-imagenet-classification-w


Outline


12


Acquisition. Calibration process of the Tobii Glasses

13

Video tutorial uploaded on YouTube.

https://www.youtube.com/watch?v=8QLuRJAgQIM

https://www.youtube.com/watch?v=8QLuRJAgQIM


Acquisition. Results of the calibration process of the Tobii Glasses

14


EgoMon Gaze & Video Dataset

15

...

7 x text files (gaze data)

7 x RAW (videos)

7 x Gaze (videos with the gaze information plotted)

13428 x frames extracted

75 x narrative images



16



17

INDOOR OUTDOOR


EgoMon. Oral Presentation

18


EgoMon. DCU and Albert College Park

19


EgoMon. Spanish Omelette

20


EgoMon. Playing cards

21


EgoMon. Botanic Gardens

22


EgoMon. Botanic Gardens (Narrative Clip)

23


EgoMon. Bus Ride

24


EgoMon. Walking to the Office

25


EgoMon. Privacy

26


EgoMon. Problems with the Gaze (Losses)

27

static

non-static


EgoMon. Processing, Eye Gaze data

28


EgoMon. Frame extraction

29

DURATION FRAMES EXTRACTED

TOTAL 3:43:41 13428

AVERAGE: 0:34:30 1918

1 fps


Outline


30


4. Visual Saliency Predictor.

31


Saliency Predictor. SalNet

32



33

...

7 x text files (gaze data)

7 x RAW (videos)

7 x Gaze (videos with the gaze information plotted)

13428 x frames extracted

75 x narrative images

...13428 x saliency models


Results of the Dataset

34


Quantitative Evaluation. Comparison Metric

35

Location-based Distribution-based

AUC-Judd, sAUC, NSS SIM, CC, EMD, KL

NORMALIZED SCANPATH SALIENCY

MIT Saliency Benchmark [online] Available: http://saliency.mit.edu/results_mit300.html

http://saliency.mit.edu/results_mit300.html


Results. Quantitative Evaluation

36


Results. Qualitative Evaluation

37

Example of GOOD results


Results. Qualitative Evaluation

38

Example of BAD results


Outline


39

Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 40

ConclusionsDataset Amount of Data Recorded

DeviceEnvironment Number of

participants

GTEA 17 sequences Tobii eye-tracker Glasses

Indoor 14

UT Ego 4 videos of 4 hours (16 h)

Looxcie wearable camera

Indoor + Outdoor 4

EgoMon 7 clean videos (4 h)7 gaze videos13428 extracted frames13428 saliency maps7 files with eye gaze data75 Narrative images

Tobii eye tracker glasses + Narrative Cip

Indoor + Outdoor 3


Future Works

Fine-tuning of saliency estimator based on the comparison metric

41


Publication

42

http://imatge-upc.github.io/egocentric-2016-saliency/



Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 43

EgoMon Gaze and Video Dataset for Visual Saliency Prediction

Technology

EgoMon Gaze and Video Dataset for Visual Saliency Prediction