Cees Snoek (UvA) @ CMC Video Formats

VideoUnderstandingWhattoExpectTodayandTomorrow?

CeesSnoek

2

Today: Find the cat

3

How difficult is the problem?

Humanvisionconsumes50%brainpower…

Van Essen, Science 1992

4

Video recognition in a nutshell

Visualization by Jasper Schulte

5

 NIST TRECVID Benchmark

 Promote progress in video retrieval research

 Open big data, tasks, evaluation and innovation

International video competition

http://trecvid.nist.gov/

Aircraft

Beach Mountain

People marching

Police/Security

Flower

Concept detection task

6

Competition overview

From University-lab to spin-off and your mobile phone

• = 1000+ others

Universities win Start-ups win

Snoek et al., TRECVID 2004-2015

7

Latest jump due to deep learning 2006 2009 2015

Mea

n av

erag

e pr

ecis

ion

Progress in video recognition

8

Video search demo’s

Social media Forensics Cultural heritage

9

Qualcomm Zeroth provides on-device video recognition

10

Tomorrow: The Internet of things that video

11

Need to understand what is happening where and when?

Visualization by Jan van Gemert

12

Proposals facilitate complex representations

1. Generate action proposals

Ground truth Super-voxel segmentation Proposals from merged voxels

[Jain et al, CVPR14]

13

2. Encode video proposals as 15,000 object scores

Actions have object preference, relation is generic

15k 15k

[Jain et al, CVPR15]

AlexNet learned from 15K objects Objects make sense for actions

14

Enables understanding without video and action examples

3. Translate objects to actions

[Jain et al, ICCV15]

Prediction Ground truth

15

 Video recognition technology is ready for use, today

 Spatiotemporal video understanding is coming soon

Conclusion

www.ceessnoek.info