Hands On: Multimedia Methods for Large Scale Video ...fractor/fall2012/cs294-2-2012.pdfHands On: Multimedia Methods for Large Scale Video Analysis (Project Meeting) Dr. Gerald Friedland,

$Page 1: Hands On: Multimedia Methods for Large Scale Video ...fractor/fall2012/cs294-2-2012.pdfHands On: Multimedia Methods for Large Scale Video Analysis (Project Meeting) Dr. Gerald Friedland,$
Hands On: Multimedia Methods for Large Scale Video Analysis(Project Meeting)

Dr. Gerald Friedland, [email protected]

1

Wednesday, August 29, 12

mailto:[email protected]


Today


Today

• Project Requirements


Today

• Project Requirements• Data available


Today

• Project Requirements• Data available• Compute Architecture at ICSI


Today

• Project Requirements• Data available• Compute Architecture at ICSI• Some Project Ideas


Project Requirements

3



• Form a team of 2 to 3 people.

3



• Form a team of 2 to 3 people.• Each team work on one project idea

3



• Form a team of 2 to 3 people.• Each team work on one project idea• Project must be on big multimedia

data, e.g. at least ten-thousands of videos, hundred-thousands of sounds pieces/images.

3



• Form a team of 2 to 3 people.• Each team work on one project idea• Project must be on big multimedia

data, e.g. at least ten-thousands of videos, hundred-thousands of sounds pieces/images.

• Team delivers written project report at the end of semester, reports on progress during the semester

3


Each Project Report...

4



• ...needs to explain the idea of the project and show evidence that the project has been performed.

4




• ...needs to report on

4




• ...needs to report on – accuracy

4




• ...needs to report on – accuracy– efficiency

4




• ...needs to report on – accuracy– efficiency– scalability

4




• ...needs to report on – accuracy– efficiency– scalability– limits of the approach

4




• ...needs to report on – accuracy– efficiency– scalability– limits of the approach

• ... and reports on problems occured.4


Project Resources

5


Project Resources• Data:

5



– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)

5




– Use your own (after discussion with instructor)

5





• Compute:

5





• Compute:– Use your own laptop initially then

5





• Compute:– Use your own laptop initially then– Use ICSI’s compute pool then

5





• Compute:– Use your own laptop initially then– Use ICSI’s compute pool then– if needed, use Amazon EC2

5





• Compute:– Use your own laptop initially then– Use ICSI’s compute pool then– if needed, use Amazon EC2– Use any other compute resource that you

have access to. 5


Project Resources II

6



• Hard disk:

6



• Hard disk:– Once in the ICSI phase use ICSI’s ttmp

structure

6




structure– At Amazon: Need to buy space if needed.

6





• Time:

6





• Time:– Start work on your project as early as

possible!

6






possible!

6






possible!

6






possible!

6






possible!

•

6


Project Idea

7


Project Idea

• Come up with your own project idea as a team, inspired by the class content, co-students, the data, and/or other input.

7


Project Idea

• Come up with your own project idea as a team, inspired by the class content, co-students, the data, and/or other input.

• Discuss the project idea with the class and the instructor

7


Available BIGDATA

8


Available BIGDATA

• MediaEval 2012

8


Available BIGDATA

• MediaEval 2012• TrecVid MED 2011

8


Available BIGDATA

• MediaEval 2012• TrecVid MED 2011• 1M Songs

8


Placing TaskAutomatically guess the location of a Flickr video:• i.e., assign geo-coordinates (latitude and

longitude)• Using one or more of:

– Visual/Audio content– Metadata (title, tags, description, etc)– Social information 9


Data Description (2010)

• Training Data– 15k videos/metadata/visual

keyframes (+features)/geo-tags– 6M photos/metadata/visual

features• Test Data

– 5k video/metadata/visual keyframes (+features)

– no geotags• Test/Training split: by UserID

10


Example

11


A ‘Good’ video

14

• Title: CIMG0254• Keywords: southamerica, june, 2008, video, pearce, vacation,

iguazufalls, iguassufalls, iguaçufalls, waterfall, argentina• Description: (None) Wednesday, August 29, 12

How about this one?

15

• Title: MVI_6423_rau• Keywords: usa08, puertorico• Description: (None)


Another ‘bad’ video

16

• Title: Stillness #1• Keywords: henrywcoestatepark, henrycoe, california, 2009,

hiking, landscape, nature, chinaholetrail, video• Description: When I hike, I like to stop at random spots and just

stand still for a few minutes and listen and look at my surroundings. On this hike, I decided to record a few seconds during those moments.


Metadata

• 98.8% of videos were annotated with at least one title, tags, or description

17


TrecVID MED 2011 detailed

16



16

• Videos all “consumer produced” , typically 1-5 minutes long



16


• Given 15 concepts, 5 for training, 10 for eval



16



• About 100 sample videos per concept



16



• About 100 sample videos per concept• Testset 2011: 50k videos, open set


TrecVID Dataset

17

Consumer-Produced, Unfiltered Videos...Wednesday, August 29, 12

What is Video Concept Detection?

18



18

• A concept (as of TrecVID MED 11):



18

• A concept (as of TrecVID MED 11):– is a complex activity occurring at a specific place and

time;



18


time;– involves people interacting with other people and/or

objects;



18



objects;– consists of a number of human actions, processes,

and activities that are loosely or tightly organized and that have significant temporal and semantic relationships to the overarching activity;



18



objects;– consists of a number of human actions, processes,

and activities that are loosely or tightly organized and that have significant temporal and semantic relationships to the overarching activity;

– is directly observable.


Event Kit?

19


Event Kit?

19

• An Event Kit consists of:


Event Kit?

19

• An Event Kit consists of:– A textual (natural language) event definition


Event Kit?

19

• An Event Kit consists of:– A textual (natural language) event definition – A textual (NL) event explication, which is a

glossart for the definition


Event Kit?

19


glossart for the definition– Evidential description: A textual listing of

attributes that are indicative of an event instance.


Event Kit?

19


glossart for the definition– Evidential description: A textual listing of

attributes that are indicative of an event instance.

– A set of illustrative video examples each containing an instance of the event.


Event Kit: Example

20

Event name: Attempting a board trick

Definition: One or more people attempt to do a trick on a skateboard,snowboard, surfboard, or other boardsport board.

Explication: Boardsports are sports where a person stands, sits, or lays on a board and moves and controls the board. Tricks consist of intentional motions made with the board that are not simply slowing down/stopping the board or steering the board as it moves. Steering around obstacles or steering a board off of a jump and landing on the ground are not considered tricks in and of themselves.

Common tricks involve actions like sliding the board along the top of anobject (e.g. a swimming pool rim or railing), jumping from the ground orthe surface of water into the air, and spinning or flipping in the air.

Evidential description:

scene: outside, often in a skatepark

objects/people: skateboard, snowboard, surfboard, ramps, rails, safety gear, crowds

activities: standing, sitting or laying on the board; jumping with the board; flipping the board and landing on it; spinning the board; sliding the board across various objects.

audio: sounds of board hitting surface during trick; crowd cheeringWednesday, August 29, 12

2011 Concepts

21

Event Category Train DevTest E001 Board Tricks 160 111E002 Feeding Animal 160 111E003 Landing a Fish 122 86E004 Wedding 128 88E005 Woodworking 142 100E006 Birthday Party 173 0E007 Changing Tire 110 0E008 Flash Mob 173 0E009 Vehicle Unstuck 131 0E010 Grooming animal 136 0E011 Make a Sandwich 124 0E012 Parade 134 0E013 Parkour 108 0E014 Repairing Appliance 123 0E015 Sewing 116 0Other Random other N/A 3755


Sample Video 1: “Board Tricks”

22


Sample Video 2: “Board Tricks”

23


Test Video: “Board Tricks”

24


Test Video: “Board Tricks”

24

NOT A POSITIVE!


25


25

The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.


25

The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.Its purposes are:


25

The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.Its purposes are:• To encourage research on algorithms that scale to

commercial sizes


25


commercial sizes• To provide a reference dataset for evaluating research


25


commercial sizes• To provide a reference dataset for evaluating research• As a shortcut alternative to creating a large dataset with APIs

(e.g. The Echo Nest's)


25


commercial sizes• To provide a reference dataset for evaluating research• As a shortcut alternative to creating a large dataset with APIs

(e.g. The Echo Nest's)• To help new researchers get started in the MIR field


26


26

• Dataset is a sqllite database of about 100 attributes for each song.


26


• Subset of 10k song attributes available


26


• Subset of 10k song attributes available• The song IS NOT part of the database.

Need to access using a different database using metadata (e.g. Grooveshark).


27


Compute Architecture at ICSI



• Each enrolled student gets account at ICSI



• Each enrolled student gets account at ICSI

• ICSI = International Computer Science Institute:http://www.icsi.berkeley.eduLocation: 1947 Center Street, 6th floor


http://www.icsi.berkeley.edu

http://www.icsi.berkeley.edu




• Accounts belong to “speech group” and allow access to Unix compute cluster of that group




• Compute Cluster is currently 160 CPUs




• Compute Cluster is currently 160 CPUs

• Main compute: “Squids” 8x8 CPUs, 8x2 GPUs





• Most important page for technical Info:https://speechwiki.icsi.berkeley.edu/speechwiki/index.php/Main_Page


https://speechwiki.icsi.berkeley.edu/speechwiki/index.php/Main_Page






• Includes “Ganglia Monitoring”






















Compute Pool


Compute Pool• Main usage policy: “Be nice to

each other”



each other”• If you want to start a job using

more than 16 CPUs, please send email to: [email protected].







• Never login into a compute machine to run jobs!







• Never login into a compute machine to run jobs!

• If in doubt or trouble: Send me email.




ICSI Compute Pool: Basic Usage

Easy compared to Amazon!



• showjobs -- Shows job currently running





• run-command -- Starts a job





• run-command -- Starts a job• ssh machine /bin/kill -1 -- -pid -- Kill a job



ICSI Compute Pool: Data Storage



• Home/Project directory (backed up, strictly quota’d)




• Local scratch space (fast, local machine, not backed up)





• Networked scratch space (not backed up)





• Networked scratch space (not backed up)

• Temporary space (so-called ttmp, networked, deleted automatically)


ICSI Compute Pool: Availabilty

There are always other jobs...Wednesday, August 29, 12

Some Project Ideas (MediaEval Dataset)



• Build the “any tag” detector



• Build the “any tag” detector• City-scale location estimation



• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector



• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector• Correlate Users among videos



• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector• Correlate Users among videos• Find videos from a certain country



• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector• Correlate Users among videos• Find videos from a certain country• Correlate video quality with tag

quality


Some Project Ideas (TrecVid Dataset)



• Build a simple TrecVid MED system!




• Build a keyword-spotter




• Build a keyword-spotter• Build a face detector




• Build a keyword-spotter• Build a face detector• Sort videos by visual similarity










Some Project Ideas (1M Song Dataset)



• Build a dubbed-song recognition system for the TrecVid MED set




• Correlate song similarity with user ratings





• Cluster and sort songs by tags and correlate with acoustic clustering






• Try to align lyrics with songs






• Try to align lyrics with songs


This Week (Lecture)

38

•Architectural Considerations for Large Scale Conten Analysis


Next Week (Project Meeting)

39

• Amazon EC2 and how to use it


Hands On: Multimedia Methods for Large Scale Video ...fractor/fall2012/cs294-2-2012.pdfHands On: Multimedia Methods for Large Scale Video Analysis (Project Meeting) Dr. Gerald Friedland,

Documents