Human-Centric Multimedia Research: Research Opportunities Nuria Oliver, PhD Telefonica Research Multimedia and Data Mining & User Modeling Scientific Director.

Human-Centric Multimedia Research:

Research Opportunities

Nuria Oliver, PhDTelefonica Research

Multimedia and Data Mining & User Modeling Scientific Director

Explosion of Digital Data

6-Fold Growth in Four Years

Information Created, Captured and Replicated

2006161 Exabytes

2010988 Exabytes

Source: IDC, 2007

Exabytes?

Book

Photo

Movie

Print CollectionUS Library of Congress

All printed material (multimedia)

All words ever spoken by human beings (5 exa)

Who will create all this data?

User* Generated

Content692 Exabytes

User* Generated

Content692 Exabytes

Organizational** Touch

Content296 Exabytes

*Consumers and Workerscreating, capturing or replicatingpersonal informaton

**Transported, hostedmanaged or secured

2010 Data988 Exabytes

563 Exabytes

Human-Centric Multimedia

Prosumer

On-the-go

Access

Cont

ext &

Co

nten

t Social

Produce

Search & Discovery Consume

Media, tags, ratings, comments

MediaMedia

A FEW RESEARCH CHALLENGES ANDOPPORTUNITIES

Multi-Modality: Content + contextMulti-modal approaches are needed to

construct novel methodologies to fuse multi-modal content and context information

Multi-modal Multimedia Content Analysis:Feature ExtractionSimilarity Metrics

Ontology definitionIndexing schemes….

Multimedia Context:Higher level knowledge

generated by users (tags, comments…)

User interaction data Wisdom of the crowd

Fusion of content and context-based featuresCreation of large collections of labeled training data

Noise filtering by aggregation of contextual information

Improved search results

Multimedia Tagging

Paris

Eiffel Tower

France Vacation

June 2009

Multimedia Tagging• User generated content is rarely annotated really

difficult, if not impossible to later find it • When annotated, it is typically done in batch, per

session, not per item• Tags significantly improve search results alone or

combined with content-based techniques• Need for novel interfaces to encourage users to

annotate content– Games with a Purpose– Annotations at the time of capture

• More research on tag expansion and automatic tagging

Multimedia Information Overload

Multimedia Information Overload

• Retrieval accuracy is not sufficient due to vast amounts of available information too many relevant results

• New orthogonal dimensions need to be used to extend the notion of relevance and improve retrieval performance e.g., aesthetics

• User generated content is of varying quality• Need for user centric approaches• Multi-disciplinary approaches: – Computer scientists, psychologists, human-computer

interaction researchers– Computer Vision, Pattern Recognition, Machine Learning,

Human Perception, Human Activity Recognition

Video and Audio Feature extractionSignature Creation

Duplicate Detection

Example: Near-Duplicate Videos

How do users perceive near-duplicate videos?

Do they care about them? Which features are important

when defining near-duplicates?

M. Cherubini, R. de Oliveira, and N. Oliver, “Understanding near-duplicate videos,” in Proceedings of ACM MM’09, (Beijing, China), pp. 35–44, ACM Press, October 19-24 2009.

Example: Multimedia Aesthetics

“The interest that a photograph, video or audio piece generates when perceived by human observers, and that incorporates both objective and subjective factors”

The Importance of Aesthetics

“Paris Louvre Night”

The Importance of Aesthetics

• User generated content has a wide range of quality and aesthetic value for the same content

• Aesthetics influence our perception of content• Highly disregarded in state-of-the-art multimedia retrieval

systems• Need for computational models of the aesthetic value of

multimedia content• Need for ground truth databases on image, audio and video

aesthetics• Need for deeper understanding of– The role of aesthetics on user preferences and satisfaction– Universal vs personal aesthetics– Domain-dependent aesthetics– Quality vs aesthetics

Personalization, Recommendation and Exploratory Search

Personalization, Recommendation and Exploratory Search

• Future multimedia search and retrieval systems will need to take into account the user’s preferences, interests and task at hand in order to return relevant content

• Huge amounts of multimedia data – Need for recommendations rather than direct search– Automatic discovery of relevant information to the users

• More research should be devoted to user modeling, personalization and recommendations of multimedia content

• Untapped research challenge: Role that the task at hand plays in determining the optimal multimedia content to retrieve for the user

Multimedia Storytelling

“The conveying of events with words, images and sounds, often with embellishement. “Stories or narratives have been shared in every culture and in every land as a means of entertainment, education, preservation of culture and in order to instill moral values.

Crucial elements of stories and storytelling include plot and characters, as well as the narrative point of view.

Multimedia Storytelling• Despite capturing large amounts of digital multimedia

content, most users rarely access the content again• Sharing the multimedia content is one of the main

reasons why users capture it• Lack of efficient and scalable tools for browsing, finding

and selecting the desired content• Multimedia Storytelling: User-friendly, semi-automatic

and scalable (space and time) multimedia tools that enable users to – Easily retrieve desired multimedia content – Create and share the story they want to create from their

content

Exemplary Workflow for MM Storytelling

StorytellingAlgorithms

Multimedia Analysis Tools

Create Story Slideshow:“Madrid Christmas 2009”

StorytellingUIFlickr e-mail communication

With story slideshow

Face Detection Face RecognitionSmile DetectionAesthetics RerankingClusteringNear-duplicate DetectionTag expansionAutomatic Classification

Identify main ActorsIdentify main ChaptersComplement contentwith external contentSelect images based on *Story Length *Target Audience *Target Device

New Multimedia Experiences

New Multimedia Experiences

• Users are increasingly seeking new ways to experience multimedia content

• Research opportunities combining:– Music + Video: High-quality visual musical experiences– Video + 3D reconstruction: 3D video– Images + Context : Mobile Augmented Reality

• Research challenges in:– Multimedia analysis: Machine learning, pattern

recognition, computer vision– User Modeling– Human-computer interaction

http://research.tid.es/multimedia

[email protected]