Arcomem training multimedia

ARCOMEM Training Material

Understanding Images and Video

ARCOMEM Training Material

Introduction

• The web is increasingly multimedia in nature• We concentrate on the multimedia aspects• Image and video content analysis, indexing,

matching and annotation• Exploit the web, social web and linked data

web

ARCOMEM Training Material

Exploring image and video reuse on the Web

ARCOMEM Training Material

• Goal– Use image analysis techniques to aggregate social contexts; in

particular to interlink the social discussion of events and topics through the content of images and video.

• Motivation– Media is often reused or reposted on social networks.

– Detection of near-duplicate multimedia artifacts provides a means to investigate and explore many facets about the documents the media is embedded within. • Some examples include:

– Aggregating documents about the same subject/event/opinion» Finding cases where media is used in differing contexts is also interesting.

– Exploring how different social groups talk about the same media

ARCOMEM Training Material

ARCOMEM Training Material

ARCOMEM US-Elections Crawl Examples

ARCOMEM Training Material

Images

• image/gif 4371• image/jpeg 667073• image/png 46644• image/svg+xml 1• image/x-icon 2• image/x-ms-bmp 167

ARCOMEM Training Material

122 Detected Duplicates

https://sphotos-a.xx.fbcdn.net/hphotos-ash3/63089_10151299959267177_1113898282_n.jpghttp://p.twimg.com/A7Oq4JRCEAA05uG.jpghttps://p.twimg.com/A7RMKgrCIAAvPdj.jpghttp://24.media.tumblr.com/tumblr_md4ttulK8d1qa5ex8o1_500.jpghttps://p.twimg.com/A7Oq4JRCEAA05uG.jpghttps://p.twimg.com/A7JYdoDCEAAktAb.jpghttps://p.twimg.com/A7JIdp7CcAAjsPF.jpghttp://p.twimg.com/A7JSUYCCUAEzW5O.jpghttp://p.twimg.com/A7NoTNOCIAEQ3ie.jpghttps://p.twimg.com/A7LsnVrCQAAWYFA.jpg...

112 Detected Duplicates

http://25.media.tumblr.com/tumblr_mc0mmx2kTA1rxxq3ro1_500.pnghttp://p.twimg.com/A7Ep0CjCMAAon-f.jpghttp://p.twimg.com/A7N3lxrCAAE_gzP.jpghttps://p.twimg.com/A5ctYtVCMAE8_8e.jpghttp://p.twimg.com/A7NMvVACUAEL9aa.jpghttp://p.twimg.com/A7EkZK8CEAE0pYu.jpghttps://p.twimg.com/A7FOSwtCUAADws1.jpghttp://p.twimg.com/A7EiFd4CcAAZdT1.jpghttp://p.twimg.com/A7JwHHbCQAAvqNN.jpg...

ARCOMEM Training Material

103 Detected Duplicates

https://p.twimg.com/A7K8WWiCAAE9Uap.pnghttps://p.twimg.com/A7Iq1DSCQAAyRBG.jpghttp://p.twimg.com/A7L7bQJCIAIaaqm.jpghttp://p.twimg.com/A7KSC5gCcAAcLTj.jpghttps://p.twimg.com/A7OqzarCYAA8L9N.jpghttps://p.twimg.com/A7OlKn-CQAA-j7X.jpghttps://p.twimg.com/A7J5CHPCUAEp-86.pnghttps://p.twimg.com/A7JOd9kCcAIy3YM.jpghttp://25.media.tumblr.com/tumblr_lyr1v4kp8J1qcjsjlo1_500.jpg...

106 Detected Duplicates

https://p.twimg.com/A7MQiKeCMAE1rmX.jpghttps://p.twimg.com/A7IFNt_CUAAqYFV.jpghttps://p.twimg.com/A7DrPLMCIAAoKb8.jpghttp://p.twimg.com/A7Y5BW9CQAAqF3E.jpghttps://p.twimg.com/A2YzbudCQAE77pr.jpghttps://p.twimg.com/A7FQF7PCUAAgWWH.jpghttps://p.twimg.com/A7EO5YFCMAAMTxB.jpghttp://p.twimg.com/A7IWHTtCUAIAtH8.jpghttp://p.twimg.com/A7FcyZdCUAAO_Vg.jpghttp://25.media.tumblr.com/tumblr_m7c9nivzwF1qfep67o1_500.jpg...

ARCOMEM Training Material

ARCOMEM Training Material

Multimedia Opinion Mining

ARCOMEM Training Material

Goals

• Investigate the use of facial analysis to classify facial expressions in images and videos found on the web.– Can be used to indicate emotion of subject.

• Investigate course-grained automatic classification using image features– For abstract opinion-related concepts

(sentiment/privacy/attractiveness)

• Investigate correlations between images and opinions mined from text– Does the same image get reused in different

documents to illustrate the same (or different) opinion?

ARCOMEM Training Material

Sentiment/privacy/attractiveness

• Experimental support for classifying visual media with respect to sentiment, privacy and attractiveness is being built into ARCOMEM.

ARCOMEM Training Material

Image-opinion correlation

• Correlations between images and opinions extracted from the text can be explored by querying the ARCOMEM database.

+ve -ve

ARCOMEM Training Material

Towards Facial Analysis in the Wild

• Detection and analysis of faces in multimedia content can help us guide and contextualise a crawl:– Recognition and expression analysis can help us

determine if an image is relevant or interesting.– Post-crawl the information can be used for visualisation.

• Current research very much based on images taken in lab-conditions; how far can we take it?

ARCOMEM Training Material

Applying the model to crawled images