Big-Data Analytics for Media Management

10/04/23 1

Krishna Chandramouli,Associate Professor,Media Engineering and Analytics Research Group, School of Information Technology and Engineering,VIT Universitykrishna.c@vit.ac.in

Big-Data Analytics for Media Management

10/04/23 2

Krishna Chandramouli,Associate Professor,Media Engineering and Analytics Research Group, School of Information Technology and Engineering,VIT Universitykrishna.c@vit.ac.in

Big-Data Analytics for Media Management

Overview Media and Internet Information Access Subjective vs Objective Indexing The Semantic Gap Evolving Strategies Social Media Analysis Indexing Large-scale Repositories Future Research Directions Take Away Message Q & A10/04/23 3

Media and internet

10/04/23 4

Media and Internet In March 2013 that Flickr

had a total of 87 million registered members and more than 3.5 million new images uploaded daily.

There are currently almost 90 billion photos total on Facebook.  This means we are, by far, the largest photos site on the Internet.

10/04/23 5

Information access

Textual search

Visual search

Search query formulation

10/04/23 6

Information Access

Traditional ordering of images is achieved through categorization of information into logical structures Creation of albums Categorizing through date/time Clustering through location

Image based search engines are gaining popularity with the increase in power of indexing schemes

10/04/23 7

Information Access

10/04/23 8

Information Access

10/04/23 9

Information Access

10/04/23 10

Information Access

10/04/23 11

Indexingsubjective or objective

10/04/23 12

Subjective vs Objective Indexing

How to uniquely name an image to make them distinguishable?

What names can be used to search images? How many names are needed to make the

images unique? Will all humans use the same names to

identify the images?

10/04/23 13

Subjective vs Objective Indexing

Humans are culturally influenced Terms contain different meanings across

boundaries and cultures Therefore, any tag/word assigned to an image

will be considered subjective Objective signatures for images are generated

from the characteristics of the images The beginning of MPEG-7 standardisation

activities.10/04/23 14

Subjective vs Objective Indexing Image characteristics exploited for objective

annotation include Colour

Colour Layout Descriptor Colour Structure Descriptor Dominant Colour Descriptor Scalable Colour Descriptor

Texture Texture Browsing Descriptor Edge Histogram Descriptor Homogenous Texture Descriptor

Shape

10/04/23 15

The semantic gap

10/04/23 16

The Semantic Gap

The semantic gap characterizes the difference between two descriptions of an object by different linguistic representations, for instance languages or symbols.

In computer science, the concept is relevant whenever ordinary human activities, observations, and tasks are transferred into a computational representation

10/04/23 17

The Semantic Gap

10/04/23 18

The Semantic Gap

10/04/23 19

Evolving strategies

Image Classification

Visual Classifier

Knowledge Assisted Analysis

Image Retrieval and User Relevance Feedback

Multi-Concept Space Search and Retrieval

10/04/23 20

Evolving Strategies The problem of Image classification and

clustering has been the subject of active research for last decade. Mainly attributed to the exponential growth of digital content.

The efficiency of the clustering and classification algorithms can be attributed to the efficiency of the machine learning approaches.

To improve the performance of machine learning algorithms, different optimisation techniques has been employed such as Genetic Algorithms.

10/04/23 21

Evolving Strategies Recent developments in applied and heuristic

optimisation techniques have been strongly influenced and inspired by natural and biological systems.

Algorithms developed from such observations are Ant Colony Optimisation (ACO) - based on the ability of

an ant colony to nd the shortest path between the food and the source compared to an individual ant.

Articial Immune System (AIS) - typically exploit the immune system's characteristics of learning and memory to solve a problem

Particle Swarm Optimisation (PSO) - inspired by the social behaviour of a flock of birds.

10/04/23 22

Evolving Strategies In the study of "Semantic Gap", machine

learning algorithms are the building blocks for bottom-up approach.

Some of the applications of efficient machine learning algorithms are: Automatic Content Annotation Knowledge Extraction Content Retrieval

In the top-down approach, Ontology provides partial understanding of human semantics.

10/04/23 23

Visual classifier

10/04/23 24

Slide: 25

Particle Swarm Optimisation In an effort to transform the social interaction of

different species into a computer simulation, Kennedy and Eberhart developed an optimisation technique named Particle Swarm Optimisation.

• In theory, the universal behaviour of individuals is summarised in terms of Evaluate, Compare and Imitate principles.

Slide: 26

Particle Swarm Optimisation

Evaluate: The tendency to evaluate stimuli – to rate them as positive or negative, attractive or repulsive is perhaps the most ubiquitous behavioural characteristic of living organisms.

Compare: In almost every aspect of life, human tend to compare with others

Imitate: Humans imitation comprises taking the perspective of the other person, not only imitating a behaviour but also realising its purpose and executing the behaviour when it is appropriate

Slide: 27

Particle Swarm Optimisation

Equations governing the motion of particles in PSO.

Slide: 28

Particle Swarm Optimisation

Pseudo code for the algorithm Step 1: Random Initialization of Particles Step 2: Function Evaluation Step 3: Computation of personal best and global

best Step 4: Velocity update Step 5: Position update Step 6: Loop to step 2, until the stopping criteria

is reached

Slide: 29

Visual Classification Framework

Self Organising Map

[X]

[X] - Input feature vectorClass 1 – RedUntrained - Black

Winner Node selected based on L2 norm

Slide: 30

Visual Classification Framework

Training of R-SOM network with PSO Algorithm

Slide: 31

Visual Classification Framework

.. .

Winner Node

)]([)()1( tmxhtmtm iciii )]([)()1( tmxhtmtm iciii

Dual Layer SOM

Dual – Layer SOM Network

Slide: 32

Chaos-Particle Swarm Classifier The elementary principle of “Chaos” is introduced to

model the behaviour of particle motion.

The theoretical discussion on Chaotic – PSO includes the notion of “wind speed” and “wind direction” modelling the biological atmosphere for position update of the particles.

Slide: 33

Chaos-Particle Swarm Optimisation

The wind speed and therefore the position update equation are presented by:

Knowledge assisted framework

10/04/23 34

Slide: 35

Knowledge Assisted Analysis

Architecture

Slide: 36

Knowledge Assisted Analysis

Machine Learning - Evaluation

Slide: 37

Knowledge Assisted Analysis Experimental Dataset

A set of 500 Images, belonging to the general category of vacation images was assembled.

The content was mainly obtained from Flickr online photo management and sharing application and includes images that depict cityscape, seaside, mountain and landscape locations.

Every image was manually annotated, i.e. after the segmentation algorithm is applied, a single concept was associated with each resulting image segment

Slide: 38

Knowledge Assisted Analysis

A subset of Database

Slide: 39

Knowledge Assisted Analysis

Comparison of Machine Learning techniques

Slide: 40

Knowledge Assisted Analysis From the results it can be seen that the combined use

of PSO optimisation technique with SOM results in better classification accuracy compared to using the latter alone.

It can be noted that the performance of PSO classier is better than the performance of SVM and GA classifiers.

Since, SVM's need large training data to accurately discriminate between image classes.

Image retrieval and user relevance feedback

10/04/23 41

Slide: 42

User Relevance Feedback

Overview of Multimedia Retrieval System

Slide: 43

User Relevance Feedback

Relevance Feedback Framework

Slide: 44

User Relevance Feedback The database used in the experiment is generated

from Corel Dataset and consists of seven concepts namely, building, cloud, car, elephant, grass, lion and tiger

The test set has been modelled for seven concepts with a variety of background elements and overlapping concepts, hence making the test set complex.

Slide: 45

User Relevance Feedback

Example images from Corel Dataset

Slide: 46

User Relevance Feedback

Average Accuracy for 7 concepts and 10 user interaction

Multi-concept search space

10/04/23 47

Multi-concept framework

Slide 48

• High-level queries“A tiger resting in the forest and guarding his

territory” • Mid-level features (context independent)“Tiger”, “Grass”, “Rock”, “Water”,……

Multi-concept framework• Mid-level features:In a constrained environment with limited number of

mid-level features, the performance of classification algorithm has found to be satisfactory

• High-level queries:Open to subjective interpretation of the concepts

and also may involve more than one mid-level feature

Main objective:• In this multi-concept framework, users are

encouraged to construct high level queries based on their preferences

Multi-concept framework

Slide 50

Mid-level feature extraction

Slide 51

• SVM Classifier • SVM Light toolbox was used to generate

semantic labels• CLD+EHD

• Multi-feature classifier (MF) • Employs a mixture of 7 visual features.

• The visual features are merged using Multi-Objective Learning (MOL)

Query space formulation

Slide 52

• Pre-processing stage: mid-level feature concept detection

• Query formulation: users to construct a high-level semantic information space

Query space visualisation Fisheye distortion technique Overview + focus

Slide 53

Query space visualisation

Slide 54

• Query space panel• Concept map panel• Concept chart panel

Experiments and Evaluation

Slide 55

• A 3500 image set collection • From Corel dataset• Natural images with many elements• Foreground and background• Rich semantic context• Fully annotated

• 10 mid-level conceptslion, water, grass, building, car, cloud, rock, tiger,

elephant, flower

• 8 high-level conceptsflower fields, modern city view, rural garden, mountain

view, waterfalls, wild life, city street, boat

Comparison of results Retrieval of high level queries using

the proposed MCB framework

Slide 56

Comparison of results Retrieval of high level queries using

SVM classification

Slide 57

Comparison of results Content-based retrieval with RF

mechanism

Slide 58

Experiments and Evaluation

Slide 59

Experiment and Evaluation

Slide 60

Experiment and Evaluation

Slide 61

User 1Landscape water, grass 0.58Modern city building, cloud 0.8Wild life lion, tiger, elephant 0.59Rural garden flower, water, grass 0.9

User 2Landscape water 0.23Modern city building 0.71Wild life lion, rock, grass, tiger, elephant 0.87Rural garden flower 0.28

User 3Landscape water, grass, cloud, car, elephant 0.59Modern city cloud, building, car 0.91Wild life lion, tiger, grass, elephant, rock 0.82Rural garden flower, water, grass 0.88

Social media analysis

10/04/23 62

Social Media Analysis

Social media is the interaction among people in which they create, share or exchange information and ideas in virtual communities and networks.

Andreas Kaplan and Michael Haenlein define social media as "a group of Internet-based applications that build on the ideological and technological foundations of Web 2.0

10/04/23 63

Social Media Analysis

Social media allows for the creation and exchange of user-generated content.

Social media differ from traditional or industrial media in many ways, including quality, reach, frequency, usability, immediacy, and permanence.

10/04/23 64

Slide: 65

Textual and Visual Analysis• Images are often accompanies with free-text

annotations, which can be used as complementary information for content-based classification

• The challenge is to extract entities from text and classify them into an arbitrary set of classes

Plansarsko lakeShepherd in Bucegi National Park

Slide: 66

Labeled Corpora(MUC,

BBN,ACE)

Textual and Visual Analysis

Slide: 67

Textual and Visual Analysis

Slide: 68

Annotated Images

Binary Segmentation Masks

Segmentation of Images Feature

Extraction

Biologically Inspired

Classifier

Training ModelCic={Sky, Rock,..}

Semantic ConceptMapping

TargetedHypernym Discovery

Wordnet

Wikipedia ClassifierFusion

Labeled Segments

Labeled Entities

Ctc = {Person, Landscape,..}

Visual analysis (KAA)Visual analysis (KAA)

Text Analysis (SCM+THD)Text Analysis (SCM+THD) Fusion Fusion

Use-case scenario

Slide: 69

Church of our Lady Mercy in Buje

BuildingBuilding

EXIF

Binary mask

PSO

Largest region size

Use-case scenario

Slide: 70

Map word to Wordnet concept

1. noun phrase

2. head noun

3. hypernym for noun phrase (with THD)

4. hypernym for head noun (with THD) Compute similarity with each of the classses

Experiments carried out with Lin similarity measure

The probability of encountering concept cis usually estimated from a large corpus

Semantic Concept Mapping

Slide: 71

Content-based analysis (KAA) restricted to classes for which the classifier has been learnt

For text-based analysis (SCM/THD), the classes have to be exhaustive - all entities are classified

Mapping from SCM/THD to KAA

Perform intersection between the individual classifier results

Select concept occupying largest area on the image

Image Class.(KAA)

Text Class.

Classifier Fusion

Indexing large-scale repositories

10/04/23 72

Indexing Large-scale Repositories

10/04/23 73

Indexing Large-scale Repositories

The textual analysis block aims to generate a list of named entities extracted from the textual metadata associated with the input video

The pre-processing framework classifies the tags into two general categories common-tags named entities

10/04/23 74

Indexing Large-scale Repositories

Common tags correspond to either action, country or associated with synset in WordNet

Named-entity tags do not have a WordNet synset and thus depend on extrenal resources to contextualise them

The objective of the pre-processing module is to ensure the named entities are disambiguated to enable a semantic similarity search

10/04/23 75

Indexing Large-scale Repositories

Bag of Articles Classifier The input of a BOA classifier is a set of labelled

instances and a set of unlabelled instances (noun chunks).

Wikipedia article titles provide an unanimous mapping between the labelled instance and a wikipedia article

Each article is described by its type (article, page, disambiguation page, category page and so forth)

10/04/23 76

Indexing Large-scale Repositories A BOA classifier requires a Wikipedia index

containing the following information about each article term vectors with term frequencies out links and popularity ranking (for most frequent sense relevance

ranking) For geo-tagging adaptation, the textual analysis

block searches for geographical named entities in the queries Wikipedia articles

The location details are extracted with the help of DBpedia using SPARQL end-point

10/04/23 77

On-going research challenges

10/04/23 78

VIT@MediaEval 2013

Social Event Detection Task

10/04/23 79

VIT@MediaEval 2013

10/04/23 80

The geographical coordinates is an important component and indicator of where an event has happened.

The event clusters are nalised through the weighted occurrence of tags among the distribution of media annotation

VIT@MediaEval 2013

10/04/23 81

The system computes the similarity between synset representing the tags and each of the categories.

We use Lin similarity measure to evaluate the semantic distance between the synset and category.

VIT@MediaEval 2013

Placing Task

10/04/23 82

VIT@MediaEval 2013

Dividing the globe into grids with a maximum of 10,000 images per grid . Starting from an initial grid that spans the entire globe, recursively subdividing grids into smaller ones once the threshold is reached.

10/04/23 83

VIT@MediaEval 2013

10/04/23 84

Future research directions

10/04/23 85

Future Research Directions

MediaEval is a multimedia benchmarking initiative that offers tasks and datasets to the research community that emphasize the human and social aspects of multimedia.

In 2014, MediaEval is offering eight classic tasks and three Brave New Tasks.

http://www.multimediaeval.org/mediaeval2014/

10/04/23 86

Future Research Directions

ImageCLEF 2014 ImageCLEF organizes four main tasks to

benchmark the challenging task of image annotation for a wide range of source images and annotation objective, such as general multi-domain images for object or concept detection, as well as domain-specific tasks such as visual-depth images for robot vision and volumetric medical images for automated structured reporting.

10/04/23 87

Future Research Directions The tasks address different aspects of the annotation

problem and are aimed at supporting and promoting the cutting-edge research addressing the key challenges in the field, such as multi-modal image annotation, domain adaptation and ontology driven image annotation.

http://www.imageclef.org/2014

10/04/23 88

Thank you!!!Q & A

10/04/23 89