ADVISE: Advanced Digital Video Information Segmentation Engine

ADVISE: Advanced Digital Video Information

Segmentation Engine

Presented by Ng Chung Wing

OutlineIntroductionOverview of ADVISE System architecture and services provided

Technologies in ADVISE Construction of Video Table-of-Contents (V-To

C) Video Summarization Video Matching

Conclusion

Introduction

IntroductionVideos is getting more popular in education, entertainment and information sharingEvident growth of video contents on the Internet

57.2% of Internet users watched video chips and 7.3% edited video clips on their personal computers (Survey by PC Data, 2000)

Important task retrieve an interested video!Two problems in video retrieval: Not enough information to describe the video

contents Difficult to search for videos with similar

contents

Introduction

IntroductionWe propose the “ADVISE” system to solve the above problemsADVISE Advanced Digital Video Information

Segmentation Engine Web-based video browsing and retrieval

system Provides a set of services:

For description of videos: Video table-of-contents (VToC) Video summarization

For searching similar videos: Video feature similarity matching

Introduction

ContributionsWe propose the framework of a video browsing and retrieval system called ADVISEWe build the image-based video description called Video Table-of-Contents (V-ToC) in ADVISEWe develop the Video Summarization Algorithm for generating video summaries in ADVISEWe propose two Video Tree Matching Algorithms, which measure the feature similarity between videos, in ADVISE

Overview of ADVISE - OutlineIntroduction

Overview of ADVISE Objective of ADVISE System Architecture Services provided by ADVISE

Technologies in ADVISEConclusion

Overview of ADVISE

Objectives of ADVISETo provide an efficient way to describe the video contentsTo save the time for browsing the whole video to know the contentsTo search videos with similarity in certain video featuresTo provide services through the Internet

Overview of ADVISE

Provide services to users of ADVISE

Major contributions

System Architecture of ADVISE

ADVISE consists of 3 modules

(I) Video preprocessing module

(II) Web-based retrieval module

(III) Video streaming server

Process source videos

Setup the Real System Server for delivering video contents

Setup the Real System Server for delivering video contents

(III) Video streaming server

Overview of ADVISE

System Architecture of ADVISE

Input Raw Video

Web-based Video RetrievalModule


Internet User Terminal

1. Query for theV-ToC to know the

video contents

2b. Return CustomizedSMIL Video Summary

to user

2a. SubmitSelection Request

XML V-TOCspresented with

XSL

Video PreprocessingModule

Video Matching

V-ToC Construction

Video MatchingResultsStreaming Video Server

Streaming Video Server

Indexed VideoDatabase

Generation ofSMIL VideoSummary

3. Query onsimilar videos

ADVISE

III

III

Overview of ADVISE

Service 1: V-ToC Presentation

Service 2: Generation of SMIL Video Summary

Service 3: Querying Similar Videos

(II) Web-based Video Retrieval Module

User interface for accessing services of ADVISEThis module reside on a web serverThere are 3 services provided by ADVISE



Internet User Terminal

1. Query for theV-ToC to know the

video contents

2b. Return CustomizedSMIL Video Summary

to user

2a. SubmitSelection Request

XML V-TOCspresented with

XSL

Video MatchingResults

Generation ofSMIL VideoSummary

3. Query onsimilar videos

1

2

3

Overview of ADVISE

Module (II) ~Service 1: V-ToC Presentation

Image-based description for the video content Use the V-ToC structure resulting from Module (I)

Used XML with XSL to generate a flexible web-based presentation

Each V-ToC show us the contents organization in a video

Overview of ADVISE

Module (II) ~ Service 2: Generation of SMIL Video Summary

Generate a video summary according to user’s preferenceUsed SMIL to deliver the customized video summary to the user

User’s input

Resulting SMIL Video Summary

Overview of ADVISE

Module (II) ~Service 3: Querying Similar Videos

Show similar videos in descending order of the similarity score Results of video matching in Module (I)

User can select matching different video features Color histogram feature Shot style feature

List of videos with differentsimilarity scores with the queryvideo

Technologies in ADVISE - OutlineIntroductionOverview of ADVISE

Technologies in ADVISE For service 1: Construction of Video Table-of-Conte

nts (V-ToC) For service 2: Video Summarization For service 3: Video Matching

Conclusion

Technologies in ADVISE - Construction of V-ToC

Construction of Video Table-of-Contents (V-ToC)

Uses Image-based video description which show

the organization of video contents Video Table-of-Contents (V-ToC)

Provide the hierarchy for structural matching of video

Video structure used in ADVISE Hierarchical tree structure with 4 levels

Storage and Presentation Use XML and XSL


Video Structure in ADVISEDecompose a video into 5 levels: Video Frames Video Shots Video Groups Video Scenes Whole Video

Hierarchical Representation of a Video


Video Structure in ADVISEExample:

Group 1

Group 3

Group 2

Scene 1

Scene 2

Video

Shots:

Shot 1

Shots 2,4,6

Shots 3,5,7

12

3

4


Video Structure in ADVISEStructure videos from the bottom level5 steps in video structuring

i. Color Histograms Extraction ii. Video Shot Boundaries Detection iii. Video Groups Formation iv. Video Scenes Formation

Vid

eoS

hot 4

Vid

eoS

hot 6 ...

Vid

eoS

hot 2(Video Shot Level of V-ToC)

...(Video Group Level of V-ToC)

Vid

eoS

hot 3

Vid

eoS

hot 5

Vid

eoS

hot 7...

VideoGroup 1

VideoGroup 2

VideoGroup 3

VideoGroup 4

VideoGroup 5

Video Scene 2Video Scene 1 Video Scene 3 ...(Video Scene Level of V-ToC)

(Video Level of V-ToC) The Whole Video


Storage and PresentationResulting Presentation of V-ToC using XML and XSL

Folding/Unfolding Video Components

Sequential Video Story Line

Enlarging Images

Basic Video Information

<?xml version="1.0"?><!DOCTYPE advise SYSTEM "./toc.dtd"><advise><video length ="25" src="rstp://localhost/video1.rm"><scene id="1"> <group id="1"> <shot id="1"> <keyframe img="./sh_1.jpg"/> <time value="0"/> </shot> <shot id="2"> <keyframe img="./sh_2.jpg"/> <time value="11"/> </shot> </group></scene></video></advise>




Conclusion

Technologies in ADVISE - Video Summarization

Video SummarizationUser may still not be able to know the exact video contents with V-ToCVideo summary can provide all types of information in the videoObjectives:

Select the major contents Shorten the duration for browsing

Difficulties No standard method to pick the important contents from vi

deo Importance of contents depends on user’s need

In ADVISE: We accept user’s input for generating video summary suc

h that the result can be the best suitable for the user


Inputs for Video Summarization Algorithm

Video features used: Human faces Male and female voices Volume level Caption text

User’s inputs for customization of the video summary

Weights of different video features

Time constraint for video summary

Clustering control constant


Video Summarization Algorithm4 steps to summarize a video

i. Combining extracted video segments ii. Scoring the extracted video segments iii. Selecting extracted video segments iv. Refining the selection result

Example:

f1

Male Voice

Female Voice

Human Faces

Volume > 35dB

Caption Text

Feature Exists

Feature Not Exist

Video Frames Sequence

f80 f100 f300 f500 f5000 f5300 f6000

Step (i)

Step (iv)

Step (ii)

e7

e1 e2

e3e4

e6

f1 f80 f100 f300 f500 f5000 f5300 f6000


Sco

re fo

r V

ideo

Seg

men

ts

22

e5

21

6

5

3

0

Step (iii)

e7

e1

e3e4

e6

f1 f80 f100 f300 f500 f5000 f5300 f6000


22

e5

21

6

5

3

e2

Selected Segment

0

Sco

re fo

r V

ideo

Seg

men

ts

u1

Clu

ster

Sco

res

VideoFrames

u2 u3

u4

SelectedCluster

ea1

ea3


e a2

SelectedSegment

ea4

ea5 e a6

Short DisjointedSpaces

ea1

ea3


ea2

SelectedSegment

ea4

ea5 e

a6

Cluster u1

Cluster u2


Video Summary in SMILSMIL presentation are delivered to user of ADVISE Can be generated instantly Can be browsed by the user on the Internet

using a stream-based protocol

Resulting SMIL video summary




Conclusion

Technologies in ADVISE - Video Matching

Video MatchingVideo Matching

Match the extracted video features Color, motion, shape, etc.

Sequential matching Non related to video structure

VToC is a tree structure Can apply tree matching algorithm Matching related to video structure

In ADVISE, we propose two tree matching algorithms (1) Non-ordered tree matching algorithm (2) Ordered tree matching algorithm (Consider temporal or

dering)


Input Features for Video MatchingTwo video features used Color histograms feature

Take the first frame of a video shot as the key frame to compare in order to reduce the computational complexity.

Compare the visual similarity. Shot style feature

Compose of camera motion and length of a video shot.

Select the first camera motion in a video shot as the representative.

Compare the similarity in video pace.


(1) Non-ordered Tree Matching Algorithm

Not constrained by temporal orderingCapture all similar components

Algorithm Extract features at the bottom level Propagate the similarity score up to the root

level

Group a1 Group b2

Video A Video B

Group a2 Group b1


(2) Ordered Tree Matching AlgorithmConstrained by temporal orderingTemporal ordering can affect the video contents Reduce the problem to match components in

order

Capture only ordered similar components

Group a1 Group b2

Video A Video B

Group a2 Group b1


(2) Ordered Tree Matching AlgorithmAlgorithm: Recursive dynamic programming

Hierarchical matching From the tree root (video level) Until shot level, extract the video feature

similarity

Reduced the complexity compare with approach (1)

sim(Videoa,Videob) =

0 0 0 0

0 D(2,2) D(3,2) D(4,2)

0 D(2,3) D(3,3) D(4,3)

c1

cb1

cb2

D

Videob

Videoa

c2 c3

D(4,3)

2

0 0 0

0 D1(2,2) D1(3,2)

0 D1(2,3) D1(3,3)

ChildSim(1,1)

=

g2

C1of Video b

D1g1

C 1of Video a

g1

g2

D1(3,3)

2

0 0 0

0 D2(2,2) D2(3,2)

0 D2(2,3) D2(3,3)

g1of Videoa

g1of Videob

D2 s1

s5

s1

s3

ChildSim( )

Conclusion

ConclusionThe ADVISE system, which enhanced video browsing and retrieval system on the Internet, is proposed.The generation and presentation of the image-based video description are developed.The automation of video summarization into SMIL format is provided.Two video tree matching algorithms for measuring the similarity between videos are proposed.

Questions & Answers

ADVISE: Advanced Digital Video Information Segmentation Engine

Documents

video contentsto

video summaries

video chips

video contentuse

interested video

video contentsdifficult

customized video summary

certain video featuresto