Automatic Key Video Object Plane Selection Using the Shape Information in the MPEG-4 Compressed Domain Berna Erol and Faouzi Kossentini, Senior Member,

Automatic Key Video Object Plane Selection Using the Shape Information in the MPEG-4 Compressed Domain

Berna Erol and Faouzi Kossentini, Senior Member, IEEE

IEEE TRANSACTION ON MULTIMEDIA, JUNE 2000

Outline

introduction MPEG-4 shape representation key VOP selection experimental results conclusions

introduction

In a typical frame-based digital video indexing and retrieval system, key frames are used to represent the salient content of a video sequence.

MPEG-4 offers and object-based representation of video, where individual video objects (VOs) are coded into separate bit streams.

Temporal instances of video objects are referred to as video object planes (VOPs), and key VOPs can be used for visual summarization of the video object content in an object-based framework.

Shape Representation

MPEG-4 supports intra coded (I)

temporally predicted (P)

bi-directionally predicted (B) VOPs three shape coding for IVOPs

transparent block

opaque block

boundary block

Shape Representation (cont.)

Seven shape coding modes for P and BVOPs

Key VOP Selection

The texture and color of a video object remain generally consistent during a video object’s lifespan.

Using the shape of a video object instead of is color or texture is potentially more computationally efficient.

Exacting the shape information from the bit stream requires very few operations

Key VOP Selection Our method is based on the significant changes in the appr

oximated shape content of video objects. Hamming distance and Hausdorff distance shape approximation :

each 16 x 16 shape blocks is represented with one value

inside

outside

at the border

Key VOP Selection

IVOPs are ideal key VOP candidates for our key VOP candidates.

Approximation of the shape coding modes for P and BVOPS

Using the Modified Hamming Distance

Approximation of the shape of an VOP by using the shape coding modes in MPEG-4

the Hamming distance

Using the Modified Hamming Distance

A slight spatial shift between two very similar shapes may result in a large Hamming distance.

Aligning the mass centers of the two shape provides a good approximation for the alignment corresponding to the smallest Hamming distance.

Using the Modified Hamming Distance

Threshold

: empirically determined parameter that is constant for all

VOPs. : is determined by the activity level of the video object.

M1 and N1 : width and height (in number of blocks) of the key VOP,

respectively.

M2 and N2 : width and height of the key VOP candidate,

respectively.

Using the Modified Hamming Distance

Using the Modified Hamming Distance

Using the Hausdorff Distance

the Hausdorff distance

d(a,b) is the Euclidean distance between these points.

The Hausdorff distance is not symmetric

h(A,B) may not be equal to h(B,A)

Using the Hausdorff Distance

Threshold

: predetermined scale-factor that is constant for a VOPs.

M1 and N1 : width and height (in number of blocks) of the key VOP, respectively.

M2 and N2 : width and height of the key VOP candidate, respectively.

Bream Using the Hamming distance based algorithm

Using the Hausdorff distance based algorithm

Weather Using the Hamming distance based algorithm

Using the Hausdorff distance based algorithm

Hall Monitor Using the Hamming distance based algorithm

Using the Hausdorff distance based algorithm

the performance vs coding rate

The intra coded shape blocks are downsampled by a factor of four.

The selected key VOPs for the Bream video object

The Effects of The Video Object Activity Level

Employing a video object activity level dependent threshold

without employing a video object activity level dependent threshold

Key VOP Selection Using I,P,and BVOPs

We exact key VOPs from the IBBBPBBBPBBB structured Bream video object bit stream.

Uncompressed (actual) Shape Data vs The Approximated Shape Data

For the Bream video object

Compared with Other Algorithms Other algorithm ( proposed by Ferman etal.)

Using the Hamming distance based algorithm

Using the Hausdorff distance based algorithm

Computational Complexity

Conclusions

We have presented a method for key VOP selection using the Hamming and the Hausdorff distance measures.

Since the decompression of the shape data is not required, the bit stream processing time is also reduced significantly.

Using the shape approximations makes the proposed algorithms less dependent on the segmentation errors and how lossy the shape information is coded.

Depending on the application and available processing resources, either one can be used for key VOP selection.