Video Trails: Representing and Visualizing Structure in Video Sequences
Vikrant Kobla
David Doermann
Christos Faloutsos
Outline
Background and Motivation Overview Video Trails Trail Segmentation Trail Classification Gradual Transition Detection Experiments and Results Conclusion
Background and Motivation
Video is a valuable information resource There are still few efficient ways to provide
access to the information the video contains Early work on indexing video treated video
sequence as collections of still images, ignored the temporal structure
Efficient analysis and representation of the temporal structure of a video is necessary
Overview
1. Generate a trail of points (Video Trails) in a low-dimensional space
2. Segment the video trails
3. Classify each of those segmented trails into two types:
Stationary (low activity) VS Transitional (high activity)
4. Detect gradual transition
Video Trails
Definition: A trail of points in a low-dimensional space where each point is derived from physical features of a single frame in the video clip
Features: DC coefficients of the luminance and chrominance components of an MPEG frame
Dimensionality Reduction (FastMap) initial feature vector
a vector in that dimensional
target dimension spaceFastMap
Example
Consider a video clip with a 320x240 frame size Each frame has 20x15 MBs( Macroblock) Each MB contains 6 DC coefficients ( 4
luminance and 2 chrominance) Totally, 20x15x6=1800 coefficients (initial vector) 1800-by-1 vector
(X1,X2,X3)
3 (target dimension) FastMap
Example
Example
Trail Segmentation
Segment the video in order to determine regions of high activity corresponding to transitions and low activity corresponding to individual shots
The problem of segmenting the video into sets of frames is transformed into the problem of splitting the video trails into smaller trails corresponding to segments of video
Splitting Algorithm
1. Start by placing the first point in a new trail
2. Consider each successive point in the sequence in order
3. Perform a test for “inclusion” of this point in the current trail
4. if (the test pass)
5. Include the point in the current trail
6. Move to the next point
7. Goto 2
8. else
9. Close the current trail with the previous point as the last one
10. Start a new trail with only the current point
11. Goto 2
“Inclusion Test”
Marginal Cost:Total cost per point in the trail Consider a clip with N frames Assume there are m points in the current
trail, denoted by set , and be the point being considered for inclusion
Define ,d is the dimensionality So the new marginal cost is
new marginal cost > previous one : not include
new marginal cost < previous one : include
Example
Example (close-up)
The sequence of frames that yield the sparse transition between the two dense clusters
Trail Classification
Classify each of those segmented trails into: Stationary (low activity) or Transitional (high activity)
Classification Criteria– Monotonicity W1=0.4 – Sparsity W2=0.3– Convex Hull Volume Ratio W3=0.2– MBR Shape W4=0.1
Monotonicity
If a trail is (close to) monotonic, in some direction,it’s likely transitional projection of distance along k
projected distance ratio
the length of MBR dimension k Minimum projected distance ratio
Monotonicity (Normalization)
Recall: W1 is the weight of monotonicity Tlow is the lower bound=1.1 Tup is the upper bound=2.0
Sparsity
Sparsity: total MBR volume per point
Average Sparsity
Sparsity Ratio
Normalize
Convex Hull Volume Ratio
The ratio of volume of the convex hull of points in a trail to the volume of MBR
Normalize
MBR Shape
Cuboidal Planar Elongated
Classification
Gradual Transition Detection
Dissolves, Fades, Wipes Difficulty: activity arising from camera or
large object motion also yields trails similar to trails resulting from gradual edits
Filter out any kind of global motion leading to a transitional trail, Analysis global motion
Results
Conclusion
Provide a compact representation of a video sequence structure
Reduce a sequence MPEG frames to a trail of points in a low dimensional space
Segment trails and classify each segment as either stationary or transitional
Detect gradual edits