Spatio-temporal Segmentation of Video by Hierarchical Mean Shift Analysis Daniel DeMenthon SMVP 2002
Jan 03, 2016
Spatio-temporal Segmentation of Video by Hierarchical Mean
Shift AnalysisDaniel DeMenthon
SMVP 2002
Motivation
• Semantic understanding of video
• Object segmentation
• Video compression
• Event detection
• Video surveillance
Related Work
• Jojic et al. Flexible sprites
• Layer extraction
• Shi and Malik normalized cuts
• Irani et al. Event detection
Space – Time Volume Segmentation
Frame to Frame
Video Stack segmentation
Patch motion (1,u,v)
Feature Space
• 7 D feature vector, three color features in CIE L*u*v*, 2 motion angles, 2 motion distances.
ux arctan180
90
vy arctan180
90
xxx ttxxD cos)2/(sin)2/( maxmax
yyy ttyyD cos)2/(sin)2/( maxmax
Mapping Pixels in Feature Space
Mean Shift Clustering
• Introduced by Fukunaga (1990) and applied to image analysis by Yizhong Cheng and Comaniciu and Meer (1997)
• Natural borders (Leung et al.)
Range Search
• ATRIA tree
• O(N log N) for small radii
• O(N) for large radii
Hierarchical Mean Shift
• First standard mean shift is run until competition with very small radius
• Weights are assigned to cluster centers equal to the sum of the weights of the member points
• Clusters are now treated as the points, and radius is multiplied with factor of 1.25 or 1.50
• Repeat until desired radius or the desired number of regions is reached
“Flower Garden” Video Sequence
88 x 60, 12 frames
Video Strands
Color Segmentation
Motion Segmentation
Faster lateral motion corresponds to lighter color
Comparison of Two Segmentation Algorithms
Comments on this Approach
•Spatially distant color patches can be clustered together
•Experiment was with small number of frames
•It is not clear if it can handle the case when video object changes the direction of motion, or when video object stops
•All features (color features and motion features) are scaled using heuristics, and that might not work for different video sequences
Conclusions and Further Work
• Hierarchical mean shift analysis is of lower empirical complexity then standard mean shift analysis
• Segmentation can be improved, the bounding areas of moving areas are jagged, usually by post-processing
• Leung et al. suggested non parametric segmentation that can be applied here