Characterizing activity in video shots based on salient points Nicolas Moënne-Loccoz Viper group Computer vision & multimedia laboratory University of.

Characterizing activity in video shots based on

salient points

Nicolas Moënne-LoccozViper groupComputer vision & multimedia laboratory University of Geneva

[email protected]

NML - CVML - UniGe 2

Outline

• Context

• Video Activity extraction– Spatial salient points– Spatio-temporal salient points– Spatio-temporal salient regions

• Results

• Conclusion

NML - CVML - UniGe 3

Context• Describe visual content of video

Index, retrieve and browse video database

• Requirements– Generic approach (v.s. domain oriented)– Local approach (v.s. global description of the content)– Computationally efficient approach

• Video activity : salient region of the video 3D space

NML - CVML - UniGe 4

Context0 50 70Frames

Activity #1

Activity #2

Description in space and time of video activity Inference based on video object and event relationships High level indexing

NML - CVML - UniGe 5

Context

• Related approaches : Spatio-temporal segmentation– Segmentation problem– Computational efficiency

• Our approach :– Spatio-temporal salient points– Spatial grouping of salient points– Temporal matching of salient regions

Set of activities

NML - CVML - UniGe 6

Overview

Salient points& trajectories

Global motionestimation

Motion outliers

Spatial grouping

Video stream

Salient extraction

Temporal matching

Salient extraction

NML - CVML - UniGe 7

Salient points

• Points in the image space– Repetitive (robust)– High information content

Scale invariant interest points (Mikolajczyk, Schmid 2001)

– One of the most robust– Salient points with characteristic scale

NML - CVML - UniGe 8

Salient point extraction

• Linear Scale-Space :

• Harris function :

• Salient points (image space) : local maxima h(v,s)

• Laplacian over scale :

• Salient points (scale space) : local maxima l(v,s) & h(v,s)

H žv , s Ÿ= s 2G žv , ? ŸL x

2 žv , s Ÿ L x L yžv , sŸ

L x L yžv , sŸ L x2 žv , sŸ

NML - CVML - UniGe 9

Salient point extraction

• Example :

scale

NML - CVML - UniGe 10

Salient point extraction

scale

NML - CVML - UniGe 11

Motion estimation

• Goal :– Find points having salient temporal behaviour

Estimate background motion model Select points that do not follow this background motion

model

• Estimation :– Compute salient point trajectories– Estimate corresponding affine motion model

NML - CVML - UniGe 12

Trajectories

• Point descriptors : Local Grayvalue Invariants

• Point distance : Mahalanobis distance

NML - CVML - UniGe 13

Trajectories

• Goodness of match :

• Candidate matching points– Matches with spatial distance below a threshold

• Relaxation process :– Disambiguating set of candidate matches– Greedy Winner-Takes-All algorithm

NML - CVML - UniGe 14

Motion estimation

• Affine motion model :

• Estimate model from trajectories– Iterative least square error estimate (Tukey M-Estimator)

select points that belong to the global motion model

Assumption : +50% points belong to the background

NML - CVML - UniGe 15

Motion estimation

Points of the background and their motion estimated using the presented approach

All points and their motion estimated by a dense motion estimator

NML - CVML - UniGe 16

Spatio-temporal salient points

• Points whose trajectory does not fit the global motion model

Outliers (moving objects)

• Points without trajectory (no matching point) New points (appearing or deformable objects)

NML - CVML - UniGe 17

Spatio-temporal salient points

Fixed camera Moving camera

NML - CVML - UniGe 18

Salient regions

• Set of spatio-temporal salient points

Feature distribution of points (RGB colour features) Spatial distribution of points

• Grouping process : Estimate salient region models

NML - CVML - UniGe 19

Feature model

• Feature description– A salient point is characterized by the feature distribution of its

neighbourhood

– Assumption : maximum of four regions in the neighbourhood of the points

– Compute the corresponding colour distributions :• K-means clustering• Gaussian model

• Gaussian models clustering– Greedy algorithm (AHC)

Set of Gaussian distributions representing the distribution of the neighbourhood of the salient points :

NML - CVML - UniGe 20

Salient region model

• Feature models– Mixture of Gaussians

Corresponding weight of each Gaussian

• Spatial model : – Estimate spatial pdf from salient points & associated scale

NML - CVML - UniGe 21

Salient region models

• Iterate a RanSaC algorithm

• Estimate salient region model– Robust estimation (Tukey M-estimator)– Cost function :

NML - CVML - UniGe 22

Salient regions

Fixed camera Moving camera

NML - CVML - UniGe 23

Temporal matching

• Spatio-temporal salient regions of arbitrary length

Matching of salient regions

• Use salient points trajectories

1. Match regions with the highest number of matching points

NML - CVML - UniGe 24

Results - Meetings

NML - CVML - UniGe 25

Results – Misc

NML - CVML - UniGe 26

Conclusion

• Contribution– Highly informational content descriptor– Generic content descriptor– Local in space and time content descriptor

• Limitation– Noisy & short activity

• Ongoing work– Temporal filtering of activity– Indexing of videos through the set of activity