Human Action Recognition Based on Spacio-temporal features

Human Action Recognition Based on Spatio-temporal Features

Nikhil Sawant and Dr. K.K. BiswasDept. of CSE, Indian Institute of Technology, Delhi

Third International Conference on Pattern Recognition and Machine Intelligence (PReMi’09)

Human activity recognitionH

ighe

r res

oluti

on

Longer Time Scale

Courtesy : Y. Ke, Fathi and Mori, Bobick and Davis, Schuldt et al, Leibe et al, Vaswani et al.

Pose Estimation

Event Detection

Action Classification

Tracking

Activity Recognition

Use Action recognition?

• Video surveillance

• Interactive environment

• Video classification & indexing

• Movie search

• Assisted Care

• Sports annotation

Broad outline of our techniqueVideo with

human actions

Motions features

Shape features

Spatio-temporal features

Learning features though AdaBoos

Action Class 1 Action Class 2 Action Class 3 Action Class n

Motion analysis using Lucas –kanade

technique

Shape analysis using Viola-Jones feaures

Combining motion and shape features over finite time interval

……………......

Target Localization

• Possible search space is xyt cube• Action needs to be localized in space and time• Target localization helps reducing search space• Background subtraction• ROI marked

Original Video Silhouette Original Video with ROI marked

Motion estimation

• Make use of optical flows for motion estimation

• Optical flow is the pattern of relative motion between the object/object feature points and the viewer/camera

• We make use of Lucas – Kanade, two frame differential method, it comparatively yields robust and dense optical flows

Noise Reduction

• Noise removal by averaging

• Optical flows with magnitude > C * Omean are ignored,

where C – constant [1.5 - 2], Omean - mean of optical flow within ROI

Organizing optical flows

• Optical flows are aggregated near the motion

• Need for representing optical flow in meaningful way

• Fixed sized grid laid over the ROI

• Magnitude and direction of Optical flows within each box bij is averaged and assigned to its centre cij

• All optical flows have same weight

Organizing optical flows (simple averaging)

• Each optical flow given a weight

• More the distance from the centre cij less is the weight and vice-versa

Organizing optical flows (weighted averaging)

• Optical flows are arranged in structured mannered

• Arranged optical flows are easier to analyze

Organizing optical flows

Shape discriptor

• Shape gives information about the action

• Viola-Jones box features used to get shape features

• Shape information combined with motion information

Spatio-temporal descriptor

TLEN

TSPAN

Spatio-temporal descriptor

• Shape and motion features combined over the span of time to form spatio-temporal features

Learning with Adaboost

• Adaboost is state of art learning algorithm

• Linear decision stumps are used as weak hypothesis

• Weak hypothesis combine to form a strong hypothesis

• Strong hypothesis is weighted sum of weak hypothesis

• Training and testing data is kept mutually exclusive

Results

Results (Weizman dataset)

Thanks You