A Bayesian Formulation For 3d Articulated Upper Body Segmentation And Tracking From Dense Disparity Maps Navin Goel Dr Ara V Nefian Dr George Bebis.

A Bayesian Formulation For 3d Articulated Upper Body Segmentation And Tracking From Dense Disparity Maps

Navin GoelDr Ara V NefianDr George Bebis

Overview Introduction Gesture Recognition System 3D Upper Body Model Initialization Segmentation Tracking Results Future work

Introduction

Applications: Human-Computer Interaction Systems, American Sign Language Recognition, Security-Monitoring Systems, Gesture Recognition.

Requirements: Background and scene independent, Robust hand free initialization, Robust tracking.

Introduction

Related Work: A. V. Nefian, R. Grzeszczuk, and V. Eruhimov. A

statistical upper body model for 3D static and dynamic gesture recognition from stereo sequences. In IEEE International Conference on Image Processing, volume 3, pages 286-289, 2001.

Success: 96% Accuracy.

Gesture Recognition System:

Assumptions: One user in the camera view,

The torso is the largest visible component,

The torso plane is approximately perpendicular to the camera,

The user’s head is approximately in vertical position.

Observation Vectors:

Observation Vectors = depth + color

Upper Body Model

]),,,[(],[ ,,, ijijijijcji

djiji hzyx OOO

Pixel

Od Oc

Threshold is chosen in such a way that the foreground pixels consist mostly on the upper body pixels.

Head:

Where,

N is a Gaussian density function, µH and CH are mean and covariance of gaussian density function.

U is a uniform distribution function, H is Head & SH is the size of head.

Upper Body Model

otherwise

NeckifYSHUCONSNHOP jiHHH

dji

Hdji

,0

),,(),|(),,|( ,,

,

Torso:

Where,

Upper Body Model

otherwise

NeckifYWNeckUcbaWNeckTOP ijTT

dij

,0

),,(),,,(),,|(

2

)2

))((exp(

2

1),,,(

2

2

2

cbOaOOcba

dy

dx

dz

z

ijijij

T is Torso & WT is Width of Torso

Arms – Linear PDF

Where,

RF is radius of Fore Arm, Fl is Left fore arm, ρij & σ is mean and covariance of Gaussian.

Upper Body Model

),(),|(),,|( FijijdijFijl

dij RUORFOP

)2

)(exp(

2

1),|(

2

22

ij

dij

ijdij

OO

EM Algorithm The E Step Compute: P(Oij | J,C)

The M step finds: J,C = arg max(P(Oij | J,C))

Where,

J are Joints & C are Components of upper body model

Head & Torso Starting with an initial position of the neck, the

parameters of the torso plane are estimated below the neck plane using the EM approach.

Using the torso center and anthropological measures of the head, parameters of the head are estimated using the EM approach.

Estimate the position of Neck:N(x) = μHe(x)

N(y) = μHe(y) + SH/2N(z) = aN(x) + bN(y) + c

Initialization

Head & Torso Estimate the position of Shoulders:

Sl(x) = µT(x) + WT/2

Sl(y) = N(y)

Sl(z) = aSl(x) + bSl(y) + c&

Sr(x) = µT(x) - WT/2

Sr(y) = N(y)

Sl(z) = aSl(x) + bSl(y) + c

Repeat the above steps for updated JB until convergence error falls under a threshold.

Initialization

Arms For each possible arm joint configuration we estimate

the mean of the linear pdfs corresponding to the upper and fore arms, and the mean of the normal pdf for hands.

For each joint configuration of the arms, we determine the best state assignment of the observation vectors.

Find the max likelihood over all joint configuration and determine the best set of joints and the corresponding best state assignment.

Initialization

Head & Torso: Estimate head and torso

parameters and position of joints using the algorithm in Initialization – 1. The value of neck is obtained from previous frame.

TRACKING

Arms: Step 1: Find 4 points in sphere that are equidistant from

original elbow position based on theta and phi values passed. These points constitute the new set of possible elbow joints to search in the current iteration. FOR(I = 0; I < 5; I++)

Step 2: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance.

Step 3: For each new elbow joint, find 4 new positions for wrist + original based on theta and phi values passed. FOR(J = 0; J < 5; J++)

Step 4: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. PDF is defined as start (elbow) and end (wrist) points of joints.

TRACKING

Arms: Step 5: For each new wrist joint, find 4 new positions for

hand center + original based on theta and phi values passed. FOR(K = 0; K < 5; K++)

Step 6: Find log likelihood of Probability Density Function (PDF) found by EM given 3d planar distance. PDF is defined as start (wrist) and end (hand center) points of joints.

Step 7: Calculate Max Tracking Log Likelihood of Single Arm. Calculate argmaxElWl{ P(Oij|{Ul, Fl, Hl},{El,Wl})} Convergence Error = Best Probability – Previous Probability

Step 8: Repeat the above steps until convergence error.

TRACKING

Success:

Results

Success:

Results

Failure:

Results

Future Work: The system collapses when hand is adjacent to head.

Integration with other systems.

Learning anthropological measures of the person.

Acknowledgements:

National Science Foundation

University Of Nevada, Reno

Intel Corporation, Santa Clara