Human Posture Recognition with Convex Programming Hao Jiang, Ze-Nian Li and Mark S. Drew School of Computing Science Simon Fraser University Burnaby, BC,

Human Posture Recognition with Convex Programming

Hao Jiang, Ze-Nian Li and Mark S. Drew

School of Computing Science

Simon Fraser University

Burnaby, BC, V5A 1S6

July, 11th, 2005 Simon Fraser University

Human Posture Recognition

Recognizing human postures is very important in vision and multimedia.

It has many applications in surveillance, human computer interaction, image and video database analysis and retrieval.

At the same time, recognizing human postures is a hard problem.


The Challenges of Human Posture Recognition It is hard to recognize human postures because:

– Articulated nature of a human body– No segmentation schemes are available for general

images or videos.– Strong background clutters.– Large appearance changes because of clothing

Different schemes have been studied.


Methods for Posture Recognition

Methods having been studied:– Silhouette based method with background

subtraction– Multi-camera based methods– Tracking body movement– Chamfer matching based schemes– Shape context based schemes

These methods are not sufficient to address the problem robustly.


The Proposed Method

We will present a matching based scheme that has the following properties:– Based on a robust convex (linear) programming

matching scheme– Work for cases where no background subtraction is

available– Able to deal with strong background clutters– Able to deal with large appearance changes


Matching Distance Transform

Canny EdgeDetection

DistanceTransform

Feature PointSelection

DelaunayTriangulation

TemplateGeneration

Canny EdgeDetection

DistanceTransform

MatchingWith LP

TemplateImage

TargetImage

Object Recognition

result


Matching as a Labeling Problem

p

q

fp

fq

Target p’

Target q’

Clutter

Target

Template Mesh

Target Image


The Labeling Problem The matching problem can be formulated as

the following optimization problem:

Matching cost Smoothing term


Convex Relaxation The original problem is a hard non-convex

problem. We convert it to LP:

|fp-fq|c’(s,j)


Properties of the Relaxation1. For convex problems, LP exactly solves the continuous extension of

the original problem.2. For general non-convex problems, LP solves the problem where

each matching surface is replaced by the lower convex hull. 3. The “cheapest” basis set for each site corresponds to the lower

convex hull’s vertices


The Effect of Covexification

For site 0

For site M-1

…

c(0,j)

c(M-1,j)

Label

:Basic Labels

Convexification

Label Label

c(i,j)

: Lower Convex Hull Vertices

Label

For non-convex problems, the relaxation replaces eachc(m,j) by its lower convex hull surface:


Searching Scheme of Simplex Method4. Using simplex method, there are at most three

adjacent non-zero weight basis labels:

: Zero-weight basis label : Non-zero-weight basis label

: non basis label : Continuous label

Searching for one site


Successive Relaxation Scheme

Single relaxation may miss the global optimum because of convexification effect

An intuitive scheme is to shrink the trust region and reconvexify the data in the smaller region

This scheme is found to be able to greatly improve the matching results


The Trust Region Shrinking


Successive Relaxation Scheme (An Example)

min C(1,1)+ C(2,2)+0.5|1-2|


Shape Recognition

We have to define the goodness of matching– Matching cost (M): Average difference of the

template and target image in the ROI.– Deformation (D): Affine transformation

compensated pairwise distance changes– Shape Context in the ROI (C).

Finally, we use M + a*D+b*C to quantify the matching


Random Dots Experiment

Noise: 50%Random

Disturbance: 5

Noise: 100%Random

Disturbance: 5

Noise: 50%Random

Disturbance: 10

Noise: 100%Random

Disturbance: 10


Matching Synthetic Images Results

(a): Template model showing distance transform; (b): Matching result of proposed scheme; (c): Matching result by GC; (d): Matching result by ICM. (e): Matching result by BP.

: GC : ICM : BP

: LP


Matching Leaves


Experiment Results

An example where traditional methods fail. (a): Template image;(b): Target image; (c): Edge map of template image; (d): Edge map of target image; (e): Template mesh; (f): Matching result of the proposed scheme; (g): ICM matching result; (h): Sliding template search result.


Gesture Recognition Results

Template Top match Second match


Gesture Recognition Results


Video Browsing Result


Video Browsing Result


Multiple Target Matching Results


Conclusion and Future Directions

We present a robust matching framework for human posture recognition

The method can be applied to multimedia data retrieval in image or video database, or human computer interaction applications

In future work:– We will add tempera information for behavior

recognition


Future work

– The successive reconvexification is in fact very general. It can be used to increase the robustness of many other matching schemes, such as BP and GC

– The proposed matching can be used for many other applications, such as tracking, object recongnition, motion estimation etc.


The End

Human Posture Recognition with Convex Programming Hao Jiang, Ze-Nian Li and Mark S. Drew School of Computing Science Simon Fraser University Burnaby, BC,

Documents

matching problem

j slide

hard nonconvex problem

hard problem

original problem

matching based scheme

matching surface

human postures