3d Pose Detection

3d Pose Detection• Used by Kinect• Accurate when the pose closely matches a stored pose• Inaccurate when novel poses are made• Can often produce shaky movement due to pose snapping

3d Pose Tracking• Calculate poses based on previous poses and current data• No datasets required• Has issues with local minima

Hybrid

• Store a collection of preprocessed poses• Start session with a detected pose• Track until a failure is hit• Detect that frame, and continue

tracking from there

Pose Tracking in Detail• Given the current set of pixels, and the calculated poses of the past

few frames, calculate the most probable current pose• Repeat the following until error is reduced• Calculate a hypothesized 3d representation (rendered image) using the 3d

camera image (observed data)• Calculate quite a lot about the rendered image:

• Edepth, Eextra, Esilhouette, Eprior

• Adjust the hypothesized pose accordingly• If that fails significantly, “detect” the current pose and try again

Optimizing the Position• Solve energy minimization problems:• Edepth: Depth data, excluding any rendered foreground pixels that match

background observed pixels• Eextra: The above depth term, but with added calculations where rendered

pixels don’t match with observed pixels• Esilhouette: The difference in silhouettes (pixel is in foreground of one image but

the background of the other)• Eprior: Evaluation of the rendered pose based on the previous rendered poses

More Pose Detecting• Grab a random subset of pixels and match them to known poses• Throw out uncertain data• 20k poses total

Setup• User must stand in a specific pose for calibration• Select random subset of pixels to determine widths of limbs

Results

Results• Roughly 1 in 200 frames

reinitialized by the detector• Very fast movement often

causes local minima• On average, 1 failure every

33 seconds• High performance running in

parallel on the GPU

http://www.youtube.com/watch?v=ICFKEOk3SyA&feature=youtu.be

Further Work• Use skinned mesh models• Increase pose set• Use color data in the

algorithm as well