Top Banner
N{sgt Suzout Igvz{xk Sgxqkxrkyy Sutui{rgx 4 4 4 Vngyk O@ Zxgiqotm Vngyk OO@ Xkiutyzx{izout Iutir{youty Nusk Vgmk Zozrk Vgmk JJ II J I Vgmk7 ul 9; Mu Hgiq L{rr Yixkkt Iruyk W{oz Sgxqkxrkyy Suzout Igvz{xk {yotm Sutui{rgx \ojkuy@ G Igyk Yz{jy lux Hngxgzgtgzygs hy \oyngr Sgsgtog nzzv@55}}}4iyk4oozh4gi4ot5 |oyngrs {tjkx znk m{ojgtik ul Vxul4 Yngxgz Ingtjxgt nzzv@55}}}4iyk4oozh4gi4ot5 yngxgz P{ry2 866: 7 Iuvyxomnzi 866: \oyngr Sgsgtog P{ry >2 866:
55

Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Feb 28, 2018

Download

Documents

truongkiet
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 1 of 35

Go Back

Full Screen

Close

Quit

Markerless Motion Capture using MonocularVideos: A Case Study for Bharatanatyam

byVishal Mamania

http://www.cse.iitb.ac.in/∼ vishalmunder the guidance of

Prof. Sharat Chandranhttp://www.cse.iitb.ac.in/∼ sharat

July, 2004

1 Copyright c©2004 Vishal Mamania July 8, 2004

Page 2: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 2 of 35

Go Back

Full Screen

Close

Quit

Overview

• Introduction

• Our Approach

• Design of the System

• Results of Implementation

• Conclusions

2 Copyright c©2004 Vishal Mamania July 8, 2004

Page 3: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 3 of 35

Go Back

Full Screen

Close

Quit

Video Processing using Computer Vision

• Video contains a lot of data – in space as well as time

Page 4: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 3 of 35

Go Back

Full Screen

Close

Quit

Video Processing using Computer Vision

• Video contains a lot of data – in space as well as time

• One frame = 320×240= 76,800pixels

Page 5: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 3 of 35

Go Back

Full Screen

Close

Quit

Video Processing using Computer Vision

• Video contains a lot of data – in space as well as time

• One frame = 320×240= 76,800pixels

• Considering 30 frames per second, 10 second video contains76,800×30×10= 23,040,000pixels

Page 6: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 3 of 35

Go Back

Full Screen

Close

Quit

Video Processing using Computer Vision

• Video contains a lot of data – in space as well as time

• One frame = 320×240= 76,800pixels

• Considering 30 frames per second, 10 second video contains76,800×30×10= 23,040,000pixels

• Consider 16 million colors (24-bit) for each pixel....

• Information Overflow makes the problem difficult

• Need intelligent methods to decide

– What is important and useful?

– What is junk?

3 Copyright c©2004 Vishal Mamania July 8, 2004

Page 7: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 4 of 35

Go Back

Full Screen

Close

Quit

1. Human Motion Capture

• Process of recording human body movements to get a compactrepresentation of human skeleton and its motion

• Recovery of global position and orientation of a subject and vari-ous body parts and joints in 3D space

4 Copyright c©2004 Vishal Mamania July 8, 2004

Page 8: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 5 of 35

Go Back

Full Screen

Close

Quit

Mechanical Method of MoCap

• Exoskeleton attached to mov-able parts and joints of humanbody

• Measure the movements ofcorresponding parts and pro-duce appropriate signals

• Drawback –Heavily obstructs body move-ments

5 Copyright c©2004 Vishal Mamania July 8, 2004

Page 9: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 6 of 35

Go Back

Full Screen

Close

Quit

Marker-based Multi-camera system ofMoCap

• Done in special MoCap labs

• Dark colored clothes required

• White reflective markers at-tached to clothes of performerat joint locations

• These markers are viewedthrough multiple cameras

• Drawback –Highly tailored environment

6 Copyright c©2004 Vishal Mamania July 8, 2004

Page 10: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 7 of 35

Go Back

Full Screen

Close

Quit

Stages in Marker-based method

• Tracking of body parts

– Image processing to locate the markers’ positions in variousviewpoints

– Establish correspondences

Page 11: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 7 of 35

Go Back

Full Screen

Close

Quit

Stages in Marker-based method

• Tracking of body parts

– Image processing to locate the markers’ positions in variousviewpoints

– Establish correspondences

• Reconstruction of skeleton

– Construction of 3D structure from 2D projections

– Establish motion parameters

7 Copyright c©2004 Vishal Mamania July 8, 2004

Page 12: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 8 of 35

Go Back

Full Screen

Close

Quit

Limitations of Marker-based Method

• Expensive – Requires specialized studios, multiple cameras, etc.

Page 13: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 8 of 35

Go Back

Full Screen

Close

Quit

Limitations of Marker-based Method

• Expensive – Requires specialized studios, multiple cameras, etc.

• Intrusive – Require that clothes be of different (dark) color; mark-ers placed

Page 14: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 8 of 35

Go Back

Full Screen

Close

Quit

Limitations of Marker-based Method

• Expensive – Requires specialized studios, multiple cameras, etc.

• Intrusive – Require that clothes be of different (dark) color; mark-ers placed

• No Live Shows – Not possible for live performances

Page 15: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 8 of 35

Go Back

Full Screen

Close

Quit

Limitations of Marker-based Method

• Expensive – Requires specialized studios, multiple cameras, etc.

• Intrusive – Require that clothes be of different (dark) color; mark-ers placed

• No Live Shows – Not possible for live performances

• No Archives – Not possible to use archive videos

Page 16: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 8 of 35

Go Back

Full Screen

Close

Quit

Limitations of Marker-based Method

• Expensive – Requires specialized studios, multiple cameras, etc.

• Intrusive – Require that clothes be of different (dark) color; mark-ers placed

• No Live Shows – Not possible for live performances

• No Archives – Not possible to use archive videos

• In-house – Can’t capture outside studio

Page 17: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 8 of 35

Go Back

Full Screen

Close

Quit

Limitations of Marker-based Method

• Expensive – Requires specialized studios, multiple cameras, etc.

• Intrusive – Require that clothes be of different (dark) color; mark-ers placed

• No Live Shows – Not possible for live performances

• No Archives – Not possible to use archive videos

• In-house – Can’t capture outside studio

• Solution – Markerless Motion Captureusing a single camera

8 Copyright c©2004 Vishal Mamania July 8, 2004

Page 18: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 9 of 35

Go Back

Full Screen

Close

Quit

2. Markerless Monocular MoCap

• Develop algorithms for obtaining motion capture data in ageneralized environment

Page 19: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 9 of 35

Go Back

Full Screen

Close

Quit

2. Markerless Monocular MoCap

• Develop algorithms for obtaining motion capture data in ageneralized environment

– No special studio required

– Using a single camera

– No artificial aids like markers, calipers

– No restriction on clothes

Page 20: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 9 of 35

Go Back

Full Screen

Close

Quit

2. Markerless Monocular MoCap

• Develop algorithms for obtaining motion capture data in ageneralized environment

– No special studio required

– Using a single camera

– No artificial aids like markers, calipers

– No restriction on clothes

• Using Bharatanatyam as an example

9 Copyright c©2004 Vishal Mamania July 8, 2004

Page 21: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 10 of 35

Go Back

Full Screen

Close

Quit

New Challenges

• Image processing (Tracking) becomes difficult

• A lot of clutter needs to be removed

10 Copyright c©2004 Vishal Mamania July 8, 2004

Page 22: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 11 of 35

Go Back

Full Screen

Close

Quit

New Challenges (Contd.)

• Reconstruction too becomes difficult

• Recover the depth information lost during recording

• Creating 3D from 2D !!

• Depth values have to be valid and consistent with each other

11 Copyright c©2004 Vishal Mamania July 8, 2004

Page 23: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 12 of 35

Go Back

Full Screen

Close

Quit

Our approach

• Tracking

– Use Domain-specific knowledge

– Information about traditional dress of Bharatanatyam

Page 24: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 12 of 35

Go Back

Full Screen

Close

Quit

Our approach

• Tracking

– Use Domain-specific knowledge

– Information about traditional dress of Bharatanatyam

• Reconstruction

– Given a 2D projection, no. of possible 3D poses is finite.

– For n links (limbs), max 2n poses possible.

– Many poses impossible to achieve physically. Discard them.

– Build a weighted graph of valid poses and find minimum weightpath across the sequence. This gives the smoothest motionsequence.

12 Copyright c©2004 Vishal Mamania July 8, 2004

Page 25: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 13 of 35

Go Back

Full Screen

Close

Quit

What others have done

• Silhouette shape analysis

• Multiple camera voxel data

• Using motion library

• Factorization

13 Copyright c©2004 Vishal Mamania July 8, 2004

Page 26: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 14 of 35

Go Back

Full Screen

Close

Quit

3. Phase I: Tracking

• What we have is a grid of pixels

• What we want is the locations of joints

• Points to be considered

– Projection Model

– Human Model

– Key Feature Tracking

– Bodyparts Labeling

– Locating Endpoints (Joints)

14 Copyright c©2004 Vishal Mamania July 8, 2004

Page 27: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 15 of 35

Go Back

Full Screen

Close

Quit

Assumptions

• Only a single person (dancer) in scene

• Dancer always in the view of camera

• Background is static

• No camera motion

• Lighting changes are limited

• Distance between dancer and camera is large

15 Copyright c©2004 Vishal Mamania July 8, 2004

Page 28: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 16 of 35

Go Back

Full Screen

Close

Quit

Scaled Orthographic Projection

• A simple approximation to perspective projection

• Can be used

– When the range of depth values of a scene is small comparedto distance from camera

– Distance between object and camera is large compared to sizeof object

Z

Y

C

q1’

q2’

q3’

p3’p3

p2’p2

p1p1’

z’

16 Copyright c©2004 Vishal Mamania July 8, 2004

Page 29: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 17 of 35

Go Back

Full Screen

Close

Quit

Human Model and Key features

• We use stick-figure representation

• Represent joints as points, bones as lines

Elbow

Wrist

Neck

Shoulder

Head

Waist

Page 30: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 17 of 35

Go Back

Full Screen

Close

Quit

Human Model and Key features

• We use stick-figure representation

• Represent joints as points, bones as lines

Elbow

Wrist

Neck

Shoulder

Head

Waist

• Key features are the points to be tracked across the sequence

• All joints in the stick-figure

• Head, Shoulder, Elbow, Wrist, Waist

17 Copyright c©2004 Vishal Mamania July 8, 2004

Page 31: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 18 of 35

Go Back

Full Screen

Close

Quit

Feature TrackingWe need to track the features across the sequence and mark themaccordingly

Page 32: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 18 of 35

Go Back

Full Screen

Close

Quit

Feature TrackingWe need to track the features across the sequence and mark themaccordingly

18 Copyright c©2004 Vishal Mamania July 8, 2004

Page 33: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 19 of 35

Go Back

Full Screen

Close

Quit

Skin color model

• We use skin color model to detect these features

• According to skin color model, normalized color components ofskins of people of different races, genders, complexion are similar

• RGB components are normalized as follows

r = R(R+G+B); b = B

(R+G+B)

Page 34: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 19 of 35

Go Back

Full Screen

Close

Quit

Skin color model

• We use skin color model to detect these features

• According to skin color model, normalized color components ofskins of people of different races, genders, complexion are similar

• RGB components are normalized as follows

r = R(R+G+B); b = B

(R+G+B)

• Plot color histogram of r and bfrom distribution of skin colorof different people.

• This histogram is clustered

19 Copyright c©2004 Vishal Mamania July 8, 2004

Page 35: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 20 of 35

Go Back

Full Screen

Close

Quit

Skin Color Model (Contd.)

We approximate the colordistribution to a Gaussian modelG(m,C) with

mean, m= E{x}, where x =[

rb

]covariance, C= E{(x−m)(x−m)T}

Page 36: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 20 of 35

Go Back

Full Screen

Close

Quit

Skin Color Model (Contd.)

We approximate the colordistribution to a Gaussian modelG(m,C) with

mean, m= E{x}, where x =[

rb

]covariance, C= E{(x−m)(x−m)T}

• Likelihood of each pixel belonging to skin is given as

likelihood = P(r,b) = exp[−0.5(x−m)TC−1(x−m)]

• This number is thresholded to get the skin regions

20 Copyright c©2004 Vishal Mamania July 8, 2004

Page 37: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 21 of 35

Go Back

Full Screen

Close

Quit

Results of skin detection

Note that the golden belt in the waist region is also detected as skincolor region.21 Copyright c©2004 Vishal Mamania July 8, 2004

Page 38: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 22 of 35

Go Back

Full Screen

Close

Quit

Labeling body parts

• After morphological operations, different blobs of skin colors areformed. Only large blobs are maintained

• Blob labels are initialized in the first frame

• These are tracked using the motion factor and proximity to previ-ous frame’s blobs

• Blobs may get merged or broken

22 Copyright c©2004 Vishal Mamania July 8, 2004

Page 39: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 23 of 35

Go Back

Full Screen

Close

Quit

Fitting ellipses to blobs

• Ellipses fitted around the boundary of blobs, using algebraic fitalgorithm

• Endpoints of major axis are endjoints of limbs

23 Copyright c©2004 Vishal Mamania July 8, 2004

Page 40: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 24 of 35

Go Back

Full Screen

Close

Quit

Shoulders’ Position

• Observation – In most cases, except when the body is tilted, theposition of the shoulders is exactly above the waist region end-points and in horizontal line with the lower end of the neck.

• Needs improvement.

• We currently use manual adjustment.

24 Copyright c©2004 Vishal Mamania July 8, 2004

Page 41: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 25 of 35

Go Back

Full Screen

Close

Quit

4. Phase II: Reconstruction

• What we have is 2D projections of joints in all frames

• What we want is 3D positions of joints in all frames

25 Copyright c©2004 Vishal Mamania July 8, 2004

Page 42: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 26 of 35

Go Back

Full Screen

Close

Quit

Calculating depth

• Observation – Given a sufficiently long sequence, each link be-comes parallel or nearly parallel at least once.

• Using anthropometric data to adjust the length values

• Given the 2D length & 3D length, depth of link can be calculatedusing basic trigonometry.

b

c

zx

y

a

2D projectionof the line

Actual Line

26 Copyright c©2004 Vishal Mamania July 8, 2004

Page 43: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 27 of 35

Go Back

Full Screen

Close

Quit

Reflective Orthographic Ambiguity

• For each link, there are two possibilities of z-values.

• One endpoint can be in front or in rear of the other endpoint.

1P (x, y, z)P (x, y, −z)2

Rear Front

Reference Plane

27 Copyright c©2004 Vishal Mamania July 8, 2004

Page 44: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 28 of 35

Go Back

Full Screen

Close

Quit

Pose Generation

• For each link, two possibilities =⇒ 2n possibilities for n links.

front rear

rearfront

rear

rearfront

Right Shoulder

Left Shoulder

Neck

front

Page 45: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 28 of 35

Go Back

Full Screen

Close

Quit

Pose Generation

• For each link, two possibilities =⇒ 2n possibilities for n links.

front rear

rearfront

rear

rearfront

Right Shoulder

Left Shoulder

Neck

front

• Not all of 2n poses are physically attainable.

• Need to put constraints to filter out impossible poses.

28 Copyright c©2004 Vishal Mamania July 8, 2004

Page 46: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 29 of 35

Go Back

Full Screen

Close

Quit

Body Constraints

• Joint Angle Limits

– Each joint of body has a maximum and a minimum limit ofangle of bend.

Page 47: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 29 of 35

Go Back

Full Screen

Close

Quit

Body Constraints

• Joint Angle Limits

– Each joint of body has a maximum and a minimum limit ofangle of bend.

• Collision Constraints

– One body part cannot penetrate through another part.

– Find distance between different links, they should be less thatsum of corresponding radii.

29 Copyright c©2004 Vishal Mamania July 8, 2004

Page 48: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 30 of 35

Go Back

Full Screen

Close

Quit

Graph Formulation

• What we have is a set of valid poses for each frame.

Page 49: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 30 of 35

Go Back

Full Screen

Close

Quit

Graph Formulation

• What we have is a set of valid poses for each frame.

• What we want is a valid pose sequence across frames.

Page 50: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 30 of 35

Go Back

Full Screen

Close

Quit

Graph Formulation

• What we have is a set of valid poses for each frame.

• What we want is a valid pose sequence across frames.

• We create a layered graph to model this situation.

• One layer for each frame.

• Each valid pose for a frame is represented as a node in thecorresponding layer.

• Edges are put between nodes in adjacent layers of transitionbetween those poses is possible.

30 Copyright c©2004 Vishal Mamania July 8, 2004

Page 51: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 31 of 35

Go Back

Full Screen

Close

Quit

Graph Formulation (Contd.)

Frame 1 Frame 2 Frame 3 Frame 4 Frame N

• Assign weights to edges

• Find minimum weight path from first to last layer, which gives op-timal path

31 Copyright c©2004 Vishal Mamania July 8, 2004

Page 52: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 32 of 35

Go Back

Full Screen

Close

Quit

Calculating Weights

• For jerk free motion, the change in angles, velocities should be assmooth as possible.

• Weight represent the difficulty of transition from one pose to an-other.

• Various possibilities tested

– Change in depths of joints

– Change in angles at joints

– Change in velocities

– Estimation based on velocity

• Last method produces the best results.

32 Copyright c©2004 Vishal Mamania July 8, 2004

Page 53: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 33 of 35

Go Back

Full Screen

Close

Quit

Final Results

33 Copyright c©2004 Vishal Mamania July 8, 2004

Page 54: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 34 of 35

Go Back

Full Screen

Close

Quit

5. Conclusions

• Captured upper body motion for Bharatanatyam sequence

• Tracking done using the domain-specific knowledge

• Tracking is not completely automatic. We need some manual in-tervention for blobs labeling and final positions

• Reconstruction done using a graph-based approach

• Reconstruction produces accurate results in majority of frames

34 Copyright c©2004 Vishal Mamania July 8, 2004

Page 55: Markerless Motion Capture using Monocular Videos: A Case ...sharat/talks/vishalm.pdf · Human Motion Capture Markerless Monocular... Phase I: Tracking Phase II: Reconstruction Conclusions

Human Motion Capture

Markerless Monocular . . .

Phase I: Tracking

Phase II: Reconstruction

Conclusions

Home Page

Title Page

JJ II

J I

Page 35 of 35

Go Back

Full Screen

Close

Quit

THANK YOU !!

35 Copyright c©2004 Vishal Mamania July 8, 2004