3D HUMAN BODY POSE ESTIMATION BY SUPERQUADRICS

3D HUMAN BODY POSE ESTIMATION BY 3D HUMAN BODY POSE ESTIMATION BY SUPERQUADRICS SUPERQUADRICS

26/02/2012 1/21

Ilya Afanasyev, Massimo Lunardelli, Nicolo' Biasi, Luca Baglivo, Mattia Tavernini, Francesco Setti and Mariolino De Cecco

Department of Mechanical and Structural Engineering (DIMS),

Mechatronics Lab.

EU-FP7-Marie Curie COFUND-Trentino Project N° 226070

ContentContent

2/21

1.1. IntroductionIntroduction2.2. The input data description The input data description 3.3. The algorithm descriptionThe algorithm description4.4. DemoDemo of of Test ResultsTest Results5.5. Conclusions Conclusions

26/02/2012

3/2126/02/2012

3D point cloud3D point cloud

Preprocessing: Preprocessing: segmentationsegmentation

Video-frames from Video-frames from multicamera systemmulticamera system

Fitting SQ Fitting SQ to 3D datato 3D data

Final Human Final Human Body pose modelBody pose model

IntroductionIntroductionWe present the 3D reconstruction and human body pose We present the 3D reconstruction and human body pose estimation system using Superquadrics (SQ) math. model and estimation system using Superquadrics (SQ) math. model and RANSAC search with a least square fitting & verifying RANSAC search with a least square fitting & verifying algorithms.algorithms.

Input data from VERITAS projectHuman Body pose estimation

algorithm under consideration

Our starting pointOur starting point

4/21

We use the multiple stereo system (8 pairs of cameras) and the We use the multiple stereo system (8 pairs of cameras) and the garment with the special clothing marks to recover 3D human body garment with the special clothing marks to recover 3D human body surface with superimposed colored markers.surface with superimposed colored markers.

26/02/2012

The multicamera system and garment belong to EU\FP7-FP7-ICT – VERITAS project: http://veritas-project.eu/

SegmentationSegmentation

5/21

The multicamera system and garment belong to EU\FP7-FP7-ICT – VERITAS project: http://veritas-project.eu/26/02/2012

The segmentation is based on clothing analysis (i.e. recognition of the The segmentation is based on clothing analysis (i.e. recognition of the special clothing marks on the garment) and divides the Human Body into special clothing marks on the garment) and divides the Human Body into 9 parts (body, arms, forearms, hips and legs). The garment doesn’t have a 9 parts (body, arms, forearms, hips and legs). The garment doesn’t have a hood, so our Human Body SQ-model doesn’t have the head.hood, so our Human Body SQ-model doesn’t have the head.

What is the input data? What is the input data?

6/21

The multicamera system and garment belong to EU\FP7-FP7-ICT – VERITAS project: http://veritas-project.eu/26/02/2012

3D video of Human Body movement has been captured from a multi-3D video of Human Body movement has been captured from a multi-camera system and consisted of camera system and consisted of 119 frames119 frames.. 3D data processed offline separately for every frame and concludes 3D data processed offline separately for every frame and concludes 3D coordinates of appr. 3D coordinates of appr. 2100 datapoints 2100 datapoints of the Human Body pose.of the Human Body pose.3D data points are accompanied with segmentation matrix, the 3D data points are accompanied with segmentation matrix, the elements of which set belonging of every point to the body or definite elements of which set belonging of every point to the body or definite limb. As the result of the clothing segmentation we have approximately limb. As the result of the clothing segmentation we have approximately 800 800 datapoints of the body, datapoints of the body, 30-7030-70 points of left/right arms, points of left/right arms, 15-2515-25 points points of forearms, of forearms, 300-600300-600 points of hips, and points of hips, and 80-15080-150 points of legs. points of legs.

7/2126/02/2012

What is the proposed methodWhat is the proposed method??We propose using the We propose using the hierarchical hierarchical RANSAC-based model-fitting RANSAC-based model-fitting technique technique with a with a composite SQ model composite SQ model of human body of human body (HB) and limbs. SQ models permit to (HB) and limbs. SQ models permit to describe complex-geometry objects with describe complex-geometry objects with few parameters and generate simple few parameters and generate simple minimization function to estimate an object minimization function to estimate an object pose. We assume shape and dimensions of pose. We assume shape and dimensions of the body and limbs are known the body and limbs are known a-prioria-priori with with correct anthropometric parameterscorrect anthropometric parameters in the in the metric coordinate system. metric coordinate system. The algorithm recovers 3D position of the The algorithm recovers 3D position of the body as the largest object (“body as the largest object (“Body Pose Body Pose SearchSearch”) and then restores the human ”) and then restores the human limbs poses (“limbs poses (“Limbs Pose SearchLimbs Pose Search”). To cope ”). To cope with measurement noise and outliers, the with measurement noise and outliers, the object pose is estimated by RANSAC-SQ-object pose is estimated by RANSAC-SQ-fitting technique. We control the fitting fitting technique. We control the fitting quality by setting quality by setting inlier thresholds inlier thresholds for limbs for limbs and body.and body.

RANSAC Body Fitting & Body Pose Estimation

Is number of inliers higher than

threshold?

RANSAC Limb Fitting & Limb Pose Estimation

Is number of inliers higher than

threshold?

Body PoseBody Pose

Repeat for every 4 limbsRepeat for every 4 limbs

Inliers, Body PoseInliers, Body Pose

Inliers, Limb PoseInliers, Limb Pose

Human Body PoseHuman Body Pose

NoNo

NoNo

YesYes

YesYes

Preprocessing: Human Body Segmentation

3D Cloud of Points3D Cloud of PointsBody Model in SuperquadricsBody Model in Superquadrics

Limb Model in SuperquadricsLimb Model in Superquadrics

Limbs Pose Search

Limbs Pose Search

Body Pose Search

Body Pose Search

Set threshold

Set threshold

Set threshold

Set threshold

HB Pose Estimation HB Pose Estimation algorithmalgorithm

8/21

RANSAC Body Fitting & Body Pose Estimation

Is number of inliers higher than

threshold?

RANSAC Limb Fitting & Limb Pose Estimation

Is number of inliers higher than

threshold?

Body PoseBody Pose

Repeat for every 4 limbsRepeat for every 4 limbs

Inliers, Body PoseInliers, Body Pose

Inliers, Limb PoseInliers, Limb Pose

Human Body PoseHuman Body Pose

NoNo

NoNo

YesYes

YesYes

Preprocessing: Human Body Segmentation

3D Cloud of Points3D Cloud of PointsBody Model in SuperquadricsBody Model in Superquadrics

Limb Model in SuperquadricsLimb Model in Superquadrics

Limbs Pose Search

Limbs Pose Search

Body Pose Search

Body Pose Search

Set threshold

Set threshold

Set threshold

Set threshold

26/02/2012

Threshold for body 55%Threshold for body 55%

Threshold for limbs 60%Threshold for limbs 60%

9/2126/02/2012

Human Body model in SuperquadricsHuman Body model in SuperquadricsWe present Human Body (HB) as a model We present Human Body (HB) as a model in 9 superquadrics – superellipsoids.in 9 superquadrics – superellipsoids.

AbbreviationsAbbreviations: : B – body, – body, LA/RA – Left/Right Arms, – Left/Right Arms, LF/RF – Left/Right Forearms, – Left/Right Forearms, LH/RH – Left/Right Hips, – Left/Right Hips, LL/RL – Left/Right Legs. – Left/Right Legs. LS – Left Shoulder, – Left Shoulder, E – Elbow,Elbow, LHJ – Left Hip Joint, – Left Hip Joint, ηLA – angle position of Left Shoulder,angle position of Left Shoulder, K – Knee, etc.– Knee, etc.

HB anthropometric parameters:HB anthropometric parameters: the shape parameters the shape parameters εε11 = = εε22 = 0.5; = 0.5;

the scaling parameters: the scaling parameters:→ → Body: Body: aa11 = 0.095, = 0.095, aa22 = 0.18, = 0.18,

aa33 = 0.275 (m). = 0.275 (m).

→ → Arms: Arms: aa11 = = aa33 = 0.055, = 0.055, aa22 = 0.15 (m). = 0.15 (m).

→ → Forearms: Forearms: aa11 = = aa33 = 0.045, = 0.045,

aa22 = 0.13 (m). = 0.13 (m).

→ → Hips: Hips: aa11 = = aa22 = 0.075, = 0.075, aa33 = 0.2 (m). = 0.2 (m).

→ → Legs: Legs: aa11 = = aa22 = 0.05, = 0.05, aa33 = 0.185 (m). = 0.185 (m).

10/2126/02/2012

Human Body model in SuperquadricsHuman Body model in Superquadrics

The explicit form of the parametric equation of the superquadrics, The explicit form of the parametric equation of the superquadrics, which is usually used for SQ representation and visualization, iswhich is usually used for SQ representation and visualization, is

,

sin)(sin

sin)(sincos)(cos

cos)(coscos)(cos

4

54

54

3

2

1

a

aa

aa

signuma

signumsignuma

signumsignuma

z

y

x

The implicit equation of superquadrics is used The implicit equation of superquadrics is used for mathematical modeling to do fitting 3D data: for mathematical modeling to do fitting 3D data:

;2/2/ .

where x,y,z - superquadric system coordinates; where x,y,z - superquadric system coordinates; η, ω – spherical coordinates; η, ω – spherical coordinates; a1, a2, a3 – the scaling parameters; a1, a2, a3 – the scaling parameters; a4, a5 – the shape parameters.a4, a5 – the shape parameters.

Cz

CyCx

11/2126/02/2012

Body model in SuperquadricsBody model in SuperquadricsThe position of Human Body is defined by the following rotation & The position of Human Body is defined by the following rotation & translation sequences of the Body Superquadrics:translation sequences of the Body Superquadrics:

The rotation matrix of BODY The rotation matrix of BODY RRBODYBODY is: is:

1. Translation of center of 1. Translation of center of BODY (xc, yc, zc), along x, y, z-coordinates.along x, y, z-coordinates.2. Rotation 2. Rotation α α among x (clockwise).among x (clockwise).3. Rotation 3. Rotation β β among y (clockwise).among y (clockwise).4. Rotation 4. Rotation γ γ among z (clockwise).among z (clockwise).

1000

0100

00cossin

00sincos

1000

0cos0sin

0010

0sin0cos

1000

0cossin0

0sincos0

0001

BODYR

The transformation matrix of BODY The transformation matrix of BODY RRBODYBODY is: is:

.

1000

100

010

001

c

c

c

BODYBODY z

y

x

RT

12/2126/02/2012

Limb models in SuperquadricsLimb models in SuperquadricsThe position of Left Shoulder according to the center of the body The position of Left Shoulder according to the center of the body coordinate system is estimated by SQ explicit equation:coordinate system is estimated by SQ explicit equation:

The transformation Left Shoulder - Left Arm The transformation Left Shoulder - Left Arm (LS-LA) can be expressed with the following (LS-LA) can be expressed with the following rotation & translation sequences:rotation & translation sequences:1. Rotation α among x (clockwise).1. Rotation α among x (clockwise).2. Rotation β among z (anticlockwise).2. Rotation β among z (anticlockwise).3. Rotation γ among y (clockwise).3. Rotation γ among y (clockwise).4. Translation of SQ center on distance a4. Translation of SQ center on distance a22

along y.along y.

B

LA

LFLA

B

LA

LALAB

S

a

aPP

1

1

sin

cos

0

2,

3

2

E

z

y

LS

z

y

1

1

sin)(sin

cos)(cos

0

2,

3

2

signuma

signumaPP LAB

S

where where RRLALA is the rotation matrix of Left Arm is the rotation matrix of Left Arm

.

1000

0cos0sin

0010

0sin0cos

1000

0100

00cossin

00sincos

1000

0cossin0

0sincos0

0001

LAR

,

1000

0100

010

0001

,, 2

a

RTT LALALALALS

LALS

LA

13/2126/02/2012

Limb models in SuperquadricsLimb models in SuperquadricsThe full transformation for every point of system “Body - Left Forearm” The full transformation for every point of system “Body - Left Forearm” (B-LF) can be calculated this way:(B-LF) can be calculated this way:

where where PPBB, , PPLFLF - coordinates of Body and Left - coordinates of Body and Left Forearm points correspondingly.Forearm points correspondingly.

The transformations: Body - Left The transformations: Body - Left Shoulder (Shoulder (B-LSB-LS) and Left Arm - ) and Left Arm - Elbow (Elbow (LA-ELA-E) are: ) are:

.

,1 BE

LFLA

ELS

LAB

LSLF

LFELF

LAE

LSLA

BLS

B

PTTTTP

PTTTTP

B

LA

LFLA

B

LA

LALAB

S

a

aPP

1

1

sin

cos

0

2,

3

2

E

z

y

LS

BPLFP

z

y

.

1000

100

010

001

B

SBLS

PT .

1000

0100

010

0001

2

a

T LAE

The transformation Elbow - Left The transformation Elbow - Left Forearm (Forearm (E-LFE-LF) is created by:) is created by:1. Rotation 1. Rotation δδLFLF among among xx (clockwise). (clockwise).

2. Translation of SQ center on 2. Translation of SQ center on -a-a22

along along yy..

.

1000

0cossin0

sincos0

00011

2

a

TT LFE

LFE

LF

14/2126/02/2012

RANSAC Body Pose SearchRANSAC Body Pose SearchWe use RANSAC ("RANdom SAmple Consensus") algorithm to find the We use RANSAC ("RANdom SAmple Consensus") algorithm to find the body pose hypothesis, i.e. 6 variables: 3 angles of rotation (body pose hypothesis, i.e. 6 variables: 3 angles of rotation (αα, , ββ, , γγ) and 3 ) and 3 translation coordinates (translation coordinates (xxCC, , yyCC, , zzCC).).

Having these variables we can calculate the Having these variables we can calculate the transformation matrix transformation matrix TTBODYBODY . We are fitting a . We are fitting a

model described by the superquadric implicit model described by the superquadric implicit equation to 3D data of the body. We are taking 6 equation to 3D data of the body. We are taking 6 points in the world coordinate system (points in the world coordinate system (xxWiWi, , yyWiWi, ,

zzWiWi) from appr. 800 data points of the body and ) from appr. 800 data points of the body and

transform them to the SQ centered coordinate transform them to the SQ centered coordinate system (system (xxSiSi, , yySiSi, , zzSiSi), using), using

Then we are calculating the Then we are calculating the inside-outside functioninside-outside function according to the according to the superquadric implicit equation in world coordinate system:superquadric implicit equation in world coordinate system:

SzSy

Sx

6s

,

1

),,( 1

i

i

i

iiii

w

w

w

BODYssss z

y

x

TzyxF

.)()()( 1

1

2

22

2

3

2

2

2

1

a

zF

a

yF

a

xFF iii

i

ssssssw

15/2126/02/2012

RANSAC Body Pose SearchRANSAC Body Pose SearchThe The inside-outside functioninside-outside function has 11 parameters: has 11 parameters:

SzSy

Sx

6s

),,,,,,,,,,,,,,( 21321 cccWWWw zyxaaazyxFFiiii

where 5 parameters are known (where 5 parameters are known (aa11,, a a22, , aa33, , εε11, , εε22) )

and 6 parameters (and 6 parameters (αα, , ββ, , γγ,, x xCC, , yyCC, , zzCC) should be ) should be

found by minimizing the cost-function:found by minimizing the cost-function:

,1)(min2

1

2 1

i

Ww

s

iii

FFxF

Thus we are fitting SQ model to random dataset by Thus we are fitting SQ model to random dataset by minimizing the inside-outside function of distance to minimizing the inside-outside function of distance to SQ surface. We used both the “Trust-Region SQ surface. We used both the “Trust-Region algorithm” and “Levenberg-Marquardt algorithm” in algorithm” and “Levenberg-Marquardt algorithm” in the nonlinear least-square minimization method. the nonlinear least-square minimization method.

After that we are evaluating number of inliers by comparing the After that we are evaluating number of inliers by comparing the distances between every point of 3D point cloud and SQ model with a distances between every point of 3D point cloud and SQ model with a distance threshold distance threshold tt (to accelerate the calculations we took the distance (to accelerate the calculations we took the distance threshold threshold t = 2 cmt = 2 cm):): .1

2

3211

iwi Faaad

16/2126/02/2012

RANSAC Limb Pose SearchRANSAC Limb Pose SearchAnalogically to Body Pose search, we are realizing RANSAC Limb Analogically to Body Pose search, we are realizing RANSAC Limb Pose Search. The main differences between RANSAC Body and Pose Search. The main differences between RANSAC Body and Limb Fitting are:Limb Fitting are:

3s

3s in using SQ pairs of limbs: arm-forearms, in using SQ pairs of limbs: arm-forearms, hips-legs.hips-legs. in picking up in picking up ss = 3 points for every limb = 3 points for every limb (although we use the body transform matrix (although we use the body transform matrix TTBODYBODY, obtained from the Body Pose Search). , obtained from the Body Pose Search).

in using 4 variables for Limbs Pose Search: in using 4 variables for Limbs Pose Search: 4 angles of rotation (4 angles of rotation (αα, , ββ, , γγ, , δδ).). in minimizing the joint cost-function of SQ in minimizing the joint cost-function of SQ pair together, considering two limbs pair together, considering two limbs simultaneously:simultaneously:

,11)(min2

)()(1

2 11

LF

i

LA

iW

LFwLAw

s

iii

FFFxF

where abbreviations where abbreviations LALA and and LFLF mean Left Arm (LA) and Forearm mean Left Arm (LA) and Forearm (LF) Limbs correspondingly (as an example).(LF) Limbs correspondingly (as an example).

17/2126/02/2012

Demo of Test ResultsDemo of Test Results

At the top: left – a pose of a human in the garment, right – “cloud of At the top: left – a pose of a human in the garment, right – “cloud of points”. At the bottom: left – the result of RANSAC-fitting to 3D data points”. At the bottom: left – the result of RANSAC-fitting to 3D data (pink points – inliers, cyan – outliers), right – final pose estimation.(pink points – inliers, cyan – outliers), right – final pose estimation.

For most of 3D video frames, the amount of inliers is more than 65%.For most of 3D video frames, the amount of inliers is more than 65%.

18/2126/02/2012

Demo of Test ResultsDemo of Test Results

At the top: left – a pose of a human in the garment, right – “cloud of At the top: left – a pose of a human in the garment, right – “cloud of points”. At the bottom: left – the result of RANSAC-fitting to 3D data points”. At the bottom: left – the result of RANSAC-fitting to 3D data (pink points – inliers, cyan – outliers), right – final pose estimation.(pink points – inliers, cyan – outliers), right – final pose estimation.

For most of 3D video frames, the amount of inliers is more than 65%.For most of 3D video frames, the amount of inliers is more than 65%.

19/2126/02/2012

Demo of Test ResultsDemo of Test ResultsThe lack of data points for arms and forearms gives the displacements of The lack of data points for arms and forearms gives the displacements of the upper limb poses from one video frame to other. It spoils the the upper limb poses from one video frame to other. It spoils the impression from the Human Body movement when preparing video impression from the Human Body movement when preparing video collecting together the individual frames processed by RANSAC-SQ-collecting together the individual frames processed by RANSAC-SQ-fitting.fitting.

This problem can be solved in future by correcting 3D Human Body Pose This problem can be solved in future by correcting 3D Human Body Pose Estimation algorithm, or improving 3D data point acquisition process, or Estimation algorithm, or improving 3D data point acquisition process, or using other sensor and segmentation techniques.using other sensor and segmentation techniques.

20/2126/02/2012

ConclusionsConclusions 3D real data of Human Body was obtained by a multi-camera system 3D real data of Human Body was obtained by a multi-camera system and structured by the special clothing analysis.and structured by the special clothing analysis. The human body was modeled by a composite SuperQuadric (SQ) The human body was modeled by a composite SuperQuadric (SQ) model presenting body and limbs with correct a-priori known model presenting body and limbs with correct a-priori known anthropometric dimensions. anthropometric dimensions. The proposed method based on hierarchical RANSAC-object search The proposed method based on hierarchical RANSAC-object search with a robust least square fitting SQ model to 3D data: at first the body, with a robust least square fitting SQ model to 3D data: at first the body, then the limbs. then the limbs. The solution is verified by evaluating the matching score (the number of The solution is verified by evaluating the matching score (the number of inliers corresponding to a-piori chosen distance threshold), and comparing inliers corresponding to a-piori chosen distance threshold), and comparing this score with admissible inlier threshold for the body and limbs. this score with admissible inlier threshold for the body and limbs. For most of 3D video frames, we achieve the amount of inliers is more For most of 3D video frames, we achieve the amount of inliers is more than 65% that means that algorithm works well.than 65% that means that algorithm works well. This method can be useful for applications dealt with 3D Human Body This method can be useful for applications dealt with 3D Human Body recognition, localization and pose estimation. recognition, localization and pose estimation. This method will also work with any 3D point cloud data acquired by This method will also work with any 3D point cloud data acquired by other sensors and segmented using any other algorithms.other sensors and segmented using any other algorithms.

AcknowledgementsAcknowledgements

21/21

Ilya Afanasyev worked under creation of the Ilya Afanasyev worked under creation of the algorithms for 3D object recognition and pose algorithms for 3D object recognition and pose estimation by support from estimation by support from EU\FP7-Marie EU\FP7-Marie Curie-COFUND – Trentino programCurie-COFUND – Trentino program..

3D data acquisition and segmentation were 3D data acquisition and segmentation were executed by UniTN team in the framework of executed by UniTN team in the framework of project VERITAS funded by FP7, EU.project VERITAS funded by FP7, EU.

The authors are very grateful to colleagues The authors are very grateful to colleagues from Mechatronics dep., University of Trento from Mechatronics dep., University of Trento (UniTN), namely Alberto Fornaser.(UniTN), namely Alberto Fornaser.

Grazie!!26/02/2012