Morphable Model - Yonseiweb.yonsei.ac.kr/hgjung/Lectures/AUE859/10. Morphable... · 2014. 12. 29. · - Automated 3D alignment of the faces with 3D-3D Absolute Orientation - Semiautomatic

E-mail: hogijung@hanyang.ac.krhttp://web.yonsei.ac.kr/hgjung

MorphableMorphable ModelModel

Graphics and Vision Research Group, University of Basel, http://gravis.cs.unibas.ch/

Volker Blanz

and Thomas Vetter, “Face Recognition Based on Fitting a 3D Morphable

Model,” IEEE PAMI, Vol. 25, No. 9, Sep. 2003, pp. 1063-1074.

SourcesSources

MorphableMorphable Model based Face Recognition SystemModel based Face Recognition System

= Deformable 3D Model= Deformable 3D Model

+ Computer Graphics Simulation of Projection and Illumination+ Computer Graphics Simulation of Projection and Illumination

Intrinsic shape and texture are fully independent of extrinsic parameter

MorphableMorphable ModelModel

- High dimensional vector of shape

- High dimensional vector of texture

- Probability density function of natural faces within face space

MorphableMorphable

ModelModel

Paradigm 1. (The approach taken in this paper)Recognition can be based on model coefficients, which represent intrinsic shape and texture of faces, and are independent of the imaging conditions

Paradigms for Model Based RecognitionParadigms for Model Based Recognition

Paradigm 2.

Viewpoint dependentRecognition System

SyntheticViews

Original (PIE images)

Reconstructions renderedInto the originals

Same reconstructions renderedwith standard illumination

Paradigms for Model Based RecognitionParadigms for Model Based Recognition

- Vector space representation of faces

- Convex combination of shape and texture vectors Si and Ti

A A MorphableMorphable

Model of 3D FacesModel of 3D Faces

Database of Three-Dimensional Laser Scans- 100 males, 100 females, aged between 18 and 45- Recorded with a Cyberware 3030PS laser scanner(512 angular steps, 512 vertical steps = 0.615mm)

Preprocessing of Raw Scans- Filling holes and removing spikes with an interactive tool- Automated 3D alignment of the faces with 3D-3D Absolute Orientation- Semiautomatic trimming along the edge of a bathing cap- Vertical, planar cut behind the ears and a horizontal cut at the neck

Constructing Constructing MorphableMorphable

Model from 3D ScansModel from 3D Scans

- Correspondence between each face and a reference face is given bya dense vector field

Correspondence Based on Optic Flow“Objects in image sequences I(x,y,t) retain their brightnesses as they moveacross the image at a velocity (vx ,vy )T”

- Minimizing the following expression

- v is assumed to be constant on each neighborhood R(x0 ,y0 ) 5x5- In each point (x0 ,y0 ), v(x0 ,y0 ) can be found by solving 2x2 linear system

Definition of Face Vectors

- Our reference face is a triangular mesh with 75,972 vertices

vertices of this mesh be located at

- The definition of shape and texture vectors is based on a reference face I0

- To encode a novel scan I

1) Compute the flow field from I0 to I

2) Convert to Cartesian coordinates

3) Coordinates and color values for the shape and

texture vectors S and T are then sampled at

Principal Component Analysis

- Compute the eigenvectors s1 ,s2 , … by SVD- The eigenvalues of C are the variances of the data along each eigenvector- Eigenvectors form an orthogonal basis

- PCA provides an estimate of the probability density within face space :

Segments- To generate a larger variety of different faces linear combinations of shape and texture are formed separately for different regions of the face : eyes, nose, mouth and the surrounding area

- For continuous transitions between the segments, we apply a modification of the image blending technique

- Estimate shape coefficient ai and texture coefficient bi

relevant parameters of the 3D scene(pose, focal length of the camera, light intensity, color and direction)

- In an analysis-by-synthesis loop, the algorithm finds model parametersand scene parameters such that the model, rendered by computer graphicsalgorithms, produces an image as similar as possible to the input image

ModelModel--Based Image AnalysisBased Image Analysis

- For initialization, the systemcurrently requires image coordinatesof about seven facial feature points,such as the corners of the eyes orthe tip of the nose

- With an interactive tool, the userdefines these points

ModelModel--Based Image AnalysisBased Image Analysis

Image Positions of Vertices- Object-centered coordinates

Illumination and Color- The illumination model of Phong approximately describes the diffuse and

specular reflection of a surface

Image SynthesisImage Synthesis

Optimizing1)2)3) 22 rendering parameters

--------

Fitting the Model to an ImageFitting the Model to an Image

Cost Function- Minimize the sum of square differences

- Minimize feature point error

Maximum A Posteriori estimator

- Ad hoc choices of and are used to control the relative weights- At the beginning, prior probability end EF are weighted high

The final iterations put more weight on EI and no longer rely on EF

Optimization ProcedureThe core of the fitting procedure is a minimization of the cost functionwith a stochastic version of Newton’s method- Select a set K of 40 random triangles in each iteration

Evaluate EI and its gradient only at their centers- Set the probability of selecting a particular triangle proportional to its area- Area calculation once every 1,000 iterations

The optimum is

- Only consider ai

- These second derivatives are computed by numerical differentiation fromthe analytically calculated first derivatives, based on 300 random vertices,at the beginning of the optimization and once every 1,000 iterations

- In each iteration, we perform small steps

Reconstructions of 3D shape and texture from FERET images

Results of Model FittingResults of Model Fitting

Reconstructions of 3D shape and texture from CMU-PIE images

- Illumination in the third image is not fully recovered, so part of thereflections are attributed to texture.

- The error of pose estimates is within a few degrees

Similarity Measure

1) Mahalanobis distance :2) Cosine of the between angle :3) CW distance :

- Overall performance is best for the side-view gallery

Recognition PerformanceRecognition Performance

1% false alarm rate- CMU-PIE : hit rate 77.5%- FERET : hit rate 87.9%

Morphable Model - Yonseiweb.yonsei.ac.kr/hgjung/Lectures/AUE859/10. Morphable... · 2014. 12. 29. · - Automated 3D alignment of the faces with 3D-3D Absolute Orientation - Semiautomatic

Documents

A Multiresolution 3D Morphable Face Model and Fitting...

CONSTRUCTING A 3D MORPHABLE FACE FROM A SINGLE …

A 3D morphable eye region model for gaze...

A Multiresolution 3D Morphable Face Model and Fitting...

Advances and Challenges in 3D and 2D+3D Human Face … ·.....

A 3D Morphable Model Learnt From 10,000...

Face Poser: Interactive Modeling of 3D Facial Expressions...

A 3D Morphable Model learnt from 10,000...

A Morphable Model For The Synthesis Of 3D...

Towards High-Fidelity Nonlinear 3D Face Morphable...

Sparse Photometric 3D Face Reconstruction Guided by...

A Morphable Model For The Synthesis Of 3D...

Large Scale 3D Morphable Models - Clemson...

A Morphable Model For The Synthesis Of 3D...

Learning Formation of Physically-Based Face Attributes ·.....

3D Face Morphable Models “In-the-Wild” · 3D Face...