Akshay Asthana, Jason Saragih, Michael Wagner and Roland G öcke ANU, CMU & U Canberra In part funded by ARC grant TS0669874.

Evaluating AAM Fitting Methods for Facial Expression Recognition

Akshay Asthana, Jason Saragih, Michael Wagner and Roland Göcke

ANU, CMU & U Canberra

In part funded by ARC grant TS0669874

BackgroundThinking Head project

http://thinkinghead.edu.au/5-year multi-institution

(Canberra, UWS, Macquarie, Flinders) project in Australia

Develop a research platform for human communication sciences

“An Approach for Automatically Measuring Facial Activity in Depressed Subjects”, McIntyre, Göcke, Hyett, Green, Breakspear, ACII 2009

Aim for this StudyActive Appearance Models

(AAM) have become a popular tool for markerless face tracking in recent years

A number of different AAM fitting methods exist

Which one should we use?We wanted to evaluate these in the context of facial expression recognition (FER)How well do AAMs generalise?How robust are these methods w.r.t. initialisation

error?How does their fitting accuracy affect the FER

accuracy?

AAMShape:

Texture:

AAM – Shape VariationShape variation

Mean

AAM – Texture VariationTexture variation

Mean

AAM – Modelling AppearanceAppearance = Shape + Texture

Mean

AAM (cont.)Alignment based on finding model parameters

that iteratively fit learnt model to the image

Initialisation After 5 iterations Converged

AAM Fitting Methods Compared in this StudyFixed Jacobian (FJ): Cootes, Edwards & Taylor, 1998

Project-Out Inverse Compositional (POIC): Baker & Matthews, 2001

Simultaneous Inverse Compositional (SIC): Baker, Gross & Matthews, 2003

Robust Inverse Compositional (RIC): Gross, Matthews & Baker, 2005

Iterative Error-Bound Minimisation (IEBM), aka Linear Discriminative-Iterative: Saragih & Goecke, 2006

Haar-like Feature Based Iterative-Discriminative Method (HFBID): Saragih & Goecke, 2007

System Overview

1

2

Experiments(1) Generalisation, (2) Robustness to

initialisation errorPerson-dependent models (PDFER): individual

modelsPerson-independent models (PIFER): general

modelsNot for POIC as has previously been shown to

not generalise well across different peopleCohn-Kanade database:

Subset of 30 subjects (15f / 15m)Total of 3424 images:

992 images for Neutral, 448 images for Anger, 296 images for Disgust, 346 images for Fear, 532 images for Joy, 423 images for Sorrow and 387 images for Surprise.

InitialisationTraditionally, beside generalisation, one of

the most challenging problems for AAMs has been robustness to initialisation error

Common face detectors, e.g. Viola-Jones, often give you an error (translation) of up to 30 pixels

We simulate this by deliberately misaligning the initial AAM: ±5, ±10, ±20, ±25 (PIFER) / ±30 pixels (PDFER)

Multi-class SVM using a linear kernel for PDFER and a Radial Basis Function kernel for PIFER

Classify expressions as Neutral or one of the ‘Big 6’ (7-class problem

Facial Expression RecognitionIn this study, we were interested in

recognising the ‘Big 6’ + Neutral expressionsSince the scope of most of the vision based

expression recognition systems is based on changes in appearance, we grouped AUs together on a ‘regional basis’

In that way, we did not have to recognise individual AUs but analysed movement patterns in various facial regions, which made the FER process more robust

FER (2)

FER Results - Video

Ground truth

Results – Person-dependent Models

Sta

bl

e“U

nst

ab

le”

Results – Person-independent Models

Sta

bl

e“U

nst

ab

le”

ConclusionsInvestigate the utility of different AAM fitting

algorithms in the context of real-time FERIterative-Discriminative (ID) approach

adopted in IEBM and HFBID boosts the fitting performance significantly and thus leads to improved FER results

More robust to initialisation error than other methods

IEBM and HFBID generalise wellRapid fitting (real-time capable) ~ as fast as

POICFuture work:

Pose-invariant FER