Confidence-Based Fusion of Multiple Feature Cues for Facial Expression Recognition Spiros Ioannou 1 , Manolis Wallace 2 , Kostas Karpouzis 1 , Amaryllis Raouzaiou 1 and Stefanos Kollias 1 1 National Technical ni!ersity of Athens, ", Iroon #olytechniou Str$, 1%& '( )o*raphou, Athens, +reece 2 ni!ersity o f Indianapolis, Athens ampus, ", Ipitou Str$, 1(% %& Synta*ma, Athens, +reece Abstract— Since facial expressions are a key modality in human communication, the automated analysis of facial images for the estimation of the displayed expression is essential in the design of intuitie and accessi!le human computer interaction syste ms" #n mos t exist ing rule-!a sed expre ssion recogniti on approaches, analysis is semi-automatic or re$uires high $uality id eo" #n this paper %e pro pose a featur e extra ction system %hich com!ines analysis from multiple channels !ased on their co nf idence, to re sult in !e tte r facial feat ure !o undary det ect ion " &he fac ial featur es are the n use d for exp res sio n estimation" &he proposed approach has !een implemented as an extension to an existing expression analysis system in the frame%ork of the #S& ERM#S pro'ect" Index Terms — Facial feature extraction, confidence, multiple cue fusion, human computer interaction I$ I NTR-.TI-NIn re cent years the re has /e en a *r o0in* int er est in impro!in* all aspects of the interaction /et0een humans and computer s, pro!i din* a realization of the term affec ti!e computin*31%4$ 5umans interact 0i th ea ch ot her in a multimodal manner to con!ey *eneral messa*es6 emphasis on certain parts of a messa*e is *i!en !ia speech and display of emotions /y !isual, !ocal, and other physiolo*ical means, e!en instincti!ely 7e$*$ s0eatin*8 3194$ Inter pers ona l co mmuni ca ti on is fo r the most part compl ete d !ia the fac e$ .e spi te common /el ief , socia l psychol o*y research has sho0n that con!ersations are usual ly dominated /y faci al e:pre ssion s, and not spo; en 0or ds, indica tin* the spea; er<s predi spos ition to0ards the lis tener$ Mehra/ian indica ted that the lin* uis tic part of a mess a*e, that is the actual 0ordi n*, contr i/ute s only for se!en percent to the effect of the messa*e as a 0hole6 the paralin*uistic part, that is ho0 the specific passa*e is !ocalized, contri/utes for thirty ei*ht percent, 0hile facial e:pression of the spea;er contri/utes for fifty fi!e percent to the effect of the spo; en messa*e 324$ This implies that the fa ci al e:pr essions form the ma =or modali ty in human communica tion, and need to /e conside red /y 5I >MMI systems$ In most real?life applications nearly all !ideo media ha!e reduced !ertical and horizontal color resolutions6 moreo!er, the face occupies only a small percenta*e of the 0hole frame and illumination is far from perfect$ When dealin* 0ith such input 0e ha!e to acce pt that co lor @ual it y and !i de o resolution 0ill /e !ery poor$ While it is feasi/le to detect the face and all facial features, it is !ery difficult to find the e:act /oundary of each one 7eye, eye/ro 0, mouth8 in order to estimate its def ormat ion from the neutral ?e:pre ssion frame$ Moreo!er it is !ery difficult to fit a precise model to each feat ure or to empl oy trac;in* since hi*h?orde r fre@ uency information is mi ss in* in such situations$ A 0a y to o!ercome this limitation is to com/ine the result of multiple feat ure e:tractor s into a final result /ased on the e!a luatio n of their performance on each frame6 the fusion method is /ased on the o/ser!ation that ha!in* multiple mas;s for eac h fe ature lo0 ers the pro /a/ ili ty tha t all of them are in!alid since each of them produces different error patterns$ II$B#RSSI-NR#RSNTATI-N An automa te d emot ion re co*niti on thr ou*h fa ci al e:pression analysis system, must deal mainly 0ith t0o ma=or research areasC automatic facial feature e:traction and facial e:pression reco*nition$ Thus, it needs to com/ine lo0?le!el ima*e processin* 0ith the results of psycholo*ical studies a/out facial e:pression and emotion perception$ Most of the e:istin* e:pression reco*nition syste ms can /e cla ssi fi ed in t0o ma= or ca te* ori esC the former inc ludes techni@ues 0hich e:amine the face in its entirety 7holistic appr oaches8 and ta;e into ac co unt pr operties such as int ensi ty 3"4or opt ica l fl o0 dist ri/uti ons and the lat ter includes methods 0hich operate locally, either /y analyzin* the motion of local features, or /y separately reco*nizin*, meas ur in*, and com/ inin* the !a ri ous facial element properties 7analytic approaches8$ A *ood o!er!ie 0 of the current state of the art is presented in 3D431(4$In this 0or; 0e estima te facial e:press ion throu*h the estimation of the M#+ EA#s$ EA#s are measured throu*h detection of mo!ement and deformation of local intransient facial features such as mouth, eyes and eye/ro0s in sin*le frame s$ Eeature defo rmati ons are esti mated /y comp arin* their states to some frame, in 0hich the person<s e:pression is ;no0n to /e neutral$ Althou *h EA#s 314pro!ide all the necessary elements for M#+?D compati/le animation, 0e cannot use them directly for the analysis of e:pressions from !id eo sc ene s, due to the a/s ence of a cle ar @ua ntit ati !e def ini tion fra me0 or;$ In order to mea sur e EA#s in real ima *e se@ uen ces , 0e ha!e to define a map pin* /et 0e en them and the mo!ement of specif ic E.# featur e points 7E#s8, 0hich correspond to salient points on the human face$
6
Embed
Confidence-Based Fusion of Multiple Feature Cues for Facial Expression Recognition
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
8/12/2019 Confidence-Based Fusion of Multiple Feature Cues for Facial Expression Recognition
Ei*ure D$ Einal mas; for the eyesEi*ure %$ All detected feature points from the final mas;s
I$ B#RSSI-N A NAGSIS
The feature mas;s are used to e:tract the Eeature #oints7E#s8 considered in the definition of the EA#s, used in this0or;$ ach E# inherits the confidence le!el of the final mas;from 0hich it deri!es6 for e:ample, the four E#s 7top,
/ottom, left and ri*ht8 of the left eye share the sameconfidence as the left eye final mas;$ ontinuin*, EA#s can
/e estimated !ia the comparison of the E#s of the e:aminedframe to the E#s of a frame that is ;no0n to /e neutral, i$e$ a
frame 0hich is accepted /y default as one displayin* nofacial deformations$ Eor e:ample, EA# J&
F
(squeeze_l_eyebro! is estimated asC
J& D$% J$11 D$% J$11
n n F F" F" F" F" = − − −
0here n
i F" , i F" are the locations of feature point i on the
neutral and the o/ser!ed face, respecti!ely, and
i # F" F" − is the measured distance /et0een feature
points i and # $
Ei*ure 9$ M#+?D Eeature #oints 7E#s8
-/!iously, the uncertainty in the detection of the feature points propa*ates in the estimation of the !alue of the EA#as 0ell$ Thus, the confidence in the !alue of the EA#, in thea/o!e e:ample, is estimated as
J& D$% J$11min7 , 8c c c F F" F" =-n the other hand, some EA#s may /e estimated in different
0ays$ Eor e:ample, EA# J1 F is estimated asC
1
J1 J$1 J$J J$1 J$J
n n F F" F" F" F" = − − −or as
2
J1 J$1 "$1 J$1 "$1
n n F F" F" F" F" = − − −As ar*ued a/o!e, considerin* /oth sources of informationfor the estimation of the !alue of the EA# alle!iates some of
the initial uncertainty in the output$ Thus, for cases in 0hicht0o distinct definitions e:ist for a EA#, the final !alue andconfidence for the EA# are as follo0sC
1 2
2
i ii
F F F
+=
The amount of uncertainty contained in each one of the
distinct initial EA# calculations can /e estimated /y1 11 c
i i $ F = −for the first EA# and similarly for the other$ The uncertainty
present after com/inin* the t0o can /e *i!en /y some t ?norm operation on the t0oC
1 27 , 8i i i $ t $ $ =The Ga*er t ?norm 0ith parameter %& *i!es reasona/le
results for this operationC
( )( )1 21 min 1, 71 8 71 8
i i i $ $ $ = − − + −
The o!erall confidence !alue for the final estimation of theEA# is then ac@uired as
1c
i i F $ = −While e!aluatin* the e:pression profiles, EA#s 0ith
*reater uncertainty must influence less the profile e!aluationoutcome, thus each EA# must include a confidence !alue$This confidence !alue is computed from the correspondin*E#s 0hich participate in the estimation of each EA#$
Einally, EA# measurements are transformed to antecedent
!alues # x for the fuzzy rules usin* the fuzzy num/ers
defined for each EA#, and confidence de*reesc
# x are
inherited from the EA#Cc c
# i x F =
0here i F is the EA# /ased on 0hich antecedent # x is
defined$ More information a/out the used e:pression profilescan /e found in 3J43'4$
$ B#RIMNTA R STS
Eacial feature e:traction can /e seen as a su/cate*ory of
ima*e se*mentation, i$e$ ima*e se*mentation into facial
features$ )han* 32(4 re!ie0ed a num/er of simple
discrepancy measures of 0hich, if 0e consider ima*e
se*mentation as a pi:el classification process, only one is
applica/le hereC the num/er of misclassified pi:els on each
facial feature$ While manual feature e:traction do not
necessarily re@uire e:pert annotation, it is clear in especially
in lo0?resolution ima*es manual la/elin* introduces an
error$ It is therefore desira/le to o/tain a num/er of manual
interpretations in order to e!aluate the inter?o/ser!er
!aria/ility$ A 0ay to compensate for the latter is Williams<
Inde: 7WI8 394, 0hich compares the a*reement of an
o/ser!er 0ith the =oint a*reement of other o/ser!ers$ An
e:tended !ersion of WI 0hich deals 0ith multi!ariate data
can /e found in 31"4$ The modified Williams< Inde: di!ides
8/12/2019 Confidence-Based Fusion of Multiple Feature Cues for Facial Expression Recognition
31'4 T$+$ .ietterich, nsem/le methods in machine learnin*, #roceedin*s ofEirst International onference on Multiple lassifier Systems, 2((($
31"4 i;ram halana and Gon*min Kim, A Methodolo*y for !aluation ofoundary .etection Al*orithms on Medical Ima*es, I Transactionson Medical Ima*in*, ol$19, No$% -cto/er 1""&
32(4 G$$)han*, A Sur!ey on !aluation Methods for Ima*e Se*mentation,
#attern Reco*nition, ol 2", No$ ', pp1JJD?1JD9, 1""93214 Gin*?li Tian, Ta;eo Kanade and effrey E$ ohn, Reco*nizin* Action
nits for Eacial :pression Analysis I Transactions -n #atternAnalysis And Machine Intelli*ence, ol$ 2J, No$ 2, Ee/ruary 2((1