Studying the dynamics of emotional expression using synthesized facial muscle movements

Journal of Personality and Social Psychology Copyright 2000 by the American Psychological Association. Inc. 2000, Vol. 78, No. 1, 105-119 0022-3514/00/$5.00 DOI: 10.1037//0022-3514.78.1.105

Studying the Dynamics of Emotional Expression Using Synthesized Facial Muscle Movements

Thomas Wehrle, Susanne Kaiser, Susanne Schmidt, and Klaus R. Scherer University of Geneva

Synthetic images of facial expression were used to assess whether judges can correctly recognize emotions exclusively on the basis of configurations of facial muscle movements. A first study showed that static, synthetic images modeled after a series of photographs that are widely used in facial expression research yielded recognition rates and confusion patterns comparable to posed photos. In a second study, animated synthetic images were used to examine whether schematic facial expressions consisting entirely of theoretically postulated facial muscle configurations can be correctly recognized. Recognition rates for the synthetic expressions were far above chance, and the confusion patterns were comparable to those obtained with posed photos. In addition, the effect of static versus dynamic presentation of the expressions was studied. Dynamic presentation increased overall recognition accuracy and reduced confusions between unrelated emotions.

Many studies have shown that judges are able to recognize facial expressions of specific emotions with a high degree of accuracy (Ekman & Rosenberg, 1997; Russell & Femgmdez-Dols, 1997). However, surprisingly enough, there has been little detailed research into the process of facial emotion recognition beyond the demonstration of recognition accuracy (or its methodological shortcomings; Ekman, 1994b; Izard, 1994; Russell, 1994). In consequence, our knowledge about the nature of the cues used in facial expression recognition is extremely limited. In this article, we will address four major issues with respect to the pertinent elements and the process of facial expression interpretation: (a) the nature of the facial movement cues used by observers to infer an underlying emotion; (b) the evidence for theoretical predictions concerning emotion-specific patterns of facial movements; (c) the importance of dynamic versus static presentations of facial expres- sion patterns; and (d) the investigation of theoretical assumptions about the temporal unfolding of facial expressions of emotion.

The Nature of the Facial Cues Used by Observers to Infer an Underlying Emotion

Most researchers in this area hold that the inference of emotion from facial expression is based on emotion-specific patterns or configurations of facial muscle movements. On the basis of earlier

Thomas Wehrle, Susanne Kaiser, Susanne Schmidt, and Klaus R. Scherer, Department of Psychology, University of Geneva, Geneva, Swit- zerland.

This project was supported by grants from the Swiss National Science Foundation (FNRS 11-39551.93/11-049629.96, for the project "Dynamic Man-Computer Interactions as a Paradigm in Emotion Research"). We thank Denise Mazenauer for her help in carrying out the second judgment study.

Correspondence concerning this article should be addressed to Thomas Wehrle, Department of Psychology, University of Geneva, 40, Bd. du Pont d'Arve, CH-1205 Geneva, Switzerland. Electronic mail may be sent to thomas.wehrle @pse.unige.ch.

105

anatomical descriptions of the facial musculature involved in the expression of emotion (Ducbenne, 1872, 1872/1990; Hjortsj6, 1970), discrete-emotion theorists have developed coding systems that allow objective descriptions of the muscle movements in- volved (Facial Action Coding System [FACS]; Ekman & Friesen, 1978--The Maximally Discriminative Facial Movement Coding System [MAX]; Izard, 1979). These descriptive systems have been used both to code posed facial expressions and to instruct posers as to the appropriate expressions for particular emotions (Ekman & Rosenberg, 1997; Lewis, Stanger, & Sullivan, 1989; Malatesta & Haviland, 1982).

However, the results of past research do not allow an assessment of the extent to which emotion recognition from facial expression is based on muscle movements (and the resulting changes in facial skin) alone. Obviously, photographs of posed facial expressions provide many additional cues such as skin color, pupil size, and physiognomy. Furthermore, the same facial muscle movements may be realized in a different fashion by different posers and may interact with the idiosyncratic facial features of the respective posers. In this research, we propose using computer synthesis of facial muscle movements to examine the extent to which facial muscle movements (presented in schematic form) are sufficient to allow for better-than-chance recognition of specific emotions.

Theoretical Predictions Concerning Emotion-Specific Patterns of Facial Movements

Among emotion theories proposing implicit or explicit predic- tions for emotion-specific facial expression patterns, two positions can be distinguished. The ftrst approach is situated in the tradition of "discrete emotion" theories and is represented by Ekman, Izard, and their respective collaborators (Ekman, 1992; Ekman & Friesen, 1975; Ekman, Levenson, & Friesen, 1983; Izard, 1971, 1991, 1992; Izard & Haynes, 1988). The second approach has been suggested in the context of appraisal theories of emotion (Scherer, 1984, 1992; Frijda & Tcherkassof, 1997; Smith, 1989; Smith & EUsworth, 1985; Smith & Scott, 1997; see also Ortony & Turner, 1990).

106 WEHRLE, KAISER, SCHMIDT, AND SCHERER

Both of these positions claim that specific patterns of facial muscle movements characterize the expressions of emotions such as joy, fear, anger, disgust, and sadness. The purpose of a second study was to generate synthetic images, constructed solely on the basis of theoretical predictions (without encoding by human pos- ers) to examine whether these pattems are recognized as well as human poses.

For the purposes of this article, the predictions made by Scherer in the context of his component process model of emotion (Scherer, 1984) are used. A first version of Scherer's predictions on facial expression was presented in a widely distributed but unpublished paper (Scherer, 1987). Because of theoretical considerations as well as empirical results found in the literature, we have used the most recent version of FACS to extend and refine Scherer's original predictions about which facial actions are likely to occur with the outcome of the postulated appraisal dimensions (called stimulus evaluation checks [SECs]) for specific emotions.

Appendix A presents brief descriptions of the predicted facial changes as well as the respective action unit identification numbers (a description of the action units is included at the bottom of the table). According to the assumed sequentiality of appraisal by component process theory, cumulative changes are thought to occur in the facial expressions, with the effect of each subsequent check being added to the effects of prior checks (see Scherer, 1992, for further details). Scherer has specified prototypical profiles of appraisal results (i.e., specific outcomes of the SECs) for most of the basic emotions postu- lated by discrete emotion theories (see Table 2 in Scherer, 1993), although he considers these as "modal" rather than "basic" emotions (see Scherer, 1984, 1994, for a detailed dis- cussion of this point). The expected changes in facial expres- sion are predicted on the basis of these appraisal patterns (see Scherer, 1987, 1992, for details of the argumentation).

As an example, Appendix B shows the postulated appraisal profile for sadness and the related action units. Column 1 shows the appraisal checks; column 2, the values of the parameters for sadness (from Scherer, 1993); and column 3, the predictions of corresponding changes in terms of action units. As shown in column 2 of the table, the value of the dimension "control" is predicted to be "very low" for sadness. This value is predicted to a result in a facial expression that includes inner brow raise (AU 1), lip comer depress (AU15), lid droop (AU41), and eyes down (AU64). For a particular emotion, these changes in action units are expected to occur in a sequential-cumulative fashion; that is, starting in row 1 with the facial changes due to the novelty check and with the elements shown for the further checks being cumu- latively added. In this research, synthetic images of facial expres- sions were produced according to these predictions, in both a static and a dynamic version.

Dynamic Versus Static Presentations of Facial Expression Patterns

In most studies on the judgment of emotion from facial expres- sion, researchers have used static facial stimuli in the form of still photographs. Apart from their questionable ecological validity (see Wallbott & Scherer, 1986), such stimuli may lack essential cues used for the differentiation of emotions. For example, one of the consistent findings in this area is that fear expressions are often

confused with surprise. One explanation for this might be that the main difference between these two emotions resides in the respec- tive temporal structure, not observable in still photographs, of the innervations in the facial musculature.

In a recent study, Lemay, Kirouac, and Lacouture (1995) did, in fact, report evidence that the dynamics of facial expression allow differentiation between fear and surprise. These authors compared static and dynamic stimuli (using emotion portrayals by actors, see Gosselin, 1989; Gosselin, Kirouac, & Dort, 1995) with regard to dimensional versus categorical judgments of emotional facial ex- pressions. In the dimensional judgment task, participants had to evaluate the similarity of 21 pairs of stimuli. Although the authors found two similar dimensions for the static and the dynamic stimuli, which they labeled pleasantness and control, fear and surprise were more clearly separated in the dynamic condition than in the static condition. In the latter case fear, surprise, sadness, and disgust formed a rather compact cluster. The differences observed between static and dynamic stimuli were even more pronounced in the categorization task (requiting intensity judgments on a 6-point scale). Indeed, fear was confused with surprise in the static con- dition only (fear was rated with a mean intensity of 1.76 as compared with 1.45 for surprise). In the dynamic condition, fear was recognized without ambiguity (the intensity rating for fear was 5.31; no other emotion was rated higher than 0.76).

Another example of the relevance of temporal aspects is the fact that observers are very sensitive to false timing of facial expres- sions (e.g., abrupt endings or beginnings of a smile) when evalu- ating the truthfulness or deceitfulness of an emotional display (Ekman, Friesen, & O'Sullivan, 1988). Although the importance of "correct" timing is widely accepted at a theoretical or phenom- enological level, only a small number of studies have investigated this aspect systematically--mostly for smiles (e.g., Frank, Ekman, & Friesen, 1993; Hess, Kappas, McHugo, Kleck, & Lanzetta, 1989). Generally, our knowledge of the temporal unfolding of facial expressions is quite limited.

Yet the systematic study of the dynamics of facial expression may provide important evidence for some of the central issues in this field of research. In this article, we advocate using systemat- ically manipulated, synthetic facial expressions to study how changes in the patterning of facially expressive features over time affect emotion inferences by judges. We claim that the data pro- duced by such studies may prove valuable by enhancing our knowledge concerning the perception of and the inference from the temporal aspects of facial expressions. In this study we also examine the role of dynamic presentation of the expressions, that is, animated sequences of the expression as compared to stills.

Theoretical Assumptions About the Temporal Unfolding of Facial Expressions of Emotion

Apart from allowing the systematic study of the difference in emotion recognition based on static and dynamic stimuli, the synthesis approach advocated in this article permits a first empir- ical investigation of contradictory theoretical accounts concerning the temporal unfolding of facial expressions of emotion.

Discrete-emotion theories claim that there are only a limited number of fundamental or "basic emotions" and that for each of them there exists a prototypical, universal expression pattern based on innate neuromotor programs (Ekman, 1992; Tomkins, 1984);

SYNTHETIC FACIAL EXPRESSIONS OF EMOTION 107

these theories thus imply that these patterns are produced as "packages. ''1 In this tradition, a process of blending or mixing the basic expression patterns is used to explain the variability of commonly observed complex emotion expressions. Despite the fact that within discrete emotion theories facial behavior is explic- itly considered to be one of the most important components of emotion, the theorists in this tradition have not made many con- crete predictions about the dynamics of facial behavior. However, the central notion of prototypical, innate programs for each of the basic emotions seems to imply that the elements of emotion- specific facial actions appear simultaneously and thus constitute the prototypical pattern. In other words, the concept of a neuro- motor program suggests that once the production of an expression is triggered, the complete prototypical pattern is executed as a whole, a process in which the dynamic unfolding of the expression is of little consequence. 2

In the following, we use the theoretical position represented by Paul Ekman (1992, 1997) as representative of discrete-emotion theories of facial expression: Although Ekman has not yet explic- itly specified in print exactly how he conceives of the microstruc- ture and the temporal unfolding of neuromotor programs that are expected to be responsible for emotion-specific facial expression patterns, he and his collaborators have repeatedly referred in their writing to issues concerning dynamic aspects of facial expression.

For example, in a discussion of the difference between sponta- neous emotional expressions (category 1) and deliberate facial actions (category 2), Ekman (1997, p. 474) lists the following six criteria to distinguish the two: morphology, symmetry, total dura- tion, speed of onset, coordination in apexes, and ballistic trajec- tory. The last two points are pertinent for the issue of describing the dynamics of facial behavior. Concerning the coordination of apexes for individual action units, it is specified that "the apexes of each of the actions involved in the expression will be more likely to overlap in category 1 than in category 2." With respect to ballistic trajectories, it is suggested that "the expression will appear smooth over its course in category 1, rather than jagged or stepped in category 2." These remarks suggest the underlying assumption that, once triggered by a motor command, the individual action units that make up a prototypical expression pattern will be pro- duced in a simultaneous fashion, with comparable ballistic trajec- tories and coordinated apexes. This interpretation is confirmed by Ekman (personal communication, June 1998), who does not ex- clude the possibility that there may be slightly different time courses of individual action units in an expression pattern due to differences in muscle mass to be moved; However, he considers it relatively unlikely that these will lead to visible offsets in the appearance of action unit changes.

In contrast to this molar approach, assuming a complete pattern of expression to be triggered by a neuromotor command, the molecular approach proposed by the appraisal theorists cited above suggests that the elements of an expression pattern, the individual facial muscle movements, might be produced by different deter- minants in an independent, cumulative fashion. These ideas are based largely on the work of the pioneers in this area, particularly Duchenne (1872/1990) and Darwin (1876), who, in addition to discussing emotion-specific expression patterns, also studied indi- vidual muscles and their functions. In this article, we will use Scherer's (1984) component-process model of emotion as a rep- resentative example of this approach.

Scherer has argued that the elicitation and differentiation of emotion can be most economically explained by a process of cognitive appraisal that reflects the significance of an event for an individual's goal and value structure, his or her coping potential, and the socionormative significance of the event (see Scherer, 1984, 1988, for details). Column 1 of Appendix A shows the major appraisal dimensions or SECs that are considered to be sufficient to account for the differentiation of all major emotions. The nature of the emotional reaction or response is expected to be shaped directly by the outcomes of the appraisal process. Thus, rather than assuming that appraisal processes will evoke one of several basic emotions with their specific response patterning, component pro- cess theory suggests that the result of each stimulus evaluation check will have a direct effect on each of the other emotion components, such as autonomic nervous system functioning and motor expression. With regard to the expression .component, Scherer (1984, 1986, 1987, 1992) has formulated concrete predic- tions about the changes in voice production, body movement, and facial expression that are expected to occur as outcomes of the postulated appraisal checks. Scherer (1992) assumes that the facial muscle configuration activated as a combination of effects linked to emotion-antecedent cognitive appraisal and associated action tendencies occurs in a sequential-cumulative manner as the ap- praisal process unfolds.

Our overall aim in this article, in which we report two separate studies, is to advocate the use of static and dynamic synthetic expressions in order to start obtaining empirical evidence for the four issues described above.

General Method: The Facial Action Composing Environment (FACE; Wehrle, 1995)

FACE is a tool for creating three-dimensional animated facial expressions in real time, including head and eye movements. It allows researchers to synthesize facial expressions on the basis of objective descriptors, as provided by Ekman and Friesen's FACS (1978). 3 FACE also permits researchers to systematically vary different aspects of the dynamics of facial expressions and to control or manipulate other perceptual cues such as accompanying head movements, head position, direction of the gaze, or even the physiognomy of a face, all of which might affect emotion infer- ences. The contours of the face are represented with splines, as are the prominent features such as eyebrows and lips. Unlike other face animation systems (see Kalra, Mangili, Magnenat-Thalmann, & Thalmann, 1992; Waters & Terzopoulos, 1991), wrinkles and furrows are also modeled (see Figures 2 to 4).

The face shown in Figures 2 to 4 was designed on the basis of the measures of an adult face, taken from a photograph. The repertoire of facial expressions for the animation was defined

However, FACS, the coding system developed by Ekman and Friesen (1978) is molecular and allows microcoding of single elements (action units). In this sense, FACS is theory-free and can be used for objectively describing facial behavior.

2 This assumption might be at the root of the widespread use of static images in studies designed to prove the universality of these patterns.

3 Musterle developed a tool for creating action units and action unit combinations on the computer screen, but only for static pictures (Musterle Mimikfaces, 1990).

108 WEHRLE, KAISER, SCI-IMIDT, AND SCHERER

using FACS. The FACS coding manual gives a detailed descrip- tion of all the appearance changes occurring with a given action unit. This description lists the parts of the face that have moved and the direction of their movements, the wrinkles that have

appeared or have deepened, and the alterations in the shape of the facial parts.

To use FACE for the animation of facial expressions, the user must indicate the action unit, or the action unit combination, he or she wants to include in the synthetic rendering of a particular facial expression. Action units can be produced asymmetrically and at different levels of intensity (following the five intensity levels of FACS). The animation can be achieved interactively or by defining

a script, comprising a sequence of facial animations that can then be played like a cartoon film. In addition, FACE allows the user to specify and to vary the intensity and time profiles of facial move- ments in terms of attack (onset), sustain (apex), and release (offset) duration.

Method

Stimuli

In the Pictures of Facial Affect set (Ekman, 1976), there are 14 different photographs for happiness, anger, ' surprise, and disgust; 13 for sadness; and 11 for fear, encoded by 5 to 6 male and 6 to 8 female posers respectively. Posers were ta'ained to contract or relax different facial mus- cles associated with various facial expressions. We chose three photo- graphs for each of the emotions--happiness, anger, sadness, fear, surprise, and disgust--that represented a low, a middle, and a high recognition score respectively. We then created corresponding synthetic stimuli by repro- ducing the action unit combinations shown. Some of the photographs are used as training examples in the FACS manual. For these, FACS coding has been provided by Ekman and Friesen (1978). A trained FACS expert (Susanne Kaiser) coded the remaining photographs. The size of the pho- tographs, as projected via a slide projector on a white screen, and the size of the synthetic stimuli that were displayed via a computer projector on the same screen, were matched.

S tudy 1

The objective of this study was to examine the first of the four

issues discussed above- - the question of whether judges will be able to recognize encoded emotions purely on the basis of facial muscle action patterns. As in much of the research in this area, the criterion chosen was above-chance recognition of the target emo- tions. The synthetic stimuli represent appearance changes related to the innervation of action units in the form of simple lines. Obviously, the recognition rate cannot be expected to be as high as for photographs of human faces. Photographs of human faces contain additional cues, such as facial features, skin color, and quality of gaze, which might provide additional emotion cues and thus facilitate the decoding of an emotional expression. At present, these features are not synthesized by FACE. If the lack of these features were to render it impossible to make inferences about the "pure" impact of facial expressions, as defined with FACS, the stimuli created with FACE would be useless for our objective. Yet, if this were the case, this would also mean that it should be equally impossible to make inferences about the expression of an emo- tional state that is based exclusively on the combinations of action units (disregarding other features such as skin color and type of gaze). I f there exists a systematic relation between emotional states and facial expressions, coded in terms of action units only, it should be found in the synthetic stimuli, as long as the effects of the action units are appropriately rendered.

In this study, we compare the recognition scores for a number of photographs from the Pictures of Facial Affect (Ekman, 1976) with synthetic images, created with FACE, which represent the salient facial action units in the respective photographs. If the judges can recognize the intended emotions with better-than-chance accuracy, this is evidence that (a) facial action unit configurations alone carry the emotion-identifying information and (b) that these facial configurations can be represented by synthetic renderings. Should recognition be at chance level, the present design does not permit us to distinguish between the alternatives (a) that the synthetic representation is not appropriate or (b) that in the absence of other facial cues facial action configurations are insufficient to allow emotion differentiation.

Participants

Sixty-three French-speaking undergraduate psychology students (mean age = 23.6 years) participated in the study. They were divided into four groups. Two groups were first shown the photographs and then the syn- thetic stimuli (N = 31, 19 women, 12 men; Group 1: n = 13, Group 2: n = 18). For the other two groups, the sequence was reversed (N = 32, 19 women, 13 men; Group 3: n = 15, Group 4: n = 17). The presentation order of the stimuli was randomized in each group.

Judgment Procedure

The stimuli were presented to the participants in group sessions. Partic- ipants were seated With enough space between them so that they could not see the responses of the others. Each stimulus was presented for 10 s. After each stimulus, participants had to choose one out of six emotion labels considered to be the most appropriate to describe the expression displayed. The choices were anger, fear, sadness, disgust, happiness, and surprise.

Results

First, we tested whether group and/or presentation order had an influence on the mean recognition scores of the 36 stimuli. We found no main effect for group, F(3, 140) = .32, p = .81, and no main effect for order of presentation, F(1, 142) = .04, p = .83. Therefore, we pooled the data from all four groups for the further analyses. For the purposes of this study, it was essential that the judges recognize the emotions in the photographs as well as in the original Ekman study (Ekman, 1976). Therefore we compared the recognition rates for the photos used (reported in Ekman, 1976) with our findings. Although the overall recognition percentage is slightly lower, the difference is not significant: M Ekman = 89.77, SD = 10.35; M photographs = 84.29, SD = 13.51; t(34) --- 1.36, p = .181. Given the cultural differences between judge popula- tions and the time of more than 20 years between the two studies, this stability of the data contributes further evidence to the claim of universality of the facial expression of emotion.

Figure 1 shows the mean recognition scores per emotion (three stimuli each) for the photographs and the synthetic stimuli used in our study (the results obtained earlier by Ekman, 1976, for the respective photographs are shown for comparison). The mean recognition score is 84.29% for the photographs (compared

SYNTHETIC FACIAL EXPRESSIONS OF EMOTION 109

Figure I. Mean recognition scores over the three stimuli per emotion for the Eknian photographs in Ekman's study (Ekman, 1976), our study, and the synthetic images in our study.

be improved by further refinement of the synthesis of the fear- specific action units. To illustrate the differences between photo- graphs and synthetic stimuli, Figure 2 (see next page) shows examples for a neutral expression and for the emotions of happi- ness, sadness, and fear (with the respective recognition scores). As shown in the figure, the synthetic version may not appropriately render the changes around the eyes and the mouth in a fear expression.

In general, however, the current results provide strong encour- agement for using synthetic images in the study of the facial expression of emotion. Given the manifold possibilities of system- aticaUy manipulating static and dynamic configurations of facial action unit patterns with FACE, this methodology is ideally suited to an experimental approach to studying the facial expression of emotion.

with 89.77% in Ekman's earlier study) and 61.72% for the syn- thetic stimuli.

As mentioned above, the synthetic images only reproduce the results of the facial action units, but the photographs show a variety of other cues, such as facial features, skin color, and type of gaze, in addition to the effects of the facial actions. Furthermore, the synthetic images render the effects of the facial actions in a highly stylized and sometimes only approximate fashion. In con- sequence, we find, as expected, that the recognition rate for the synthetic stimuli is significantly lower than that of the photo- graphs: M photographs = 84.29, SD = 13.51; M FACE = 61.72, SD = 25.03; F(1, 17) = 14.04, p < .01 (p = .002).

Of course, the issue is not to demonstrate the equivalence of the synthetic images to photographs but to establish that judges can, based on stylized renderings of facial action unit configurations in isolation, infer the emotions encoded by Ekman's (1976) original posers with better-than-chance accuracy. To determine whether the percentages of correct judgments for each emotion exceed the level one would expect to obtain by chance or under the condition of participants' guessing, we followed Ekman's (1994b) sugges- tions and set the chance level for happiness at 50%, for surprise at 33%, and for the four negative emotions at 25%. As shown in Table 1 (see next page), the emotions encoded in the synthetic images are indeed--with the exception of fear--recognized with better-than-chance accuracy.

The data show that there are important differences between the emotions studied. Whereas synthetic expressions of sadness and happiness are recognized as well or even better than the photo- graphs, synthetic stimuli for disgust and particularly fear are less well recognized. The data in the confusion matrix shows that the three synthetic disgust stimuli ($27, $55, $64) are consistently confused with anger ($27:50.79 anger; $55:61.90 anger; $64: 23.81 anger), which is also true for the photographs. 4 This is not true, however, for the synthetic fear stimuli, where confusions are not limited to one consistent alternative category.

Overall, the data suggest that synthetic facial action unit con- figurations in isolation, independent of other facial cues such as physiognomy or permanent wrinldes, may be sufficient to allow better-than-chance recognition of specific emotions. However, there is a problem with the emotion of fear. Although the design of the present study does not allow us to disentangle what the cause of the low recognition rate is, we suspect that the recognition could

Study 2

This study was intended to address issues two to four, as outlined in the introduction. We chose a set of 10 emotions to study these issues. Most recognition studies use stimulus sets containing expressions of basic emotions (happiness, surprise, anger, sadness, fear, and disgust). Because only one positive emotion is included, there is a danger that the overall recognition accuracy percentage is inflated by the fairly obvious positive- negative distinction. Therefore, we added three additional positive stimuli to happiness (pride, elation, and sensory pleasure). For the negative emotions, we included several instances from different emotion families, forming pairs, namely, cold anger versus hot anger, fear versus anxiety, and sadness versus desperation (see Scherer, 1986). Given the potential modulating role of the intensity of an expression, each of these 10 emotions was synthesized in a low and a high intensity version.

Testing Theoretically Predicted Facial Expression Patterns for Different Emotions

Concrete predictions for the facial action unit patterns expected to characterize these 10 emotions (suggested by Scherer, 1987; as revised by the current authors, and shown in Appendixes A and B) were synthesized with FACE. The major question to be asked was whether the synthetic images generated on the basis of these theoretically derived predictions would produce recognition scores that are comparable to those obtained with the set of Ekman photos.

Studying the Effect of Static Versus Dynamic Presentation

As outlined in the introduction, the question here is to deter- mine, using the same stimulus material, to what extent a dynamic presentation of the synthetic expression (i.e., showing the change from a neutral face to the final pattern) will improve recognition accuracy as compared to static presentation (i.e., showing only the "frozen" final pattern).

4 Numbers of the respective photographs in the Pictures of Facial Affect set.

110 WEHRLE, KAISER, SCI-IMIDT, AND SCHERER

Table 1 Recognition Rates (%)for the Ekman (1976) Photographs and the Synthetic Images

Variable Happiness Surprise Fear Anger Disgust Sadness

Chance 50.00 33.33 25.00 25.00 25.00 25.00 Minimum % for p = .01 64.68 46.80 37.71 37.71 37.71 37.71 Photographs 97.60** 88.17"* 75.86** 85.10"* 90.57** 68.14"* FACE 98.03** 58.02** 28.41 67.87** 43.89** 81.89"*

Note. FACE = Facial Action Composing Environment. **p < .01, one-tailed z test.

Exploring the Role o f Simultaneous Versus Sequential Onset o f Individual Action Units

As shown above, discrete-emotion theory can be expected to postulate the simultaneous onset o f all action units participating in an emotion-specif ic configuration (on the basis o f the assumption o f the operation o f a neuromotor program), whereas component- process theory (Scherer, 1984, 1987, 1992) postulates a sequential onset, synchronized with the unfolding o f the appraisal process. In this study, we varied the simultaneity or sequentiality (based on Scherer ' s predict ions on cumulative effect) o f the onset o f the synthetic action units within the dynamic presentat ion condition. The question is whether either o f these presentat ion methods produces superior results with respect to recognit ion accuracy. If so, one might suppose that the respect ive form of onset or temporal unfolding is closer to what occurs in real expressions, thus pro-

Neutral (092)

Neutral

Happy (085) 9~.s% (ioo%)

Happy (085) !77.8%

Sad(OS7) 74.6% (100%)

Sad (087) 90.5%

rear (o88) 82.5% (lOO%)

Fear (O88) 22.3%

Figure 2. The upper part shows example photographs from the Pictures of Facial Affect (Ekman, 1976) used in this study (with the exception of neutral). Numbers in parentheses are the respective photographs in the Pictures of Facial Affect. Percentages refer to the recognition scores in our study (Ekman's percentages are in parentheses). The lower part shows examples of synthetic images corresponding to the photographs in the upper part. Percentages refer to the recognition scores in our study. From Pictures of Facial Affect, by P. Ekman, 1976, Palo Alto, CA: Consulting Psychologists Press. Copyright 1976 by P. Ekman. Reprinted with permis- sion.

viding indirect evidence for the relative appropriateness of discrete-emotion versus component-process models.

Method

Stimuli

We synthesized animated expressions of 10 emotions according to the theoretical predictions presented in Appendix A. These predictions often allow for several different alternatives. For the purpose of this article, however, we had to decide on a specific pattern for each emotion to create concrete stimuli. Appendix B provides an example of the parameters predicted for sadness.

Additionally, each of these 10 patterns was produced with low and nigh intensity levels. Hence, the total stimulus set consisted of 20 stimuli (see Figure 3), which were prepared by defining script files of the sequences of postulated facial expressions for each emotion. The script files defined the order in which the action units were added, including the specification of durations for onset, apex, and offset. Three versions of the script files for three different conditions were created (see below).

Conditions o f Stimulus Presentation

In Conditions 1 and 2, the facial expressions were presented in a dynamic cartoon style. The conditions differed in the manner in which the respective action units were combined, as shown in Figure 4. The dotted lines in Figure 4 illustrate the sequential cumulation of appraisal-specific action unit combinations resulting in a final pattern, as postulated by Scherer (1987). The solid line illustrates the simultaneous appearance of the final pattern, representing Ekman's presumed position (for more de- tails, see below). Condition 3 used a static, still-image presentation of the complete expression. Because reliable quantitative data about the exact timing of facial expressions are lacking at present, timing of the dynamic stimuli (in terms of onset, apex, and offset duration) was determined on the basis of approximation and intuition.

Condition 1. The stimuli for the sequence condition show the theoret- ically predicted cumulative changes in the facial expressions (see Figure 4). Either the action units predicted for a subsequent check are added to the action units of prior checks or the intensity of already existing action units is modified.

Condition 2. In the movement condition, all action units producing the final pattern of the predicted sequence are animated simultaneously (see Figure 4). These stimuli illustrate the postulated synchronization of all action units seen as constituting a prototypical, innate, and universal expression pattern.

Condition 3. The stimuli for the static condition are snapshots of the final, complete pattern.

To produce the stimulus sets, the stimuli of the three conditions were animated on-line on the computer screen, and this animation was video- taped. The stimuli for the three conditions were presented in a fixed, randomized order.

SYNTHETIC FACIAL EXPRESSIONS OF EMOTION I 11

cold anger (~nerv~)

[]

hot anger (facht)

happiness(content)

sadness (malheureux) !

[]

pleasure hap.(heureux)

m []

despair(d~sesp~rQ Iii

f i l l

elation fjoyeux) n~

[ ]

anxiety (inquiet)

pride ~fieQ

fear (effrayQ nl

I'm

Figure 3. The 20 stimuli used in the second study. Numbers refer to presentation order.

Participants

Judges in this study were 114 French-speaking undergraduate psychol- ogy students (mean age = 25.8 years) divided into three groups among the three experimental conditions, In the first group, sequence condition, there were 27 female and 9 male judges. In the second group, movement condition, there were 31 female and 9 male judges. In the third group, static condition, there were 30 female and 8 male judges.

Judgment Procedure

The videotaped stimulus sets were presented to the participants in group sessions, using a between-participants design. Participants were seated with enough space between them so that they could not see the responses of the others. The experimenter stopped the videotape after each presen- tation of a stimulus to allow all participants to fill out a questionnaire that asked them to rate to what extent each stimulus expressed one of the 10 target emotions by indicating the intensity of the perceived emotion on a 5-point scale) Participants were requested to choose only one of the emotion labels presented in the questionnaire. To reduce the potential ambiguity of the labels, the meaning of each emotion adjective was explained in a short phrase on a separate sheet.

Results

The detailed results (in percentages) are shown in Table 2. The column "correct" reports the percentage of judgments that matched

the theoretical predictions. The column "similar" shows the per- centages of judgments in which an emotion qualitatively similar to the target emotion was selected. For the negative emotional ex-

pressions, "similar" emotions are emotions that form the other element of the emotion pairs cold anger-hot anger, fear-anxiety,

and sadness-despair. For the positive emotional expressions, the percentage for similar emotions is the total percentage of judg- ments in which any other positive emotion was selected. The column "major confusion" indicates when another emotion is reported more frequently than the hypothesized emotional state. For example, this was the case for the stimulus "hot anger-high intensity," which was interpreted by 50% of the subjects as being an expression of fear.

To determine whether the percentages of correct judgments of the synthetic facial expressions reach statistical significance, we applied a strategy similar to that chosen by Ekman (1994b). Participants in our study had to choose among 10 answers, and therefore one could estimate random probability at l/lo (10%). Like Ekman, we set chance estimates at a higher level, adopting a more conservative strategy. For each of the negative emotional stimuli,

5 The judges were presented with the French adjectives of the emotion labels as shown in Figure 3.

112 WEHRLE, KAISER, SCHMIDT, AND SCHERER

Hot Anger

Neutral Novelty high Goal obstructive Control high Final expression

Power high

1 + 2 + 5 4 + 7 4 + 7 + 1 0 + 1 7 + 2 4

Neutral Final expression

Figure 4. The two ways of stimulus presentation for Condition 1 (sequence) and Condition 2 (movement) for hot anger. The dotted lines illustrate the sequential cumulation of appraisal-specific action unit combinations resulting in a final pattern, as postulated by Scherer (1987). The solid line illustrates the simultaneous appearance of the final pattern, representing Ekman's (1997; personal communication, June 1998) presumed position. Action unit numbers and names: 1 (inner brow raiser), 2 (outer brow raiser), 4 (brow lowerer), 5 (upper lid raiser), 7 (lid tightener), 10 (upper lip raiser), 17 (chin raiser), 24 (lip press).

the chance level was set at 17% or t/6, reflecting the probability of choosing any 1 of the 6 negative emotion terms, excluding the positive ones. Using the same procedure, the chance level was set at 25% or 1/~ for the positive expressions. The data presented in the "correct" columns of Table 2 indicate that the recognition scores in the three conditions sequence, movement, and static reach statis-

tical significance for 13, 1'5, and 9 stimuli, respectively (*p < .05; **p < .01, one-tailed z test).

We first tested the influence of intensity on the recognition scores over all conditions by calculating an analysis of variance (ANOVA) for Condition × Intensity. There was no main effect for condition and no interaction, but there was a tendency for intensity, F(1, 59) = 3.49, p = .067: High-intensity stimuli were slightly better recognized than low-intensity stimuli. 6 In general, as shown in Table 2, there were fewer confusions for the stimuli at the high-intensity level than at the low-intensity

level. We then tested whether the 10 emotions were recognized with

different degrees of accuracy in the different conditions by running an ANOVA for Condition × Emotion. There was a tendency for condition, F(2, 59) = 3.19, p = .055, and a significant main effect for emotion, F(9, 59) = 7.06, p < .001, but no interaction effect. Figure 5 shows the recognition scores for the 10 emotions (mean of the two intensity levels) in the three presentation conditions. We will discuss these results in further detail below.

Because, generally, there were no interaction effects with con- dition, we did not systematically include emotion and intensity as independent variables in the further analyses of our hypotheses.

Testing Theoretically Predicted Facial Expression Patterns for Different Emotions

The results in Table 2 show that, in our applying a strict criterion of correct identification of each of the 10 emotions at the appro- priate intensity levels, the predictions fall somewhat short of optimal performance. In many cases, especially for stimuli of low intensity, the recognition score does not exceed chance expecta- tion. Clearly, in terms of such a demanding criterion, both the theory and the synthesis of facial movement need to be improved.

However, such a strict criterion has never been applied in this area of research. To compare the success of the synthetic images based on theoretical predictions from component-process theory, we need to modify our standard of comparison somewhat. In particular, in most of the research to date, judges had to identify one of four to six answer aiternatives--that is, highly different, discrete emotions. Apart from the fact that in the case of a limited number of response alternatives, one cannot rule out the possibility that the high recognition accuracy found is based more on a process of correct discrimination than it is on real recognition (see Banse & Scherer, 1996), the relative similarity of the stimuli to be distinguished needs to be taken into account. In consequence, in order to compare the results of the synthetic stimuli based on theoretical predictions with those recreating the posed expressions

o One reason for this result might be that "low-intensity emotion cate- gories," like sadness, are less well recognized when they are presented with a high-intensity facial expression. The high-intensity sadness stimuli are more often rated as desperation (see Table 2).

SYNTHETIC FACIAL EXPRESSIONS OF EMOTION

Table 2 Accuracy of Judges' Recognition (in %)for the I0 Target Emotions on Two Intensity Levels

113

Sequence Movement Static Major Major Major

Emotion Correct Similar Total confusion Correct Similar Total confusion Correct Similar Total confusion

C A - 31.4"* 11.4 42.8 Anx 45.7 32.4** 16.2 48.6 Anx 32.4 11.1 11.1 22.2 Anx 44.4 CA+ 44.4** 27.8 72.2 32,5** 32.5 65.0 27.8* 27.8 55.6 H A - 33.3** 27.8 61.1 28.2* 30.8 59.0 15.8 15.8 31.6 Anx 36.8 HA+ 25.0 2,8 27.8 Fear 50.0 52.5** 25.0 77.2 44.7** 18.4 63.1 S a d - 52.8** 8,3 61.1 70.0** 7.5 77,5 57.9** 5.3 63.2 Sad+ 36.1"* 63.9 100.0 41.0"* 59.0 100.0 42.1"* 57.9 100.0 D e s - 38.9** 30.6 69.5 28.9* 42.1 71.0 26.3 21.1 47.4 HA 28.9 Des+ 52.8** 16.7 69.5 36.1"* 41.7 77.8 38.9** 41.7 80.6 A n x - 20.0 5.7 25.7 CA 31.4 28.9* 15.8 44.7 CA 26.3 23.7 15.8 39.5 CA 26.3 Anx+ 11.1 5.6 16.7 HA 44.4 12.8 5.1 17.9 CA 33.3 13.2 2.6 15.8 CA 42.1 F e a - 13.9 27.8 41.7 HA 30.6 30.0* 10.0 40.0 18.4 10.5 28.9 HA 28.9 Fea+ 55,6** 0.0 55.6 40.0** 0.0 40.0 28.9* 0.0 28.9 HA 50.0 H a p - 37.1" 45.8 82.9 28.2 46.2 74.4 30.6 25.0 55.6 Hap+ 58.3** 41.7 100,0 46.2** 41.0 87.2 42.1"* 55.2 97.3 E l a - 66.7** 27.8 94.5 40.0* 52.5 92.5 23.7 68.5 92.2 Hap 39.5 Ela+ 88.9** 11.2 100.0 65.0** 35.0 100.0 47.4** 52.6 100.0 P lea - 28.6 62.9 91.5 Pride 40.0 30.8 46.1 76.9 Pride 33.3 14.3 71.4 85.7 Pride 51.4 Plea+ 50.0** 50.0 100.0 37.5* 60.0 97.5 42.1"* 53.1 95.2 Pr id - 5.6 77.7 83.3 Hap 50.0 12.8 56.4 69.2 Hap 41.0 0.0 70.2 70.2 Hap 48.6 Prid+ 8.3 86.1 94.4 Hap 52.8 5.0 85.0 90.0 Hap 35.0 7.9 84.2 92.1 Hap 39.5

Note. CA = cold anger; HA = hot anger; Sad = sadness; Des = desperation; Anx = anxiety; Fea = fear; Hap = happiness; Ela = elation; Plea = pleasure happiness; Prid = pride; - = low intensity; + = high intensity. With respect to the negative emotions, "similar" refers to the respective other part of the emotion pairs HA--CA, sadness-desperation, and anxiety-fear. With respect to the positive emotions, "similar" refers to all the other positive emotions. * p < . 0 5 . **p < .01 .

o f human actors, we needed to pool the data in such a way as to obtain comparable classes of emot ions (i.e., average the recogni- tion scores over members of the same emot ion family and different degrees of intensity). The results o f this operation are shown in Table 3. These data (comparing rows 1 and 2) suggest that al-

though the recognit ion rates for theoretically based static stimuli

are generally somewhat lower than those for photograph-der ived static stimuli, they are generally in the same ballpark.

This is quite remarkable because the photographs of human

expression portrayals are generally selected with respect to their prototypicality and, in the present study, we used photographs that had a fairly strong record in terms of recognizabil i ty in earlier

Figure 5. Recognition scores for the 10 emotions (mean of the two intensity levels) in the three presentation conditions. CA = cold anger; HA = hot anger; Sad = sadness; Des = desperation; Anx = anxiety; Hap = happiness; Ela = elation; Plea = pleasure happiness.

114 WEHRLE, KAISER, SCHMIDT, AND SCHERER

Table 3 Comparison of Recognition Rates (in %)for Poser-Based (Study 1) and Theory-Based (Study 2) Synthetic Images of Facial Expression

t(57) = .54, p = .59, two-tailed. In consequence, the present results do not privilege one theoretical account of the temporal unfolding of emotional expression over another.

Image Happiness Fear Anger Sadness

Study 1

Static 98.03 28.41 67.87 81.89

Study 2

Static 86.04 28.28 43.13 72.80 Movement 85.96 35.65 62.45 81.57 Sequence 93.33 34.93 50.95 75.03

Note. Only the emotions used in both studies are listed; for Study 2, data for the same emotion family and two degrees of intensity were averaged.

research. In contrast, the theory-based stimuli were constructed exclusively on the basis of abstract predictions, using cognitive appraisals rather than prototypical expressions as a basis. As Table 3 shows, the recognition rates for dynamic theory-based synthetic stimuli are still higher than static stimuli (given the lack of appropriate data, they cannot be compared with video-recorded expressions). In consequence, the current data seem to encourage a theoretical approach that tries to predict emotion-specific facial expressions on the basis of the adaptive consequences of particular appraisal results.

Studying the Effect of Static Versus Dynamic Presentation

The results in Table 3 suggest that there are indeed significant differences in the recognition scores between static and dynamic conditions. Because we have hypothesized that the facial stimuli would be better recognized in the dynamic conditions (sequence and movement) than in the static condition (see also Lemay et al., 1995), we tested this hypothesis by using planned orthogonal comparisons. Specifically, we contrasted the means of correct judgments in the static condition with the movement and sequence conditions (contrast coefficients are -2 , 1, and 1, respectively). The mean recognition scores in the dynamic conditions are signif- icantly higher than in the static condition: M sequence = 37.94, SD = 21.33; M movement = 34.92, SD = 15.80; M static = 27.84, SD = 15.27; t(57) = 1.77, p < .05, one-tailed. Furthermore, a detailed examination of Table 2 reveals that most confusions were found in the static condition. These results sup- port the hypothesis that the dynamic presentation of emotional expressions adds important cues that tend to improve recognition. This result is in line with those found by Lemay et al. (1995). As predicted, fear was better recognized in the dynamic conditions than in the static condition, where there was a marked confusion with hot anger.

Exploring the Role of Simultaneous Versus Sequential Onset of Individual Action Units

To test the difference between the two dynamic conditions, we computed a second planned orthogonal comparison between the two dynamic conditions (the respective contrast coefficients are 0, -1 , 1). The results show this difference to be nonsignificant,

Discussion

We will start with a methodological comment concerning the validity of the synthetic stimuli of facial expression of emotion. Results from both studies suggest that the synthetic images and animations generated by FACE are processed in a similar fashion as photographs of facial expressions and yield rather comparable results with respect to recognition rates and confusion patterns. Although FACE displays only lines and shapes with no texture mapping, the synthetic animation seems to include relevant fea- tures of facial expressions. This is shown by the fact that the judges were able to identify different expression patterns created with FACE.

One explanation for this encouraging result might be that the facial repertoire was created on the basis of detailed descriptions of the appearance changes produced by each action unit in the FACS manual. As mentioned above, we designed the appearance changes not only for features like eyebrows but also to represent changes in the shape of the facial regions involved and the resulting wrinkles. As a consequence of this strategy, combinations of action units showed the same appearance changes in the synthetic face as described in the FACS manual. These changes were not specified but emerged from adding the vector information of the respective single action units.

This phenomenon is illustrated by the effect of a combination of AU1 (inner brow raiser) and AU4 (brow lowerer). The FACS manual describes the appearance changes for this combination: "The combination of these two A U s . . . pulls medially and up on the mid to inner portions of the upper eyelid and the eye cover fold, pulls the lateral portion of the brow down, producing an oblique triangular shape to the upper eye lid and the eye cover fold." (Ekman & Friesen, 1978, p. 37). Stimulus 9 in Figure 3 shows this type of combination of AU1 and AU4. As can be seen, such an oblique shape also occurs in the animation, emerging from the separate definitions of the respective action units.

Both apparent validity emanating from inspecting the emergent synthetic images and the pattern of empirical results obtained in the two studies reported here strongly encourage further use of the FACE system in the experimental study of emotion inferences from facial features. The possibility of manipulating each action unit separately to obtain realistic patterns of action unit combina- tions allows us, for the first time, to study the theoretical predic- tions of emotion theorists regarding emotion-specific expression patterns in a controlled, experimental fashion.

This strategy seems all the more promising because the results of Study 1 suggest that the informational cues allowing judges to differentiate emotions from facial expressions consist largely of facial muscle movements rather than ancillary skin features. In consequence, it should be possible, by means of systematic ma- nipulation of facial configurations with the help of FACE, to determine the relative contribution to the variance in judgements of different action units and action unit combinations. Similar experiments, using systematic manipulation of synthesized acous- tic parameters, have produced important insights into the nature of

SYNTIqETIC FACIAL EXPRESSIONS OF EMOTION 115

the vocal communication of emotion (Ladd, Silverman, Tolkmitt, Bergmann, & Scherer, 1985; Soberer & Oshinsky, 1977).

In Study 2, a first attempt was made to obtain empirical evi- dence, at least with respect to recognition, for the predictions made by Scherer's component-process model (which derives the hypoth- eses for facial patterning from presumed adaptive consequences of appraisal results). The results showed that synthetic images pro- duced on the basis of these predictions produced recognition rates that were slightly lower than, but generally comparable to, those obtained with synthetic images of highly selected, prototypical poses by human actors. There is reason to expect that prediction success can be improved further. Careful inspection of the data from Study 2 has yielded a number of hints as to where fine-tuning of the predictions is needed.

One particularly salient issue is that the current prediction chart (Appendix A) predicts too many action units for specific emotions. In consequence, judges may be confused by a multitude of cues that are unlikely to be seen in naturalistic expressions. It seems that in normal expressions only subsets of action unit configurations are produced (see also Galati, Soberer, & Ricci-Bitti, 1997; Gos- selin, Kirouac, & Dorr, 1995). Thus, it will be necessary to establish co-occurrence and exclusion rules to specify the partial configurations likely to occur. Clearly, more empirical research using observation and micromeasurement of dynamic facial ex- pressions of emotion in naturalistic contexts will be needed to allow this degree of refinement of the theory. In particular, quan- titative and temporal aspects require further investigation, espe- cially with regard to the respective length, on the average, of onset, apex, and offset duration of spontaneous emotional facial expres- sions. The same is true for more successful prediction of low- intensity forms of expression and of subtle differences between members of a particular emotion family.

Yet the component-process approach, based on appraisal, that is advocated here seems promising with respect to developing more differentiated models of facial expression, allowing researchers to distinguish more than a very limited number of "basic" emotions. This is particularly true with respect to the positive emotions, which have been rather neglected by most emotion theorists, both of the discrete and the valence traditions alike.

In discussing the results with respect to the theoretical predic- tions of Ekman (1992) and Scherer (1992), we will focus on the results concerning the differentiation of positive emotions. Few judgment studies involve more than one positive emotion and therefore our knowledge about this aspect is rather vague. Ekman (1992) and Scherer (1992) make different predictions with respect to the differentiation of positive emotions. The differences concern not only timing (sequential vs. simultaneous action unit onset) but also the question of whether different types of positive emotions, including specific configurations of facial patterns (different action unit combinations), can be identified.

Ekman (1994a) states,

Most interesting is the likelihood that positive emotions such as amusement, contentment, excitement, pride in achievement, satisfac- tion, sensory pleasure, and relief, all share a single signal---a partic- ular type of smile [Ducherme smile] (Ekman, Davidson, & Friesen, 1990). I am convinced that no further facial signal will be found that differentiates among these positive emotions. (p. 18)

In the context of the component-process approach, we predict that these different positive emotions do produce different facial patterns because the respective underlying appraisal profiles are not identical. This hypothesis is supported by a study by EUsworth and Smith (1988) designed to measure the degree of differentiation in the appraisal patterns associated with different pleasant emo- tions. They found that positive emotions, and their associated appraisals, are somewhat less differentiated than negative emo- tions, although there was evidence for a considerable differentia- tion among six pleasantly toned emotions (interest, hope/confi- dence, challenge, tranquillity, playfulness, and love). Our results show that judges differentiated well between high-intensity levels of pleasure, happiness, and elation. The discrimination between these different types of smiles was better for the animated stimuli than for the static stimuli, with the best discrimination scores in the sequence condition. In this condition, even the low-intensity stim- uli for happiness and elation were well recognized.

These results are consistent with those found in the small number of studies on the differentiation of smiles (e.g., B~inninger- Huber & Ranber-Kaiser, 1989; Ricci-Bitti, Caterina, & Garotti, 1996). B~inninger-Huber and Rauber-Kaiser studied the extent to which naive observers are able to differentiate among selected types of smiles, as measured by FACS. The smile types "phoney smile," "sadistic smile," and "Chaplin smile," described by Ekman (1985), and "coy smile," described by Stettner, Ivery, and Haynes (1986), were portrayed by two FACS-trained encoders. Observers using a list of 17 bipolar adjective scales rated the stimuli (slides). The results show that the individual types of smiles were clearly differentiated by the observers but that, at the same time, judg- ments depended on the person encoding the facial expressions. Moreover, these authors found that even very subtle differences in facial cues, head position, and gaze direction can be responsible for judgment differences between facial expressions.

Ricci-Bitti et al. (1996) studied four different types of smiles that they called "sensory pleasure smile," "formal-unfelt smile," "joy smile," and "elation smile." In a first series of studies, actors encoded these different types of smiles. The authors used a role- playing technique and asked the actors to imagine a specific episode. To avoid the variance due to the personal expressive style adopted by different actors, the authors conducted a complemen- tary study using a set of synthetic faces produced by the Mustede Mimikfaces program (1990; see Footnote 3). The actors' portray- als as well as the synthetic stimuli were presented as static pictures. Ricci-Bitti and colleagues found different types of smiles to be differentiated by clear behavioral descriptors. A sensory pleasure smile, for example, was clearly distinguished from other types of smiles by the presence of closed eyes in combination with the Duchenne smile mentioned above.

These results suggest that more systematic study of a possible differentiation of smile types is rather promising. Using tools such as FACE to study these issues empirically not only eliminates individual encoder differences but also helps to explore the tem- poral characteristics that might differentiate between smile types, and more generally, between different members of an emotion family. The results in the second study, which uses synthetic face animation to produce examples of each theoretical model, show that there are differences in judges' assessments.

The results concerning the importance of dynamic information show that presenting dynamic stimuli compared with static stimuli

116 WEHRLE, KAISER, SCHMIDT, AND SCHERER

actually improves the recognizability and differentiation of the facial patterns, This result is in line with observations reported by Lemay et al. (1995), who also found better recognition scores for the dynamic stimuli in their categorical task. However, they did not find fundamental differences in the dimensional task; that is, the geometrical configuration was similar for the static and the dynamic condition. Lemay et al. conclude from their results that "judging the similarity of facial expressions" versus "categorizing facial expressions" may be a different task in terms of the under- lying recognition processes. There is a strong need for systematic research on the respective functions of movement-independent morphological patterns and movement-dependent temporal pat- terns of emotional facial expressions. What functions are involved in the encoding (display) and the decoding (judgmen0 processes? How are these functions related to memory processes (again in terms of encoding and decoding; i.e., storage and recall)? As mentioned before, in order to address these issues we need more refined, micro-analytical studies to determine the nature of the temporal-spatial signatures of different emotions as video- recorded under natural conditions. Data sets that allow this type of analysis currently exist but have not yet been systematically ana- lyzed. However, as Ekman said, on the basis of close study and measurement of more than 10,000 spontaneous facial expressions, he has found that usually the action units in an emotional expres- sion reach apex at the same time (Ekman, personal communica- tion, September 16, 1999).

Finally, approaches oriented more consistently toward the anal- ysis of dynamic features of facial expression are likely to encour- age further interest in the nature of the underlying mechanism. In the introduction, we suggested that there are at least two competing theoretical accounts--one, emanating from discrete emotion the- ory, suggesting simultaneous onset of facial action unit activation as a result of neuromotor programs; the other, based on appraisal- theoretic notions, suggesting a sequential-cumulative onset based on the adaptive consequences of specific event evaluations. In Study 2, we made a preliminary attempt to investigate the relative compatibility of each approach with the perception of dynamic facial expressions. Although the results were inconclusive, the approach still seems viable and the methodological approach ad- vocated here provides a promising tool for further critical exper- iments. As suggested by one of the reviewers, we are now planning an experiment that allows us to test the predicted time pattern by comparing it with a random ordering.

The design of critical experiments in this domain clearly re- quires more precise specification of pertinent details by the respec- tive theorists. For example, would proponents of neuromotor ap- proach predict simultaneous onset of all dements of an expressive pattern with equal intensity and postulate the same point in time as apex for all action units involved? Or, with respect to the component-process model, given the transitory nature of appraisals and their motor efference in expression, what is the decay function of action unit innervation once it is no longer sustained by appro- priate appraisals?

Another important issue concerns the nature of the apex of an expression episode in a system with sequential cumulative inner- vations of muscles. There might be several local apexes, depend- ing on the accumulation and combination of different action units and their intensity. Once one enters into the details of synthetic facial expression production according to theoretical models, the

demands for additional specification of details and theoretical predictions will grow exponentially. Thus, the adoption of the procedures presented here might benefit not only methodological but also theoretical development. One of the advantages of begin- ning to design critical experiments is that attention is directed toward hitherto underspecified parts of particular theoretical mod- els. This in itself, quite apart from the significance of specific data analyses, seems to justify further experimental work using dy- namic synthetic facial expressions.

References

B~inninger-Huber, E., & Rauber-Kaiser, S. (1989). Die Differenzierung versclaiedener Lacheltypen: FACS-Codierung und Einsch~tzungen. Eine Untersuchung zur Eindrncksbildung [Different types of smiling: FACS- coding and judgments]. Schweizerische Zeitschrift fiir Psychologic, 48, 21-34.

Banse, R., & Scherer, K. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70, 614-636.

Darwin, C. (1876). The expression of the emotions in man and animals. London: Murray.

Ducherme, G. B. A. (1872). Mechanisme de la Physionomie Humaine: Ou, Analyse Electrophysiologique de l'expression des passions [The mech- anism of human facial expression: Or, the electrophysiological analysis of the expression of passions]. Pads: Bailli~re.

Ducbenne, G. B. A. (1990). The mechanism of human facial expression (R. A. Cuthbertson, Ed. and Trans.). Cambridge and New York: Cam- bridge University Press. (Original work published 1872)

Ekman, P. (1976). Pictures of facial affect. Palo Alto, CA: Consulting Psychologists Press.

Ekman, P. (1985). Telling lies. New York: Norton. Ekman, P. (1992). An argument for basic emotions. Cognition and Emo-

tion, 6, 169-200. Ekman, P. (1994a). All emotions are basic. In P. Ekman & R. J. Davidson

(Eds.), The nature of emotion: Fundamental questions (pp. 15-19). Oxford, England: Oxford University Press.

Ekman, P. (1994b). Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique. Psychological Bulletin, 115, 268- 287.

Ekman, P. (1997). What we have learned by measuring facial behavior. In P. Ekman & E. L. Rosenberg (Eds.), What the face reveals (pp. 469- 484). Oxford, England: Oxford University Press.

Ekman, P., Davidson, R. J., & Friesen, W. V. (1990). The Duchenne smile: Emotional expression and brain physiology: II. Journal of Personality and Social Psychology, 58, 342-353.

Ekman, P., & Friesen, W. V. (1975). Unmasking the face. Englewood Cliffs, NJ: Prentice-Hall.

Ekman, P., & Friesen, W. V. (1978). Manual for the facial action coding system. Palo Alto, CA: Consulting Psychologists Press.

Ekman, P., Friesen, W. V., & O'Sullivan, M. (1988). Smiles when lying. Journal of Personality and Social Psychology, 54, 414-420.

Ekman, P., Levenson, R. W., & Friesen, W. V. (1983, September 16). Autonomic nervous system activity distinguishes between emotions. Science, 221, 1208-1210.

Ekman, P., & Rosenberg, E, L. (Eds.). (1997). What the face reveals. Oxford, England: Oxford University Press.

Ellswotth, P. C., & Smith, C. A. (1988). Shades of joy: Patterns of appraisal differentiating pleasant emotions. Cognition and Emotion, 2, 301-331.

Frank, M. G., Ekman, P., & Friesen, W. V. (1993). Behavioral markers and recognizability of the smile of enjoyment. Journal of Personality and Social Psychology, 64, 83-93.

Frijda, N. H,, & Tcherkassof, A. (1997). Facial expressions as modes of action readiness. In J. A. Russell & J. M. Fernzlndez-Dols (Eds.), The

SYNTHETIC FACIAL EXPRESSIONS OF EMOTION 117

psychology of facial expression (pp. 78-102). Cambridge, England: Cambridge University Press.

Galati, D., Scherer, K. R., & Ricci-Bitti, P. (1997). Voluntary facial expression of emotion: Comparing congenitally blind to normal sighted encoders. Journal of Personality and Social Psychology, 73, 1363-1380.

Gosselin, P. (1989)i QualiMs expressives et communicatrices des config- urations faciales dmotionneUes [Expressive and communicative quali- ties of emotional facial configurations]. Unpublished doctoral disserta- tion, Universit6 Laval. Qutbec.

Gosselin, P., Kirouac, G., &Dort, F. Y. (1995). Components and recog- nition of facial expression in the communication of emotion by actors. Journal of Personality and Social Psychology, 68, 1-14.

Hess, U., Kappas, A., McHugo, G. J., Kleck, R. E., & Lanzetta, J. T. (1989). An analysis of the encoding and decoding of spontaneous and posed smiles: The use of facial electromyography. Journal of Nonverbal Behavior, 13, 121-137.

Hjortsjt, C. H. (1970). Man's face and mimic language. Lurid, Sweden: Student-Litteratur.

Izard, C. E. (1971). The face of emotion, New York: Appleton-Century- Crofts.

Izard, C. E. (1979). The maximally discriminative facial movement coding system (MAX). Newark: University of Delaware.

Izard, C. E. (1991). The psychology of emotions. New York, NY: Plenum Press.

Izard, C. E. (1992). Basic emotions, relations among emotions, and emotion-cognition relations. Psychological Review, 99, 561-565.

Izard, C. E. (1994). Innate and universal facial expressions: Evidence from developmental and cross-cultural research. Psychological Bulletin, 115, 288-299.

Izard, C. E., & Haynes, O. M. (1988). On the form and universality of the contempt expression: A challenge to Ekman and Friesen's claim of discovery. Motivation and Emotion, 12, 1-16.

Kalra, P., Mangili, A., Magnenat-Thalmann, N., & Thalmann, D. (1992). Simulation of muscle actions using rational free form deformations. Computer Graphics Forum, 2, 59-69.

• Ladd, D., Silverman, K., Tolkmitt, F., Bergmann, G., & Scherer, K. (1985). Evidence for the independent function of intonation contour type, voice quality, and F0 range in signaling speaker affect. Journal of the Acous- tical Society of America, 78, 435-444.

Lemay, G., Kirouac, G., & Lacouture, Y. (1995). Expressions faciales 6motionelles spontan6es dynamiques et statiques: comparaison d'6tudes de jugement cat6goriel et dirnensionel [Dynamic and static emotional facial expressions: A comparison of categorical and dimensional judg- ment studies]. Canadian Journal of Beha~,ioral Science, 27, 125-139.

Lewis, M., Stanger, C., & Sullivan, M. W. (1989). Deception in 3-year- olds. Developmental Psychology, 25, 439-443.

Malatesta, C. Z., & Haviland, J. M. (1982). Learning display rules: The socialization of emotion expression in infancy. Child Development, 53, 991-1003.

Musterle Mimikfaces (Program Face, PC version) [Computer software]. (1990). Mtinchen, Germany: MPI ftir Psychiatrie.

Ortony, A., & Turner, T. J. (1990). What's basic about basic emotions? Psychological Review, 97, 315-331.

Ricci-Bitti, P. E., Caterina, R., & Garotti, P. L. (1996). Different behavioral

markers in different smiles. In N. H. Frijda (Ed.), Proceedings of the Eighth Conference of the International Society for Research on Emo- tions (pp. 297-301). Storrs, CT: ISRE Publications.

Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bul- letin, 115, 102-141.

Russell, J. A., & Ferngmdez-Dols, J. M. (1997). The psychology of facial expression. Cambridge, England: Cambridge University Press.

Scherer, K. R. (1984). On the nature and function of emotion: A compo- nent process approach. In K. R. Scberer & P. Ekman (Eds.), Approaches to emotion (pp. 293-318). HiUsdale, NJ: Edbaum.

Scherer, K. R. (1986). Vocal affect expression: A review and a model for future research. Psychological Bulletin, 99, 143-165.

Scberer, K. R. (1987). Toward a dynamic theory of emotion: The compo- nent process model of affective states. Geneva Studies in Emotion and Communication, 1, 1-98. Available on the World Wide Web: http:// www.unige.ch/fapse/emotion/papers/papers.html.

Scherer, K. R. (1988). Criteria for emotion-antecedent appraisal. In V. Hamilton, G. H. Bower, & N. H. Frijda (Eds.), Cognitive perspectives on emotion and motivation (pp. 89-126). Dordrecht, the Netherlands: Nijhoff.

Scberer, K. R. (1992). What does facial expression express? In K. T. Strongman (Ed.), International review of studies on emotion, VoL 2 (pp. 139-164). Chichester, England: Wiley.

Scherer, K. R. (1993). Studying the emotion-antecedent appraisal process: An expert system approach. Cognition and Emotion, 7, 325-355.

Scherer, K. R. (1994). Toward a concept of "modal emotions." In P. Ekman & R. J. Davidson (Eds.), The nature of emotion: Fundamental questions (pp. 25-31). Oxford, England: Oxford University Press.

Scherer, K. R., & Oshinsky, J. (1977). Cue utilization in emotion attribu- tion from auditory stimuli. Motivation and Emotion, 1, 331-346.

Smith, C. A. (1989). Dimensions of appraisal and physiological response in emotion. Journal of Personality and Social Psychology, 3, 339-353.

Smith, C. A., & Ellsworth, P. C. (1985). Patterns of cognitive appraisal in emotion. Journal of Personality and Social Psychology, 4, 813-838.

Smith, C. A., & Scott, H. S. (1997). A componential approach to the meaning of facial expression. In J. A. Russell & J. M. FeruAndez-Dols (Eds.), The psychology of facial expression (pp. 229-254). Cambridge, England: Cambridge University Press.

Stettner, L. J., Ivery, E., & Haynes, O. M. (1986, September). The human smile: Analysis of structural features and perceived meanings. Paper presented at the meeting of the International Society for Human Ethol- ogy, Tutzing, Germany.

Tomkins, S. S. (1984). Affect theory. In K. R. Scherer & P. Ekman (Eds.), Approaches to emotion (pp. 163-196). Hillsdale, NJ: Erlbaum.

Wallbott, H. G., & Scherer, K. R. (1986). Cues and channels in emotion recognition. Journal of Personality and Social Psychology, 51, 690- 699.

Waters, K., & Terzopoulos, D. ( 1991 ). Modeling and animating faces using scanned data. Journal of Visualization and Computer Animation, 2, 123-128.

Wehrle, T. (1995). The Facial Action Composing Environment [Unpub- lished computer software]. Geneva, Switzerland.

(Appendixes follow)

118 WEHRLE, KAISER, SCHMIDT, AND SCHERER

A p p e n d i x A

P r e d i c t i o n s o f F a c i a l E x p r e s s i o n C h a n g e s ( I n d i c a t e d as A c t i o n U n i t C o m b i n a t i o n s ) f o r T w o O u t c o m e s o n S p e c i f i c

A p p r a i s a l D i m e n s i o n s

Appraisal dimension Outcome A Outcome B

Novelty High Low

Suddenness (sensory-motor) 1 + 2 + 5 + 26/27 Familiarity (schematic) - - 4b + 7 Predictability (conceptual) - - 4b + 7 longer than familiarity, less intense than dissonant

Intrinsic pleasantnes¢ High Low

Taste: 6 + 12 + 25/26 Taste: (9/10) + 16 + 19 + 26 Sight: 5a + 26a Sight: 4 + 7/(43)/44 + 51/52 + (61/62) Smell: 26a + 38 Smell: 9 + (10) + [15 + 17] + (24) + 39 Sounds: 12 + (25) + 43 Sounds: each combination of the others

Goal significance

Concern relevance

Outcome probability Expectation

Conduciveness Urgency

High focusing responses: lower intensity of the cumulation of the two first SECs

Probable higher intensity for future responses Consonant--

Conducive: 6 + 12 Low deamplification, low tension

Low terminating of the responses of the two In'st SECs

Not probable lower intensity for future responses Dissonant reactivation of novelty responses: 1 + 2 + 5 or

4 d + 7 Obstruct: 4e (long) + 7 + 17 + 23/24 High intensification, high tension

Coping potential

Cause: agent

Cause: motive

Control

Power

Adjustment

Internal personal (self) + external not personal (natural agent) attribution less intense than external personal attribution

Not intentional diminution of intensity of existing and future responses

Low: 15 + 25/26 + 41/42/43 + 54 + 61/62 + 6 4 ( 1 + 4 )

Low: 20 + 26/5 + freezing

Low holding the existing patten

External personal attribution (other) intensify existing and future responses, more intense than internal personal or external not personal attribution

Intentional intensify existing and future responses, more intense than not intentional

High intensify existing and future responses

High: 5 + [10 + 23 + 25]/[17 + (23) + 24] + 38 + (53/57)

High deamplificatiun, (12)

Compatibility standards

Internal

External

Surpassed

Self: 17 + 24 + (53)

Self: 17 + 24 + (53) Other: direct gaze + 1 + 2 + 5 + 26

Violated

Self: 41/42/43 + 54 + 55/56 + 61/62, + 64 (gaze avoidance + head down)

Self: 41/42/43 + 54 + 55/56 + 61/62 + 64 Other: 4 + 10L/R + (12x) + [53 + 64] (gaze avoidance +

head down)

Note. Action unit numbers and names: 1 (inner brow raiser); 2 (outer brow raiser); 4 (brow lowerer); 5 (upper lid raiser); 6 (cheek raiser); 7 (lid tightener); 9 (nose wrinkler); 10 (upper lip raiser); 12 (lip comer puller); 14 (dimpler); 15 (lip comer depressor); 16 (lower lip depressor); 17 (chin raiser); 19 (tongue show); 20 (lip stretcher); 23 (lip tightener); 24 (lip pressor); 25 (lips part); 26 (jaw drop); 27 (mouth stretch); 38 (nostril dilate); 39 (nostril compress); 41 (lid droop); 42 (slit); 43 (eyes closed); 51/52 (head turn left/fight); 53/54 (head up/down); 55/56 (head tilt left/right); 61/62 (eyes left/right); 63/64 (eyes up/down). Special characters and their signification: / = alternatives within groups, ( ) = optional, [ ] = together, - - = no changes predicted. Items in boldface represent specific outcomes on an appraisal dimension. SECs = stimulus evaluation checks. a For non-sensory-modality related experiences (e.g., aesthetic experiences): generalization involving any subcombination.

SYNTHETIC FACIAL EXPRESSIONS OF EMOTION

A p p e n d i x B

P red i c t i ons for the Appra i s a l Pa t t e rn and the R e l a t e d A c t i o n U n i t s ( A U ) for Sadnes s

119

Appraisal dimensions Sadness ~ Action units b

Novelty Suddenness Low Familiarity Low Predictability

Intrinsic pleasantness Goal significance

Concern relevance Outcome probability Very high Expectation Conduciveness Obstruct Urgency Low

Coping potential Cause agent Cause motive Chance or negligence Control Very low Power Very low Adjustment Medium

Compatibility standards External Intemal

AU4a + AU7a

Intensify future intensity

AU4b + AU7b + AU17b + AU23b

AUlc + AUI5c + AU41 + AU64 AU20c + AU26

Note. AU numbers and names: 1 (inner brow raiser); 4 (brow lowerer); 7 (lid tightener); 15 (lip comer depressor); 17 (chin raiser); 20 (lip stretcher); 23 (lip tightener); 26 (jaw drop); 41 (lid droop); 64 (eyes down). After AU numbers, the intensity of the movement is noted on the 5 intensity levels of the Facial Action Coding System, from a (trace level) to c (highest intensity).

For empty cells, the prediction is open. b For empty cells, there are no changes in facial behavior predicted.

Received July 14, 1999 Accepted July 14, 1999 •