Humaine wp6 workshop 10/03/2005 1 annotation of emotions in meetings in the AMI project Roeland Ordelman* & Dirk Heylen Human Media Interaction University.

humaine wp6 workshop10/03/2005

1

annotation of emotionsin meetings

in theAMI project

Roeland Ordelman* & Dirk HeylenHuman Media Interaction

University of TwenteThe Netherlands


2

overview

• on the AMI project: ~100 hours of audio/video recordings of project meetings

• on annotating the emotional state of the participants in the meetings

• on emotion annotation in AMI• on implementing the annotation tool in

the NXT framework


3

AMI in brief

• European Integrated Project of the IST 6th FWP, initiated in January 2004 and involving 15 partners

• aims at advancing the state-of-the-art in important basic technologies such as human-human communication modeling, speech recognition, computer vision, multimedia indexing and retrieval

• produces tools for off-line and on-line browsing of multi-modal meeting data, including meeting structure analysis and summarizing functions

• makes recorded and annotated multimodal meeting data widely available for the European research community, thereby contributing to the research infrastructure in the field


4

on the AMI meetings

• scenario-based meetings – participants are asked to carry out a certain task and are

provided with role-restricted information– design project, involve one particular task (remote control

design)– ~35 hours (48 meetings)

• real design project meetings on– real design projects– such as engineering student projects, brochure or poster

design, etc. – ~35 hours

• other real or scenario-based meetings– cover a wider variety of meeting types, topics, behaviours,

etc. – ~30 hours


5

• smart meeting rooms at three different sites (TNO, Edinburgh, IDIAP)

– video (side camera, close up)– audio (mic-array, lapel, headset, manikin)– whiteboard strokes– pen

on the recording of the meetings


6

ami annotations

• many features of meeting interactions are annotated:– speech– gestures– dialogue acts– posture– emotion

• information is used for (at least) three purposes: – primary meta-data source for multi-featured browsing of

the recorded meetings.– training recognition algorithms that eventually should be

able to provide automatically generated meta-data. – evidence on the basis of which theoretical models of human

multi party interaction can be developed.


7

properties of the data

• ~100 hours of meeting data• emotion annotation of every single

speaker in meeting: 4 x 100 hours !• only a very small proportion can be

expected to show emotional states in any strong sense

• neutral state will cover much data


8

emotion annotation

• no general agreement on how to annotate or label emotional content in a natural database

• a number of emotion annotation or labeling schemes have been proposed in the literature

• given the gradations and subtlety of emotions occurring in natural data, the labeling of emotion using category labels is not straightforward


9

emotion annotation in AMI

Given:

1. the amount of data to be annotated, and2. the expected gradations and subtlety of emotions

occurring in meeting data

a dimensional labeling approach complemented by a categorical labeling scheme seemed most appropriate in the context of AMI

the FeelTrace software developed at Queens University Belfast (Cowie et al.) is reported to produce good quality annotations within a reasonable amount of time


10

FeelTrace

• judge the emotional experience of the participants of the meetings on two dimensions: arousal and valence.


11

annotation with FeelTrace in AMI

• survey on meeting specific landmarks• pilot annotations• re-implementation of FeelTrace


12

landmarks survey (1)

• (main investigator: Vincent Wan, University of Sheffield)

• Method:– list of 243 terms describing emotions– participants (37) had to select twenty emotions that they

most frequently perceived in their meetings– participants from various companies and with various job

descriptions, including lecturers, researchers, managers, secretaries and students.

– 243 terms were clustered by meaning into groups. The most frequently chosen one or two labels were shortlisted from each group. Taking some labels from each group ensures that there is sufficent coverage of the emotion space.


13


• list of 26 `meeting domain specific’ emotional words:

at ease, bored, joking, annoyed, nervous, satisfied, frustrated, amused, relaxed, interested, cheerful, uninterested, disappointed, agreeable, contemplative, encouraging, sceptical, friendly, attentive, confused, confident, decisive, impatient, concerned, serious, curious


14


• second survey to determine where each of the shortlisted labels should appear in FeelTrace emotional space:– first presented participants with the five labels:

anger, irritation, sadness, happiness and contentment (presented so that participants unfamiliar with FeelTrace would get some minimal experience in its use).

– emotional words were presented twice: the first to allow the participant to gain additional training and the second to collect data.


15

example results 1


16

example results 2


17

example results 3


18

resulting landmark distribution


19

annotation trials

• we have 15-20 annotators available• coming weeks: first annotation trials• what do we want to learn:

– inter-annotator agreement– distribution of emotional states in meeting

data (how much is neutral)– annotator experience with tool– effect of using/not using landmarks– validation of manual, annotation area, etc


20

trial annotation set-up

• set of 4 meetings, each of about 20 minutes, 4 speakers per meeting

• 10 minute segments (0-10,10-20)• (opt.) `dummy’ pass with Feeltrace• `offical’ pass with FeelTrace• with and without landmarks• no categorical labeling (yet)


21

implementation in NXT

• Belfast implementation of dimensional approach knows some limitations:– no cross-platform support– cannot easily be tailored to specific needs

from different `stakeholders’ of annotations (e.g., additional categorical of labeling of longer segments is hard in current setup)


22

stakeholders of annotation

• corpus developer– defines the structure of corpus, maintains the data, takes

care of a proper data/tool distribution (CVS, validation, time)

• corpus annotator– needs to know as little as possible about configuration,

installation and version control issues, only the annotation process itself is of concern for the annotator

• data consumer– interested in the annotations for analysis and may want to

configure in detail the annotation process (tool functionalities such as redo, fastforward, landmarks/no-landmarks, ect)

• tool developer– creates the tool that serves the needs of the other users,

keeping technical issues in mind


23

NXT

• Nite XML toolkit • defines a data storage format that can easily

be shared across a multitude of annotation and analysis tools for the many different aspects of a multi-modal corpus.

• the NXT libraries provide many ready-made components that facilitate easy development of new tools.

• expertise readily available at University of Twente


24

draft list of requirements• easy distribution of data (and tools) to and from the annotators (e.g., annotate

certain segments of a file, CVS functionality, batch functionality)• easy management of multiple annotators, possibly working on the same data

(e.g., CVS functionality)• validation functionality or possibilities to plug this into the process• as many input formats as possible on as many platforms as possible• customizable landmarks, dimensions (1D, 2D, 3D), shortcuts• categorical labeling options within the tool• video control (fastforward/backward)• progress bar• replay annotation aligned with video• selection of multiple video signals (close up, wide angle) • color coding of the 2D space (provide a priori color feedback instead of

feedback with a changing color of the cursor)• easy configuration in general

→ discuss requirements with AMI/HUMAINE researchers


25

roadmap

• starting first trial annotations• investigate trial results• discuss re-implementation of FeelTrace in

NXT with interested parties and create an first implementation version

• follow-up trials, FeelTrace-NXT versions• monitor process and discuss results with

researchers in the field (AMI, Humaine)

Humaine wp6 workshop 10/03/2005 1 annotation of emotions in meetings in the AMI project Roeland Ordelman* & Dirk Heylen Human Media Interaction University.

Documents

humaine wp6 workshop

data slide

netherlands slide

straightforward slide

curious slide

field slide

time slide

ami survey