So nification of the facial expressio ns New Interaction Techniques Department of Computer Science University of Tampere 27 February 2003 Guo Zhiguo Ioulia.

So nification of the facial expressio ns

New Interaction Techniques

Department of Computer Science University of Tampere

27 February 2003

Guo Zhiguo Ioulia Guizatdinova [email protected] [email protected]

Contents

• Introduction to the problem

• Background & related research

• Objectives

• Methods

• Resources & experimental setup

• Preliminary results

• Prospectives

p 02_23 27.02.03 Sonification of the facial expressions

Sonification is the use of non-speech audio to convey information

more specifically sonification is the transformation of data

relations into perceived relations in an acoustic signal for the

purposes of facilitating communication or interpretation [1]

p 03_23 27.02.03 Sonification of the facial expressions

Introduction to the problem

The main question for sonification of the facial expression is:

How to convey the emotional expressions/facial features/ of some

person presented like any visual imaging into the sounds

(e.g. earcons, music chords) or whatever ?

Introduction to the problem

p 04_23 27.02.03 Sonification of the facial expressions

The sonification system for facial expressions should allow the

transformation of the visual information (attributes specifically

describing emotional expression/state) into sound patterns

This kind of sounds (“eARmoticons”) should evoke similar

emotional feelings of the listener

that is, to provide an emotional access through “audio snapshot”

when visual perception is blocked or imaging is hidden for

observer

p 05_23 27.02.03 Sonification of the facial expressions

Introduction to the problem

Some facial expressions can hardly be

transformed by sound (e.g., congenital or

non-classified features)

Blind people have a wide experience with

real environmental sounds, while the

congenital blinded can have not any

association to visual facial imaging; only

tactile memory of some facial features can

be insufficient or non-adequate

p 06_23 27.02.03 Sonification of the facial expressions

Background & Related Research

There is not signs’ system/auditory or tactile/specifically & briefly describing facial features/expressions. Only some general emotional states are presented like artificial set of static visual/tactile symbols [2, 11].

p 07_23 27.02.03 Sonification of the facial expressions

Background & Related Research

Happiness

: )

Sadness

: (

Frustration

: / >:-(

Madness

There have been a lot of approaches to data sonification of like math

functions, simple graphics, diagrams, …

p 08_23 27.02.03 Sonification of the facial expressions

http://www.visualprosthesis.com/javoice.htm

The vOICe Java Applet

Background & Related Research

p 09_23 27.02.03 Sonification of the facial expressions

Relevantly, there are a lot of earcons used in MS Windows

and other software, sound signaling for special equipment

and application…

sound soundsoundsound

Background & Related Research

Objectives

The goal of this project is

designing and usability investigation of the sonification

system for attributes specifically describing facial expression

to provide an emotional access through eARmoticons when

visual perception is blocked or imaging is hidden for

observer

p 10_23 27.02.03 Sonification of the facial expressions

Schedule

Project’s activity Weeks Expected results

Overview on the topicDefinition of the interaction scenarios & research methods

1Sonification & visual

attributesScenarios

Design of software prototypeDesign & collection of sound files

1Software prototype

Sound database

Classification of the facial expressions by soundRevision of software

1Sonification System for

facial attributesRevised software

Usability exploration of the new interaction techniques

2 Data

Data analysis Report writing

12

Project paper

p 11_23 27.02.03 Sonification of the facial expressions

Research Methods

• constructive research to design and simulate the new techniques, software tools and doing usability evaluation of novel sound objects – eARmoticons

• empirical research is based on crossmodal information transfer and perception of intermodal events

p 12_23 27.02.03 Sonification of the facial expressions

Preliminary Modelling

p 13_23 27.02.03 Sonification of the facial expressions

Facial expression

(image, picture, video, etc)

sounds

Video processing

Facial expression-to-sound mapping transformation system

Recognition of the sound patterns to facial expression

As a whole

Facial landmarks

Sound Database

Resources & Experimental Setup

p 14_23 27.02.03 Sonification of the facial expressions

• Subjects (people with normal vision)

• Stimuli – eARmoticons – earcons specifically & briefly describing

facial features/expressions

FACS – Facial Action Coding System [ 7, 8]

a prototype of the sonification system for facial expression transformation

Resources & Experimental Setup

p 15_23 27.02.03 Sonification of the facial expressions

Experimental procedure

Resources & Experimental Setup

p 16_23 27.02.03 Sonification of the facial expressions

Data analysis:reaction timeaccuracy = error

number

Preliminary results

p 17_23 27.02.03 Sonification of the facial expressions

Classification of the emotions by Ekman & Friesen (1975)

http://www.paulekman.com/

surprise happiness angerdisgustsadness fear

NEUTRALPOSITIVE NEGATIVE

p 18_23 27.02.03 Sonification of the facial expressions

Inner brow raiser

Outer brow raiser

Brow lowerer

Upper lid raiser

Cheek raiser

Lid tightener

Lid droop

Slit

Eyes closed

Squint

Facial landmarks.Eye & eyebrow

Preliminary results

p 19_23 27.02.03 Sonification of the facial expressions

Lip pressor

Lip corner puller

Lips part

Jaw drop

Mouth stretch

Lip funneler

Lip suck

Cheek puffer

Dimpler

Lip corner depressor

Lower lip depressor

Lip puckerer

Lip stretcher

Lip tightener

Chin raiser

Facial landmark. Mouth

Preliminary results

p 20_23 27.02.03 Sonification of the facial expressions

Nose wrinkler

Upperupper lip raiser

Nasolabial deepener

Facial landmark. Nose

Preliminary results

The sonification system of facial expressions which

includes new sound mapping techniques to display

facial features and expressions

Prospectives

p 21_23 27.02.03 Sonification of the facial expressions

1. http://www.icad.org/websiteV2.0/Resources/glossary.html#S2. http://www.tsbvi.edu/Education/vmi/tactile_symbols.htm#EMOTIONS3. http://www.mcsdtechcenter.org/multimedia/sounds-samp.htm4. http://www.matthawkins.co.uk/html/sound_fx_index.htm5. http://kin-dza-dza.etel.ru/ http://www.soundobject.org/6. http://viswiz.gmd.de/~eckel/publications/eckel98a/eckel98a.html7. http://www-2.cs.cmu.edu/afs/cs/project/face/www/facs.htm8. http://www.paulekman.com/9. http://www.ifeelpixel.com/download/whatsnew.htm 10. http://www.symbols.net/emoticons.htm11. http://www.fingertek.com/smilies.htm12. http://www-unix.mcs.anl.gov/~kaper/Sonification/13. http://www-pablo.cs.uiuc.edu/Project/Pablo/PabloSonification.htm

references

p 22_23 27.02.03 Sonification of the facial expressions

27.02.03 Sonification of the facial expressions

Thank yo u for yo urattentio n!