Affective UX: Challenges in UX involving affective computing

Design experience first; design software afterwards:

Affective UXChallenges in UX design involving affective computing

Alfredo SnchezUniversidad de las Amricas Puebla

01WORLD IA DAY 2016PRESENTATION TITLE HEREHEADER OPTIONSub head or short descriptionSome kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.Some kind of illustration or image?

2006-2014, Alfredo Snchez

http://www.udlap.mx

Architecturing happinessHow do we know users are happy?

Can interactive systems detect happiness (or the lack thereof)?

Can interactive systems intentionally produce happinesss?

Can technology at least not get in the way? 2006-2016, Alfredo Snchez

AgendaTerms of affectionAffective computingDetectionSynthesisMediationAffect in UXRecent developmentOpen issues

Terms of affection


EmotionA reaction experienced usually toward a specific object and typically accompanied by physiological and behavioral changes in the body


MoodA conscious state of mind (typically lasting longer than an emotion)

joydepression 2006-2014, Alfredo Snchez

SentimentAn attitude or opinion a state that results from attributes assigned to objects, which are grouped into categories


The affect spectrumemotionmoodsentiment

may triggermay prompt

Duration( - ) ( + ) 2006-2014, Alfredo Snchez

9

Affective computing

2006-2014, Alfredo Snchez(Computer sytems) related to, arising from, or deliberately influencing affective phenomena

MIT Affective Computing Group

Affective ComputingRosalind Picard1995

Affective computing (for engineers) The study and development of systems and devices that can recognize, interpret, process, simulate and mediate human affects.


Why affective computing?


Why affective computing?

2006-2014, Alfredo SnchezI choose not to feelhttp://animala.mx/noticias/2015/02/27/adios-mr-spock/

Why Affective ComputingReason is, and ought to be, only the slave of the passionsDavid Hume, 1711-1776

Without passions we would lack all motivation, all impulse or drive to act, or even to reason 2006-2014, Alfredo Snchez

Why affective computing?A love/hate relationship with the computer and ITs in general 2006-2014, Alfredo Snchez

https://www.youtube.com/watch?v=ZcZBTB-dkqQ

Why affective computing? (for engineers) Studies by many researchers show a vital role of emotion in many processes:

PerceptionDecision makingCreativityComprehensionMemorySocial interaction 2006-2014, Alfredo Snchez

http://www.bintegra.com/wp-content/uploads/2012/05/Fotolia_socila_users_S.jpg

Some application areasEducation intelligent tutoringHealth monitoring patientsSafety monitoring drivers / pilotsSecurity detecting liesE-commerce attitudes towards products or servicesVideo games producing engaging environments, enriching UXUser studies incorporating emotion into evaluation


Why affective computing (for our community)Emotion is the way we communicate our experience 2006-2014, Alfredo Snchez

Main aspects of affective computing

perception

synthesismediation 2006-2014, Alfredo Snchez

(of emotions, moods and sentiments)

19

Affect detection


Emotion detection: From facial cues 2006-2014, Alfredo Snchez

http://npx.autismspot.com/facial_expression.jpg

Emotion detection: From postural cues 2006-2014, Alfredo Snchez

http://instridebodywork.com/wp-content/uploads/2013/05/postures-of-emotion1.jpg

Emotion detection: From voice cues 2006-2014, Alfredo Snchez

http://pandodaily.files.wordpress.com/2013/07/whisper-in-ear.jpg?w=584&h=329http://gigaom2.files.wordpress.com/2013/07/emotion-speech.jpg?w=300&h=200&crop=1

Emotion detection: From usage patterns 2006-2014, Alfredo Snchez

http://web.deepnetsecurity.com/products2/typesense.asp

Keystroke dynamicsNavigation patternsEye tracking

Emotion detection: From brain activity 2006-2014, Alfredo Snchez

Emotiv EPOC sensorHonda BCI system

Emotion detection: From sensors of autonomic activitySkin conductance increases even with a little surprise or anxiety

Blood volume/pressure increases with anger or stress

Respiratory rhythm varies with fear, sadness, happiness, anxiety, fear

Muscle tension increases with surprise and anxiety


Affectiva sensor

NeXus sensor

Emotion detection: From sensors of autonomic activity 2006-2014, Alfredo Snchez

Know your inner selfW/Me

Hack happinessFeel (by Sentio)

Some general techniques to support affective computingFor detection, synthesis and mediation 2006-2014, Alfredo Snchez

Facial Action Coding System (FACS) 2006-2016, Alfredo Snchez

http://www.theroadtosiliconvalley.com/tag/tim-roth/

FACSFacial Action Coding System (FACS), very popularDefines Action Units (AUs), contraction or relaxation of one or more face muscles


FACS (for universal emotions)

http://www.paulekman.com/facs/ 2006-2014, Alfredo Snchez

Posture, motion and gesture analysis 2006-2014, Alfredo Snchez

Posture analysis

http://people.ict.usc.edu/~gratch/CSCI534/ACII-Handbook-GestureSyn.pdf 2006-2014, Alfredo Snchez

Posture and motion analysisE.g. arm movements and emotion: 2006-2014, Alfredo Snchez

http://people.ict.usc.edu/~gratch/CSCI534/ACII-Handbook-GestureSyn.pdf

Gesture analysis 2006-2014, Alfredo Snchez

http://people.ict.usc.edu/~gratch/CSCI534/ACII-Handbook-GestureSyn.pdf

MPEG-4MPEG-4 (ISO144996) defines 86 Face Animation Parameters (FAPs)196 Body Animation Parameters (BAPs) 2006-2014, Alfredo Snchez

Some existing toolsFaceReader, SHORE, eMotion, RealEyes 2006-2014, Alfredo Snchez

Some challengesPrototypical vs. non-prototypical emotions Pure vs. combined emotionsMultiple subjectsPartly covered faces / bodiesTracking face or body movementLighting conditions

Integrating techniquesSupporting application development 2006-2014, Alfredo Snchez

Emotion in UX evaluation


Evaluating user experience with novel technologies impact of emotion (Gestures + contextual menus - video)

2006-2014, Alfredo Snchez[Magallanes, Molina, Mndez& Snchez 2012]

Heuristic evaluationProblems observed in actual use 2006-2014, Alfredo SnchezIssues identified by expertsIDIssueOccurrencesP1Movement of objects using two fingers.13P2Size of the keyboard.8P3Finding help for the user.5P4Finding the toolbar associated with each object.4P5Navigating using the fan menu.3P6Navigating between nodes.3P7Identifying the icon for linking to new pages 2P8Incomplete or inaccessible menus.1

IDTaskIssue1Insert two images, change their position and size.P12Enter a text next to the images as a caption.P23Obtain the toolbar for an image.P3, P44Pop a menu and discover how it works.P55Build a hierarchy of linked workspaces.P6, P76Get the menu in different positions. Cause the menu to go off the screen.P87Insert two images, change their position and size.P18Enter a text next to the images as a caption.P2

Additional feedback on emotions, based on Product Emotion Measurement Tool (PrEmo)

[Desmet 2002]Participants were asked to select an animated representation in correspondence with their emotional reactions

2006-2014, Alfredo Snchez[In collaboration with Universidad del Cauca, Colombia]

Positive emotions in spite of problems! 2006-2016, Alfredo Snchez

Positive emotions associated with novelty?Negative emotions associated with known interface elements?[Magallanes, Molina, Mndez& Snchez 2012]

ImplicationsTechniques for evaluation of usability and user experience are needed that take affect into account

Framework developed at UniCauca, Colombia[Mndez, Collazos, Granollers,Snchez 2014; Mndez 2015] 2006-2014, Alfredo Snchez

EmoValA framework for emotional evaluation designed to enrich existing UX evaluation methodsThree major componentsEmoVal-C: Behavioral evaluation (facial expressions, gaze, posture, movements)EmoVal-S: Self-report of emotions while using a systemEmoVal-F: Physiological sensing, mainly brain activity during system usageSupporting website (http://artemisa.unicauca.edu.co/~ymendal/EmoVal) 2006-2016, Alfredo Snchez

[Mndez 2015]

EmoVal: Detailed support / facial expressions 2006-2016, Alfredo Snchez

[Mndez 2015]

EmoVal: Detailed support / posture and movement 2006-2016, Alfredo Snchez

[Mndez 2015]

Building capabilities for affect detection


Current scenario: Extending an app to include more techniques is not straightforward

Application domainTechniqueTechnologies 2006-2014, Alfredo Snchez

Goal: Integration platform

2006-2014, Alfredo Snchez[Snchez et al. 2015]

Architecture for implementation


Simple data model and messaging protocol


Prototypical apps: Kinect-based emotion detection 2006-2014, Alfredo Snchez

14 sorpresa[Snchez et al. 2015]

Prototypical apps: Kinect-based emotion detection 2006-2014, Alfredo Snchez[Snchez et al. 2015]

Prototype evaluation


Prototypical apps: Android-based emotion self reporter 2006-2014, Alfredo Snchez

[Snchez et al. 2015]

Prototype:Monitoringservices


BCIs for UXhttp://www.gadgetify.com/wp-content/uploads/2014/09/15/epoc.png

Experimenting with EPOC+ (video)

What about sentiment?


Sentiment analysisBy definition,The use of NLP and computational linguistics to identify attitudes in speakers or writers with respect to specific subjects 2006-2014, Alfredo Snchez

http://datameer2.datameer.com/blog/wp-content/uploads/2012/10/22nd.png


Applications of sentiment analysisDetecting / studying:

Responses to marketing/political campaigns User satisfactionBias in newsAppropriateness of contents for different audiences 2006-2014, Alfredo Snchez

Some techniquesMostly heuristic methodsSocial media analysis

Keyword-basedSemantic approachesAnnotated datasetsSupport Vector Machines (SVMs) for classification


Expression ArchitectureA flexible architecture enables Expression to listen from any data source and have an independent component-based front-end.Expression Sentiment Analysis Platform at UDLAPArchitecture[Cervantes, Gutirrez, Gutirrez, Castillo,Snchez, Wan 2015]

The front end of Expression is based on sentiments cards that represent topics of interest.Sentiment cardshttps://github.com/Innova4D/expression

Resaltar el concepto de REACTIVO

A Sentiment Card graphically summarizes opinion trends in social media

Sentiment Card

Aqu es importante mencionar que se pueden visualizar diferentes fuentes de datos, solamente debe conectarse al social data manager y es una de las cosas interesantes del sentiment card. Incluso los usuarios pueden escribir comentarios desde la aplicacin para contribuir.

Some challengesLanguage subtleties Irony, sarcasm, abbreviations, misspellings, idiomsRepresentativeness Dissatisfied users tweet more activelyContext dependent semantics SpamClassification beyond positive or negative 2006-2014, Alfredo Snchez

Affect mediation


Mediation: Conveying affect through ICTsVideoconferencing

Immersive environments

Asynchronous communicationEmail, forums, blogs 2006-2014, Alfredo Snchez

Some issues and challengesAudio and video quality over InternetQuality of multimedia experience(http://i-guide.org/)Metrics, assessment methods

Gaze awareness

3D modelingIssues shared with affect synthesis


Emotion over low data rates:Sending codes instead of full videoIn collaboration with University of Agder, Norwayhttp://www.paulekman.com/facs/

SensingEncodingTransmittingReconstructing[Lamei, Li, Snchez 2014]

Conveying mood


Instant messagingEmoticons work for emotions, instant reactions associated to specific statements

Mood is longer lasting, overarchingIdentifying mood can be a challenge, even for the person in that affective state 2006-2014, Alfredo Snchez

Russells Circumplex Model can help identify mood

Two-dimensional, valence-arousal model

Consistent spatial location in self-report experiments across individuals and across cultures/languages 2006-2014, Alfredo Snchez

[Russell 1983]

Extending Russells model - Intensity 2006-2014, Alfredo Snchez

Extended, discrete model 2006-2014, Alfredo SnchezUser testing showedaccuracy in affect identification

But using the model directlywould result in a cluttered interface

Gradual revelationannoyedhappysadcalm

Representative affectives states per quadrant (according to user testing)sleepyarousedupsetsatisfied 2006-2014, Alfredo Snchez

78

Emoticon design 2006-2014, Alfredo Snchez

Prototype (Russkman)

[Snchez et al. 2006] 2006-2014, Alfredo Snchez

80

Prototype


81

Prototype


82

EvaluationChat sessions among 2-3 subjectsHypothetical situations (upcoming exams, good grades, recent party, lost wallet )Subjects were able to specify their mood rapidlySubjects were able to identify mood of other users (high correlation)Persistent emoticons helped in keeping awareness of moods 2006-2014, Alfredo Snchez

Affect synthesis: Modeling and displaying emotion and mood

Synthesizing emotionsIf an emotion (or mood or sentiment) is recognized in the user, the computer (or an adaptive environment) could (should?) respond accordingly

Approaches to emotion synthesisReasoning about emotionsgiven a set of conditions, produce specific reactionsCharacters with personality (temperament, sentiment) 2006-2014, Alfredo Snchez

85

Emotion synthesis

Characters with animated facial expressions(facial cues for sensing also apply here)[CSLU] 2006-2014, Alfredo Snchez

86

Responding to emotion

Affective CompanionText from automatic speech recognizer and EmoVoice(for the user)Natural language understanding[Cavazza et al. 2010] 2006-2014, Alfredo Snchez

FACS applied to emotion synthesis

FACSGen [Roesch et al. 2010] 2006-2014, Alfredo Snchez

Affective animated avatarsDesign using popular tools and a reduced set of FACS AUs

Available at http://chopis.com

Testing in Mxico and China


Some challengesPrototypical vs. non-prototypical emotions Pure vs. combined emotionsIntegrating facial, posture and voice expressions Offering service platforms for app construction 2006-2014, Alfredo Snchez

Perspectives of affective UXAffect is key in UX design and evaluation

Opportunities for research and development to improve detection, synthesis and mediation

Work needed in mood and sentiment analysis

Opportunities for inter-disciplinary collaboration 2006-2014, Alfredo Snchez

Affective computing becoming mainstream? 2006-2014, Alfredo Snchez

Pepper, SoftBanks affect-aware robot, 1,000 units sold out, December 2015, for about USD 1,800Involves facial recognition, video and audio processing, sensors, machine learning,

But waitForward-looking statements are by their nature subject to various risks and uncertainties, including, without limitation, a decline in general economic conditions, general market conditions, technological developments, changes in customer demand for products and services, increased competition, risks associated with international operations, and other important factors, each of which may cause actual results and future developments to differ materially from those expressed or implied in any forward-looking statement. [] Use of or reliance on the information in this presentation is at your own risk.

http://webcast.softbankmobile.co.jp/en/conference/20140605/index.html 2006-2014, Alfredo Snchez

Affective UXChallenges in UX design involving affective computing

Alfredo SnchezUniversidad de las Amricas Pueblahttp://ict.udlap.mx

01WORLD IA DAY 2016PRESENTATION TITLE HEREHEADER OPTIONSub head or short descriptionSome kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.Some kind of illustration or image?

ReferencesCervantes, O., Gutirrez, F., Gutirrez, E., Castillo E., Snchez, J. A., Wan, W. 2015. Expression: Understanding affective content in social media. 7th Latin American Conference on Human-computer Interaction (CLIHC 2015), Cordoba, Argentina, Nov. 18-23. Available from the ACM Digital Library (http://dl.acm.org/citation.cfm?doid=2824893.2824903).Magallanes, Y., Molina, A., Mndez, Y., Snchez, J. A. 2012. Combining gestures and graphical elements for collaboration using multitouch surfaces. Proceedings of the 22nd International Conference on Electronics, Communications and Computers (CONIELECOMP 2012, Feb.). IEEE Computer Society Press, 173-178.Magallanes, Y., Molina, A., Mndez, Y., Snchez, J. A. 2012. Towards an emotional validation of heuristic approaches for usability evaluation. Acta Universitaria 22, 3, 119-125. (http://www.actauniversitaria.ugto.mx/index.php/acta/article/view/351). Mndez, Y. A., Collazos, C. A., Snchez, J. A., Merchn Paredes, L. 2012. Criterios para la evaluacin de superficies multitctiles y sus aplicaciones. Lmpsakos 7 (Jan.), 64-68. (disponible en http://www.funlam.edu.co/revistas/index.php/lampsakos/article/view/847). Mndez, Y. A. 2015. Marco de referencia para la captura semi-invasiva y analisis de las emociones en el proceso de evaluacion con usuarios de sistemas interactivos. Tesis doctoral. Universidad del Cauca, Colombia. Disponible en http://www.unicauca.edu.co/doctoradoce/publicaciones/Monografia_Mendez.pdf


ReferencesSnchez, J. A., Hernndez, N. P., Penagos, J. C., Ostrvskaya, Y. 2006. Conveying mood and emotion in instant messaging by using a two-dimensional model for affective states. Proceedings of the Symposium on Human Factors in Computer Systems (IHC 2006, Natal, Brazil), 66-72. Available from the ACM Digital Library (http://dl.acm.org/citation.cfm?id=1298033&CFID=582956970&CFTOKEN=61490837). Mndez, Y. A., Collazos, C. A., Granollers, T., Snchez, J. A. 2014. Evaluating interactive systems from an emotional perspective. Revista Cientfica Guillermo de Ockham 12(1), 43-49. (disponible en http://www.redalyc.org/pdf/1053/105332478005.pdf). Snchez, J. A., Corts, X., Starostenko, O., Cervantes, O., Wan, W. 2015. An Extensible Platform for Seamless Integration and Management of Applications for Emotion Sensing and Interpretation. Journal of Ambient Intelligence and Smart Environments 7(1), 5-19. (available from http://iospress.metapress.com/content/y1g64422t78107v7). Starostenko, O., Corts, X., Snchez, J. A. 2015. Unobtrusive emotion sensing and interpretation in smart environments. Journal of Ambient Intelligence and Smart Environments 7, 59-83. (avalilable from http://iospress.metapress.com/content/w2681p2827l2u22p/).
