Body and gestures (2): models, algorithms , applications Corso di Interazione uomo-macchina II Prof. Giuseppe Boccignone Dipartimento di Scienze dell’Informazione Università di Milano [email protected] http://homes.dsi.unimi.it/~boccignone/l
Body and gestures (2):models, algorithms , applications
Corso di Interazione uomo-macchina II
Prof. Giuseppe Boccignone
Dipartimento di Scienze dell’InformazioneUniversità di Milano
[email protected]://homes.dsi.unimi.it/~boccignone/l
DeMeuse (1987)
Non verbal behavior//De Meuse
A. Vinciarelli, M. Pantic, H. Bourlard, Social Signal Processing: Survey of an Emerging Domain,Image and Vision Computing (2008)
Posture and gesture interaction
Posture and gesture interaction
• There are two main challenges in recognizing posture and gestures:
• detecting the body parts involved in the gesture (e.g. hands)
• addressed by selecting appropriate visual features: these include, e.g., histograms of oriented gradients , optical flow, spatio-temporal salient points and space-time volumes .
• modeling the temporal dynamic of the gesture
• addressed by using techniques such as Dynamic Time Warping , Hidden Markov Models, and Conditional Random Fields .
Analysing postures and gestures
• The primary goal of gesture recognition research is to create a system which can identify specific human gestures and use them to convey information or for device control.
• A gesture may be defined as a physical movement of the hands, arms, face, and body with the intent to convey information or meaning.
• Gesture recognition, then, consists not only of the tracking of human movement, but also the interpretation of that movement as semantically meaningful commands
Analysing postures and gestures
• Like in the case of gestures, machine recognition of walking style (or gait) has been investigated as well, but only for purposes different from SSP, namely recognition and identification in biometric applications
• The common approach is to segment the silhouette of the human body into individual components (legs, arms, trunk, etc.), and then to represent their geometry during walking through
• vectors of distances , symmetry operators , geometric features of body and stride (e.g. distance between head and feets or pelvis)
Analysing postures and gestures
Analysing postures and gestures//application areas
• Automatic posture recognition has been addressed in few works, mostly aiming at
• surveillance
• activity recognition
• Few works where the posture is recognized as a social signal
• to estimate the interest level of children learning to use computers
• to recognize affective state of people
Analysing postures and gestures
http://www-prima.imag.fr/
Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.
Body gestures and postures//Generative model
Bodygesture
(mental conceptof)
Bodyposture
(limb motion)
Visual images
G P V
Tpg Tvp
G
P
V
P(V, P, G)=P(V | P) P(P | G) P(G)
Body gestures and postures//Generative model
Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.
G
P
V
P(G | V) =P(V | G) P(G)
P(V )
P(G) ∑H P(V | P) P(P | G)
∑G ∑H P(G) P(V | P) P(P | G)
=
Body gestures and postures//Generative model: inference
Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.
pointing
G
P
V
Body gestures and postures//Generative model: more complex model
Production and perception of body gestures. Body gestures originateas a mental concept G, are expressed (Tpg) through limb motionmotion B, and are perceived (Tvb) as visual images V.
pointing
body
walking golf swing
Body gestures and postures//Generative model
Gt
Pt
Vt
Gt+1
Pt+1
Vt+1
P(Gt+1 | Vt+1) ≈
P(Vt+1 | Gt+1) P(Gt+1 | Vt) ≈
P(Vt+1 | Gt+1) ∑Gt P(Gt+1 | Gt) P(Gt | Vt)
body
Body gestures and postures//Generative model: inference
Gt
Pt
Vt
Gt+1
Pt+1
Vt+1
body
Body gestures and postures//Generative model: inference
Body gestures and postures//Generative model: architecture
Body gestures and postures//Generative model: body models
tracked body parts indexed by different colors
Body gestures and postures//limb segmentation
tracked body parts indexed by different colors
Body gestures and postures//limb segmentation
Body gestures and postures//Body-part parameterization
ellipse convex hull
Body gestures and postures//Limb pose estimation: head
Body gestures and postures//Limb pose estimation: arm
Body gestures and postures//Limb pose estimation: leg
Body gestures and postures//Body-part parameterization
Body gestures and postures//Estimating body posture
Hand gestures
• Taxonomy of hand gestures for HCI
• Visual interpretation of hand gestures can help in achieving the ease and naturalness desired for Human Computer Interaction (HCI).
Hand gestures
• Classical use in HCI:
• In a computer controlled environment one wants to use the human hand to perform tasks that mimic both the natural use of the hand as a manipulator, and its use in human-machine communication
Hand gestures
• Classical use in HCI:
• In a computer controlled environment one wants to use the human hand to perform tasks that mimic both the natural use of the hand as a manipulator, and its use in human-machine communication
Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.
Hand gestures//Generative model
Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.
Hand gestures//Generative model
Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.
Hand gestures//Generative model
Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.
Hand gestures//Generative model
G
H
V
P(V, H, G)=P(V | H) P(H | G) P(G)
Production and perception of gestures. Hand gestures originateas a mental concept G, are expressed (Thg) through arm and handmotion H, and are perceived (Tvh) as visual images V.
Hand gestures//Generative model
G
H
V
P(G | V) =P(V | G) P(G)
P(V )
P(G) ∑H P(V | H) P(H | G)
∑G ∑H P(G) P(V | H) P(H | G)
=
Hand gestures//Generative model
Hand gestures//Generative model
Gt
Ht
Vt
Gt+1
Ht+1
Vt+1
P(Gt+1 | Vt+1) ≈
P(Vt+1 | Gt+1) P(Gt+1 | Vt) ≈
P(Vt+1 | Gt+1) ∑Gt P(Gt+1 | Gt) P(Gt | Vt)
Hand gestures//Generative model
• 3D hand model-based models of gestures use articulated models of the human hand and arm to estimate the hand and arm movement parameters. Such movements are later recognized as gestures.
• Appearance-based models directly link the appearance of the hand and arm movements in visual images to specific gestures
Hand gestures//Generative model: spatial models
Hand gestures//Generative model: spatial models
Hand gestures//Generative model: spatial models
3D Textured volumetric model
3D wireframe volumetric model.
3D skeletal model
Binary silhouette.
Contour
Gestures for augmented reality
Kinect style://http://www.openni.org/
Kinect style://http://www.openni.org/
Kinect style://http://www.openni.org/
OpenInterface//http://www.openinterface.org/platform/
Recognizing Affective Dimensions from BodyPosture
• Behavioral studies have shown that posture can communicate discrete emotion categories as well as affective dimensions.
• In the affective computing field, while models for the automatic recognition of discrete emotion categories from posture have been proposed, there are no models for the automatic recognition of affective dimensions from static posture.
• A study by Andrea Kleinsmith and Nadia Bianchi-Berthouze
Recognizing Affective Dimensions from BodyPosture (Kleinsmith and Bianchi-Berthouze)
Recognizing Affective Dimensions from BodyPosture
Recognizing Affective Dimensions from BodyPosture
Recognizing Affective Dimensions from BodyPosture
Recognizing Affective Dimensions from BodyPosture
Recognizing Affective Dimensions from BodyPosture
Recognizing Affective Dimensions from BodyPosture
Recognizing Affective Dimensions from BodyPosture
Recognizing Affective Dimensions from BodyPosture
Multimodal emotion recognition//Gunes & Piccardi
Multimodal emotion recognition//Gunes & Piccardi
Multimodal emotion recognition//Gunes & Piccardi
Multimodal emotion recognition//Gunes & Piccardi
Multimodal emotion recognition//Gunes & Piccardi
Multimodal emotion recognition//Gunes & Piccardi
Multimodal emotion recognition//Gunes & Piccardi
Multimodal emotion recognition//Gunes & Piccardi
• Results show that emotion classification using the two modalities achieves better recognition accuracy in general, outperforming the classification using the face modality only
• using expressive body information adds accuracy to the emotion recognition based on the face alone.
• early fusion seems to achieve a better recognition accuracy compared to late fusion.
Multimodal emotion recognition//Gunes & Piccardi
Anthropomorphic Embodied ConversationalAgents (Cowell)
• Interaction with a computer should be as easy as interacting withother people, taking advantage of the multimodal nature ofhuman communication
• Focus revolved around behaviors that portray a credible fac-ade, thereby helping embodied conversational agents (ECAs) to form a successful cooperative dyad with users
Anthropomorphic Embodied ConversationalAgents (Cowell)
Anthropomorphic Embodied ConversationalAgents (Cowell): design suggestions
Non verbalbehavior
Anthropomorphic Embodied ConversationalAgents (Cowell): design suggestions
Anthropomorphic Embodied ConversationalAgents (Cowell): applications
The mobile device landscape