MACHINE VISION GROUP Computer Vision in Human-Computer Interaction Matti Pietikäinen Machine Vision Group Department of Electrical and Information Engineering and Infotech Oulu University of Oulu, Finland http://www.ee.oulu.fi/mvg Invited talk in 2010 Autumn Seminar and Meeting of Pattern Recognition Society of Finland, M/S Baltic Princess, 26.11.2010 MACHINE VISION GROUP Contents • Introduction: Example applications • Case project: Affective human-robot interaction • Research on key computer vision methods for HCI in Oulu • Case project: Vision-based mobile HCI • Some challenges for future research
17
Embed
Computer Vision in Human-Computer Interaction · 2. Recognizing emotions of a person and providing an affective robot’s response on the basis of person’s emotions 3. HRI demonstration
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MACHINE VISION GROUP
Computer Vision in Human-Computer Interaction
Matti Pietikäinen
Machine Vision Group
Department of Electrical and Information Engineering and Infotech Oulu
University of Oulu, Finland
http://www.ee.oulu.fi/mvg
Invited talk in 2010 Autumn Seminar and Meeting of Pattern Recognition Society of
Finland, M/S Baltic Princess, 26.11.2010
MACHINE VISION GROUP
Contents
• Introduction: Example applications
• Case project: Affective human-robot interaction
• Research on key computer vision methods for HCI in Oulu
• Case project: Vision-based mobile HCI
• Some challenges for future research
MACHINE VISION GROUP
Example application: Microsoft’s Kinect
- Controller free interface for Xbox 360
- Range and color cameras
- Brings 3D imaging to mass markets
MACHINE VISION GROUP
Example application: Human-robot interaction
- Communication must be easy and natural
MACHINE VISION GROUP
Example application: Smart environments
MACHINE VISION GROUP
Example application: Vision-based mobile HCI
– User authentication
– Vision-based interaction
MACHINE VISION GROUP
Case project: Affective human-robot interaction
• Development of natural, affective human-computer interfaces (HCI) is of great interest in building future ubiquitous computing systems
• In 15 years servicing robots (social robots) will become a part of our everyday lives
• We should be able to inteact with robots in a natural way, like in human-human interaction
• Computer vision will play a key role building affective HCI and HRI (human-robot interaction) systems of the future
• A joint research on Affective HRI is in progress by Machine Vision Group Intelligent Systems groups of the University of Oulu
• Supported by the Academy of Finland, European Regional Development Fund and University of Oulu
MACHINE VISION GROUP
Objectives
• To develop leading edge approaches for affective HRI
• Both ”face to face” interaction and remote interaction (using mobile
phones or PDAs equipped with cameras) will be considered
• The robot should be able to
- detect and identify the user
- recognize user’s emotions
- communicate easily by understaning speech and gestures
- provide a natural response based on its observations
- learn its behavior and tasks it is supposed to do
- utilize its motion in different levels of interaction (robot
embodiment)
MACHINE VISION GROUP
Work packages
WP1: Machine vision for human-robot interaction
WP2: Human-robot interaction
WP3: Experimental validation of affective HRI
MACHINE VISION GROUP
WP1: Machine vision for human-robot interactionPerson identification
• Face recognition, gender recognition, age estimation
• Speaker identification
• Gait recognition, clothing (recognition from a distance)
Recognition of emotions
• Facial expression recognition
• Speech, body movements
Methods for human-robot communication
• Hand gestures and speech (”face to face” communication)
• Body movements (e.g. waving hands) (communication from a distance)
• Using camera motion (communication with a mobile device)
• A talking avatar on robot’s flat panel display
On-line machine learning and adaptation
MACHINE VISION GROUP
WP2: Human-robot interaction
Embodiment and learning in HRI:
• Our goal is to investigate how a robot could learn to interact with a person in a socially interesting way
• A possible approach is illustrated in the following figure
MACHINE VISION GROUP
An example target system
The desired state in this example could be modified as a function of time as the the person and the robot get acquainted, e.g. the robot could learn to change its behavior in a pace of interaction.
MACHINE VISION GROUP
WP3: Experimental validation of affective HRI
Different types of tasks of affective HRI are experimentally validated and
demonstrated, e.g.
1. Recognizing a person and providing a personalized robot’s response
2. Recognizing emotions of a person and providing an affective robot’s response on the basis of person’s emotions
3. HRI demonstration where the robot utilizes its learned behaviors in order to initialize, and maintain natural interaction with a person.
4. System-level demonstrations, e.g. robot guide for the visitors in a smartenvironment
MACHINE VISION GROUP
Research of key computer vision methods for HCI in Oulu
1. Face recognition
2. Facial expression recognition
3. Visual speech recognition
4. Video synthesis for face animation
5. Object detection and recognition
6. Tracking of moving objects
7. Object re-identification in camera networks
8. Recognition of actions and gait
MACHINE VISION GROUP
Face analysis: research challenges
Face is a dynamic and non-rigid object which is difficult to model because of
the large variability in its appearance.
The appearance of the face varies due to changes in pose, facial expression,
illumination, aging, occlusion, presence of glasses etc.
MACHINE VISION GROUP
Face analysis using local binary patterns
• Face recognition is one of the major challenges in computer vision
• We proposed (ECCV 2004, PAMI 2006) a face descriptor based on LBP’s
• Our method has been adopted by many leading scientists
• Excellent results in face recognition and authentication, face detection,
• The project will address some of the issues of direct
(spoofing) attacks to trusted biometric systems. This is an issue
that needs to be addressed urgently because it has recently
been shown that conventional biometric techniques, such as
fingerprints and face, are vulnerable to direct (spoof) attacks.
• Coordinated by IDIAP, Switzerland
• We will focus on face and gait recognition
MACHINE VISION GROUP
Dynamic texture descriptors for motion analysis
• We proposed (PAMI 2007) simple spatiotemporal LBP descriptors for
dynamic texture recognition outperforming the state-of-the-art
• They have been applied to facial expression regonition (PAMI 2007),
face and gender recognition from video sequences (Pattern Recogn.
2009), visual speech recognition (IEEE T Multimedia 2009), and
recognition of actions and gait (BMVC 2008, ICB 2009, MVA 2010) -
with excellent results
• Our approach has potential for significant contributions in many
applications and fundamental problems of motion and activity analysis
MACHINE VISION GROUP
Demo for facial expression recognition
Future challenge:
How to recognize spontaneous expressions insteadof acted ones?
MACHINE VISION GROUP
Demo for visual speech recognition
MACHINE VISION GROUP
Video synthesis for face animation
• Dynamic texture synthesis
(ICIP 2009) • Video-realistic speech animation
(ICVGIP 2010)
MACHINE VISION GROUP
Object detection and recognition
MACHINE VISION GROUP
Tracking of moving objects
MACHINE VISION GROUP
Object re-identification in camera networks
• Person re-identification using global color context (VS 2010)
MACHINE VISION GROUP
Recognition of actions and gait
B F S B F S
MACHINE VISION GROUP
Dynamic textures for action recognition
• Formation of the feature histogram for an xyt
volume of short duration
• HMM is used for sequential modeling
• State-of-the-art results for Weizmann and KTH
databases (BMVC 2008, MVA 2010)
Feature histogram of a bounding volume
MACHINE VISION GROUP
Case project: Vision-based mobile HCI
• Hand motion can be used for controlling mobile devices
• Camera becomes a motion sensor
• A fast and efficient method for
image motion estimation has
been proposed (CVIU 2007)
– Uncertainty analysis is performed
for each motion feature
• Symbian implementation
Local motion analysis: (a) 16 subregions and selected feature blocks, (b) feature motion estimates and associated error covariances.
MACHINE VISION GROUP
Interactive Panorama Builder
Motion estimation system calculates shift, rotation and scale in real timeWhen frame is suitable for stitching (high quality), the user receives feedback and instructions
MACHINE VISION GROUP
Scene Panorama Builder
MACHINE VISION GROUP
Automatic Device Activation
The principle: the light is not on if no one is watching
Example case: recognizing context and sequences of actions
MACHINE VISION GROUP
Virtual 3D Display
Face-tracking based 3D rendering
• Face-Tracker obtains 2D face position at 25 fps• Face distance to the screen estimated with face size• Face distance estimated using Motion Estimation Library
MACHINE VISION GROUP
Camera Assisted Multimodal UI
• UI concepts that rely on multiple sensors of modern mobile devices
• Used for recognizing context and sequences of actions
• The key motivation is to hide start-up latencies of the functionalities from the user
MACHINE VISION GROUP
Some future challenges for computer vision in HCI
• Face analysis in natural environments
- changing illumination, varying view and distance, different sensors etc.
• Recognition of spontaneous facial expressions
- instead of acted expressions
• Recognition of natural human actions
- instead of acted ones
• Person tracking, identification and activity recognition in multicamera
networks
• Use of multimodal information
- e.g. emotions from facial expressions, body movements, speech, biosignals