Inaugural Meeting 7 th April 2008 Adaptive Systems Research Group Prof. Chrystopher L. Nehaniv Dr. Joe Saunders Prof. Kerstin Dautenhahn
Inaugural Meeting
7th April 2008Adaptive Systems Research Group
Prof. Chrystopher L. NehanivDr. Joe Saunders
Prof. Kerstin Dautenhahn
UH - Where are we?
Human-Robot Interaction Research @ UH
• Current European FP6 Robotics Projects:– Cogniron (IP): Cognitive
Robot Companion– Robotcub (IP):
Developmental Robotics (iCub)
– IROMEC (STREP): Robotic Toys for Children
• New FP7 projects: – LIREC (IP)– ITalk (IP)
• Networks: Euron, euCognition
• Researchers:– Academics:
• Prof. Kerstin Dautenhahn• Prof. Chrystopher L. Nehaniv• Dr. Rene te Boekhorst• Dr. David Lee
– Postdocs:• Dr. Ben Robins• Dr. Ester Ferrari• Dr. Joe Saunders• Dr. Hatice Kose-Bagci• Dr. Aris Alissandrakis• Dr. Khenglee Koay• Dr. Nuno Otero
– Research Assistants/PhD students• Michael L. Walters• Assif Mirza• Qiming Shen• Dag Syrdal• Josh Wainer
UH - Related EU Projects
RobotCub – Social Interaction, Developmental Robotics (FP6 IP, September 2004 – August 2009)
Cogniron – Robot Social Spaces, Proxemics, Learning and Development, Imitation
(FP6 IP, Jan 2004 – Feb 2008)
LIREC - LIving with Robots and intEractive Companions (FP 7 IP, March 2008 – June 2012)
UH - The Team
Principal Scientists
Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn
Researchers
Dr. Joe Saunders
+
* Own Effort
Research Assistant/
FellowNew Hire
Dr. Caroline Lyon*
Hello
Frank FörsterPhD Student*
UH - Approach to ITALK
A robot will use what it learns individually and from others sociallythrough grounded sensorimotor interaction
to bootstrap the acquisition of language via aspects of ideomotor theory,
and will use its language abilities in turn to drive its learning of social and manipulative abilities.
This creates a positive feedback cycle between using language and developing other cognitive abilities.
Like a child learning by imitation of its parents and interacting with the environment around it, the robot will master basic principles of structured
grammar, like negation, by using these abilities in context.
“In the beginning, there was the act.”- Goethe
UH - Approach to ITALK
Major Open Areas for Robot/Agent Development of Language:
1) deixis, gesture and reference;2) predication;3) negation;4) emergence of syntactic categories;5) compositionality
Current Work and Open Problems: A Roadmap for Research into the Emergence of Language of Communication. In C. Lyon, C.L. Nehaniv, A. Cangelosi (editors), Emergence of Communication and Language, pages 1-27, Springer-Verlag, 2007
UH - Viewpoint
Extending C.S. Peirce, latter L. Wittgenstein, & R. Garrett Millikan
To understand (and create!) language use capacity bottom up:(proto-)representations need NOT separate “what-to-do” from “what is-the-case”
(pushmi-pullyu reps.)activity of manipulating and communicating socially in a context of embodied action is
necessarymajor purpose of language is to manipulate the environment, including the social
environment, via communication(other purposes: to bootstrap learning, to regulate interaction, encoding
temporally/spatially remote experience stories – cf: Ho et al. 2008,Mirza et al., 2008)to understand the grammar of a word/linguistic construction look how it is usedmeaning is created in an embodied process of semiosis with interpretant, signal (sign),
signified.
Outline and UH Role
Five Major Work Packages
Action DevelopmentConceptualisationSocial Interaction – Work Package LeaderLanguage EmergenceIntegration and Bootstrapping of Cognition
Robotic Experiments and Demonstrations
Based on iCub robotSimulation (where necessary)Prior to iCub on Kaspar
Outline and UH Role
• Development of artificial embodied agents that are able to acquire complex behavioural, cognitive and linguistic skills through individual and social learning.
Specifically•To learn to handle and manipulate objects and tools autonomously
•To cooperate and communicate with other robots and humans
•To adapt their abilities to changing internal, environmental and social conditions
Aim
UH will build upon our social learning architecture - ROSSUM (RObot Self-Imitation and Scaffolding Utility Mechanism)
Saunders J, Nehaniv C.L, Dautenhahn K. and Alissandrakis A.,(2007), Self-imitation and Environmental Scaffolding for Robot Teaching, International Journal of Advanced Robotics Systems, Vol.4, Issue 1, pp.109- 124
Scaffolded Skills <-> Scaffolded Language
Outline and UH Role
AbilitiesAgents learn to handle objects
UH:
Investigate, by extending existing studies on Ideomotor theory applied to physical robots, how to develop more complex compositional and hierarchical actions via scaffolded learning.
Outline and UH Role
AbilitiesAgents learn to handle objectsAgents create and use embodied concepts
UH:
Our current studies on robot teaching and interaction ensure that the robots experiences are grounded in its own internal and external perceptual states. The crucial aspects of these states are selected using information theoretic approaches.
AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skills
UH:
Using our existing social learning and teaching robotic architecture (called ROSSUM) which is now operating on our humanoid robot (called Kaspar) we intend to study relationships between deixis, gesture and reference. Exploit concept of shared meaning arising through similar gestures and also study ideas of ‘negation’.
Outline and UH Role
AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skillsAgents develop linguistic abilities
UH:Investigate relationships between actions and objects via hierachical scaffolding
Outline and UH Role
AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skillsAgents develop linguistic abilitiesAgents transfer knowledge between cognitive domains
UH:
Study the HRI aspects of verbal or gestural communication.
Outline and UH Role
UH Existing Work on ROSSUM and its Relation to Language
Teaching via Assisted/self
ImitationImitation
CorrespondenceProblem
Affordances andEfffectivities
EmbodiedGrounded
Sensory-motorConcepts
Steps toLanguage Acquisition
How can a child discover affordances and effectivities for action?:
• by trial-and-error – may be slow• by observational imitation – but correspondence problem exists• by assisted/self-imitationCaregivers invite infants to imitate – by putting infants on both sides of the correspondence problem
Scaffolded Teaching via Assisted Imitation
Assisted/Self-Imitation
• Putting through of infants movements• Acting in tandem• Linking effectivities of the infants body with affordances for action• Scaffolding taught competencies
(* Affordances, Effectivities and Assisted Imitation - Patricia Zukow-Goldring, Michael Arbib (2007).)
UH Existing Work on ROSSUM and its Relation to Language
Correspondence Problem and Difficulties in Observational Learning
Matching Movements?
Matching Goals/Effects/States?
Matching Combinations of Movements?
What does matching mean?
The Correspondence Problem*
*C.L.Nehaniv & K.Dautenhahn (2002) – The Correspondence Problem
UH Existing Work on ROSSUM and its Relation to Language
The Correspondence Problem in Imitation
Relative Position
Demonstrated Effect Corresponding Effects
Relative Displacement
Absolute Position
(also see Aris Alissandrakis, Chrystopher L. Nehaniv, Kerstin Dautenhahn and Joe Saunders (2005))
UH Existing Work on ROSSUM and its Relation to Language
UH Existing Work - Robots Learning Socially
using ROSSUM
Teaching via Assisted/self
Imitation
Inverse andForward models
ExtendedIdeomotor
Theory
Imitation
CorrespondenceProblem
Affordances andEfffectivities
ReafferencePrinciple
EmbodiedGrounded
Sensory-motorConcepts
Steps toLanguage Acquisition
UH Existing Work on ROSSUM and its Relation to Language
From Analyse der Empfindungen – Ernst Mach - 1922
Body is visually perceived in exactly the same way as the environment – representation
is the same
Distinction between body and environment must be
based on a non-visual criteria
Distinction achieved by voluntary control
Actions are considered to be creations of the will
UH Existing Work – Ideomotor Theory
Ideomotor Theory : Lotze (1852), James (1890) and Műnsterberg (1888)
How can voluntary behavior or ‘will’ be explained?
All actions are held as ‘images’ of the sensory feedback they produce.
Sensory stimulations result from internal feedback they produceplus
external feedback from the effects on the environment
Actions are generated based on imagining their sensory consequences
UH Existing Work – Ideomotor Theory
Why is this important for a robotic learning system?
One powerful way of learning is through imitation
Extend the Ideomotor Idea to allow imitation – Extended Ideomotor Theory*
If an action can be invoked from thinking about it…..
then
the same action could be invoked by seeing it performed by someone else
*An Ideomotor Approach to Imitation - Wolfgang Prinz (2005)
Extended Ideomotor theory thus supports the idea of learning through experience and
learning through observation
UH Existing Work – Ideomotor Theory
UH New Work – extending ROSSUM
Teaching via Assisted/self
Imitation
Inverse andForward models
ExtendedIdeomotor
Theory
Imitation
CorrespondenceProblem
Affordances andEfffectivities
ReafferencePrinciple
EmbodiedGrounded
Sensory-motorConcepts
Learning viaObservational
Imitation
“Putting oneself In the shoes of
Another”TOM?
Learning viaAssociation,
Inference andReferential intent
Adultto
Child?(Teaching)
Childto
Adult?(Learning) Steps to
Language Acquisition
Perceived State Action State Resulting From Action Execution
Action State Resulting From Action Execution
Action State Resulting From Action Execution
Action State Resulting From Action Execution
.
.
.
ROSSUM Behavioural Component / Memory Model
‘Ideas of Movement’From Previous Learning Effects of Movement
Perceived State
Perceived State
Perceived State
Acts as an Inverse Model Acts as a Forward Model
Similarity
Matching
Current State
Similarity
Matching
Model State
Goal-directed Task
Sequence
Primitive
Behaviour
Primitive Primitive Primitive
Goal-directed Task
Primitive Primitive
Example of a Trained Hierarchy
State
Primitive
Behaviour
Using task scaffolding to build competencies
UH The First Year – Experimental and Theoretical
Research Questions What is the relationship between deixis, gesture and reference ----------- D3.1 (M24)
- Related to hypothesis on the emergence of language based on mirror neurons- Shared meaning arises through similar affordant gestures- the human tutor reveals to the robot via deixis, gesture and reference relevant effectivities and affordances
How can negation arise though the mechanisms of communicative social interaction?---D3.2 (M48)
-Experiments to be carried out where facial and manual gestures used in social interaction to indicate negation- This would involve inhibition signals operating on existing scaffolded behaviours
UH The First Year – Technical Considerations for ROSSUM
Implement Forward ModelForward model side of ROSSUM to be implemented (relates to D1.1)
Gaze Following and Observational ImitationImplement direct imitation via human body tracking either as combination of TOF depth sensor/magnet field trackers/ eye gaze tracker (as Yu/Ballard, S. Knoop)
Object DetectionExtend capabilities for enhanced object recognition by robot using colour segmentation, shape, texture (as Yu/Ballard, Deb Roy).
Word/Phoneme Detection?Study possibility of word/phoneme recognition (as per Deb Roy)
Assisted or Self ImitationExtend current model to further process dynamic movements
Thank you
Adaptive Systems Research Group