Learning Prepositions for Spatial Relationships in BOLT Soar Workshop 2012 James Kirk, John Laird 6/21/2012 1.

1

Learning Prepositions for Spatial Relationships in BOLT

Soar Workshop 2012James Kirk, John Laird

6/21/2012

2

Outline

• Prepositions in BOLT• Compositional Spatial Relationships• Learning Process in Architecture with SVS• Projection• Conclusions

3

Prepositions/Spatial Relationships in BOLT

• “What is to the right of the pantry?”• “Put the triangle in the stove.”• “Pick up the red object to the right of the pantry.”

4

Spatial Relationships as Compositions

y-alignedz-alignedx-greater than

Z

Y

X

“right of”

“Intersecting”y-alignedz-alignedx-aligned

“Above”z-greater than

Other potential compositions:

“In front of”y-less thanz-alignedx-aligned

5

Learn new prepositions that map to spatial relations “The red triangle is right of the blue sphere”

6Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory

Preposition – Spatial RelMapping

Verb – Operator Mapping

Noun/Adjective – Perceptual Symbol

Mapping

Semantic Memory

Primitive Verb – Operator Mapping

Episodic Memory

Agent’s Experiences

Spatial Visual System

Spatial PrimitivesAction

Phase 1Perception

- Receive object information from

input link

- Create internal representations

Working Memory

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Human: “The red triangle is right-of the blue sphere” Word – Category

Mapping

7Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 2Message Parsing

- Parse the given sentence

- Create usable message

structures

“The red triangle is right-of the blue sphere”

Working Memory

LGSoar

Obj1- bl3- sp7

Obj2-re23- tr17

Obj3- bl3- sq8

relation-message ^word right-of ^o1 red, triangle ^o2 blue, sphere


Mapping

8Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 3Interaction

Stack

- Designate message purpose

as preposition learning

Working Memory

LGSoar


Interaction

Prep. LearningObj1- bl3- sp7

Obj2-re23tr17

Obj3- bl3- sq8


Mapping

9Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

Working Memory

Interaction


Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learning

Prep Learning

right-ofred, triangle

blue, sphere


Mapping

10

Word – Category Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- Lookup the category of red

from smem

Working Memory

Noun Learning

red color

Prep Learning

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learningright-of

red, triangle

blue, sphere

Human: “The red triangle is right-of the blue sphere”

Indexing

11


Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- Lookup the symbol for red + color from smem

Working Memory red color

Noun/Adjective – Perceptual Symbol Map

re23Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8


red, triangle

blue, sphere

re23, triangle


Indexing



12


Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- Update the descriptions with

perceptual symbols

Working Memory

Noun/Adjective – Perceptual Symbol Map

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8


red, triangle

blue, sphere

re23, trianglere23, tr17

bl3, spherebl3, sp7


Indexing


13


Mapping

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Prep Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- Look for the objects matching the description

Working MemoryObj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8


re23, tr17

bl3, sp7

Obj2

Obj1


Indexing


14

Obj2

Obj1

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- SVS updates model of objects in the world from the perception input.



Obj2-re23-tr17

Obj3- bl3- sq8


Obj1Obj2Obj3Spatial Primitives

Prep Learning


15

Obj2

Obj1

Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- SVS is queried for primitive

spatial relationships

between Obj1 and Obj2


Prep Learning


Obj2-re23-tr17

Obj3- bl3- sq8


Obj1

Obj2

Obj3

Obj2

Obj1?

Spatial Primitives


16Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- The results of the query are

stored in working memory.


Prep Learning

y -alignedz-aligned+x

Working Memoryright-of

Obj1- bl3- sp7

Obj2-re23-tr17

Obj3- bl3- sq8

Prep. Learning


17Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory




Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- List of spatial primitives are

abstracted into general spatial

relationship and mapped to “right-

of.”


Prep Learning


Obj2-re23-tr17

Obj3- bl3- sq8

SR11

y-aligned z-aligned +xright-of

Prep. Learning


18



Perception

LGSoar

Interaction

Indexing

Verb Learning

Noun Learning

Action Knowledge

Procedural Memory



Mapping

Semantic Memory


Episodic Memory




Phase 4Preposition

Learning

- Learned spatial relationship mapping to “right-of” is

stored in semantic memory.


Prep Learning


Obj2-re23-tr17

Obj3- bl3- sq8

SR11

y-aligned z-aligned +xright-of

Prep. Learning


19

Refining learned knowledge

Is this triangle also to the right of the square?

Disjunctive primitive relationships in training are removed.

20

Learning is interactive

• Learning can be initiated by instructor or agent

“What is behind the blue sphere?”- “I don’t know the preposition behind. Please teach me with

examples.”“The green square is behind the red triangle.” “Finished teaching.”- “The blue square is behind the blue sphere.”

21

Spatial Projection

“Put the object to the right of the pantry.”

Z

Y

X

“right of”

22

Current Uses

• Describing the scene• Specifying objects in the scene• Tracking objects and relationships– “The goal is that the red triangle is inside the

pantry”• Projecting relationships back to a point in 3D

space to perform actions

23

Evaluation

Nuggets• Learns very fast• Works well for simple relationships and projection/tracking

Coals• Cannot learn complex relationships with many objects• Cannot learn certain types of relationships such as diagonal• Not robust to mistakes in training examples

24

Future Work

• Complex multi-object relationships– For example in a line or a circle

• More complex spatial relationships– Using distance/contacting etc.– Pattern matching (diagonal)– Will require more training examples

• Projection with multiple relationships– “put this inside the pantry and to the right of the blue object”

25

Questions?

Learning Prepositions for Spatial Relationships in BOLT Soar Workshop 2012 James Kirk, John Laird 6/21/2012 1.

Documents

red triangle

o1 red

o2 blue

red object

tr17obj3 bl3

bl3 sq8human

sphereobj1 bl3

learningobj1 bl3