Recognizing Locations on Objects by Marcus Lewis

Recognizing locations on objects

Marcus Lewis — November 2017 [email protected]

Agenda

• The problem, in context: Detecting locations relative to objects

• Inspiration:Principles of grid cells

• A solution:Recall, Normalize, Vote

Part 1 The problem, in context: Detecting locations relative to objects

Consider the inputs to the brain.

Sensors move.The brain is very good at making sense of sensorimotor sequences.

Sensors are often independent.Familiar objects often cause novel combinations of sensory features.

The body is an array of sensors.With both vision and touch, the brain receives an array of sensory features.

?

Sensorimotor sequences

Model ofobjects

?

Features + Locations relative to object

solvable

Sensorimotor sequences

Model ofobjects

These locations

aren’t relative to the body.

These locations

aren’t relative to the body.

They’re relative to the object

They’re relative to the object

These locations

aren’t relative to the body.

?

Features + Locations relative to object

solvable

Sensorimotor sequences

Model ofobjects

solvable?

Features + Locations relative to body

solvable

This Talk

Sensorimotor sequences

Model ofobjects

Features + Locations relative to object

Given the sensor’s location relative to

the body…

…calculate its location relative

to the object.

Part 2 Inspiration: Principles of grid cells

A rat

An environment

Electrode

Entorhinal cortex

Cell

Electrode

Entorhinal cortex

Cell

Each grid cell fires at multiple equally spaced locations.

Electrode

Cell

Entorhinal cortex

A Room

Firing fields of 16 different cells.

Electrode

Cell 1 Cell 2Cell 3

1 2 3 4

8765

1211109

16151413

1 2 3

8765

1211109

16151413

1 2 3 4

8765

1211109

16151413

1

65

109

151413

1 2 3 4

8765

1 2 3 4

8765

1211109

161514

2 3 4

87

1211

16

1 2 3 4

8765

4 1

5

4

13 Entorhinal cortex

A Room

Room 1

After the rat learns a room, grid cells anchor to the

room consistently.

Room 2

They anchor to other rooms differently.

“Purely Translational Realignment in Grid Cell Firing Patterns Following Nonmetric Context Change”

Marozzi, Ginzberg, Alenda, Jeffery (2015)

A cool grid cell trick: Modules

Module 2Module 1

+ +

Module 3

Entorhinalcortex

= Big space of unique locations

Principles of grid cellsThe cell activity moves between a fixed

set of locations.

These locations are mapped onto the environment.

=+Body

Motion2. Updating it from motion cues

The brain determines the location by…

1. Recalling it from sensory cues +Feature

=

Part 3 A solution: Recall, Normalize, Vote

Sensory 1

Object

Input

Object

Input

Location Sensory 2Location

Column 1 Column 2

Layer 2

Layer 4

Column 1 Column 2

Object

Input

Feature relative to

Object

Object

Input

Feature relative to

Object

Body relative to

Object

Feature relative to

Body

Feature relative to

Body

New:Recall past locations of

sensed features.

New:Vote on the body’s location

relative to the object.

Layer 2

Layer 4

Layer 6A

Sensory 1 Sensory 2

BBABA

BBA

B

A A AB

AA B

AA

BA B

BA

A

A

Object 1

Object 3

Object 2

Object 4

BA B

BA

A

A

Demo:

BBABA

Visualizing the solution

All objects are learned at points in a large “location” space.

ABB

BBABA

All objects are learned at points in a large “location” space.

Visualizing the solution

BA

ABB

BBABA

All objects are learned at points in a large “location” space.

Visualizing the solution

BA

ABB

All objects are learned at points in a large “location” space.

BA B

BA

A

A

Visualizing the solution

BA B

BA

A

A

BA

ABB

All objects are learned at points in a large “location” space.

A

BA

Visualizing the solution

BA B

BA

A

A

BA

ABB

All objects are learned at points in a large “location” space.

AAB

BA

Visualizing the solution

BA B

BA

A

A

BA

ABB

All objects are learned at points in a large “location” space.

AAB

BBA

A

Visualizing the solution

BA

AA

AB

BBB

B

AA

This “location” space now contains maps of multiple objects.

Visualizing the solution

Map ofObject 1

Map ofObject 2

Distance between isn’t meaningful

BA

AA

AB

BBB

B

AA

BBABA

To recognize a location, each sensor votes on the body’s location.

Visualizing the solution

BA

AA

AB

BBB

B

AA

BBABA

To recognize a location, each sensor votes on the body’s location.

Visualizing the solution

BA

AA

AB

BBB

B

AA

BBABA

To recognize a location, each sensor votes on the body’s location.

Visualizing the solution

BA

AA

AB

BBB

B

AA

BBABA

2

To recognize a location, each sensor votes on the body’s location.

Visualizing the solution

BA

AA

AB

BBB

B

AA

BBABA

3 22

2 2

To recognize a location, each sensor votes on the body’s location.

Visualizing the solution

BA

AA

AB

BBB

B

AA

BBABA

3

To recognize a location, each sensor votes on the body’s location.

Visualizing the solution

solvable?

Features + Locations relative to body

solvable

This Talk

Sensorimotor sequences

Model ofobjects

Features + Locations relative to object

“Do I remember any objects that contain this

arrangement of features?”

The holistic system does this:

Summary

Location is probably one of the brain’s main primitives.

Grid cells suggest an approach to location: • Cortex has a “space” of locations that it can process. • Cortex maps objects into this space.

Cortex could recognize these locations in 3 steps: Recall, Normalize, Vote

BA

A

BB

Thank you.