Tagged network (colored clique network) COGNITIVE 2015 by Stephen Larroque

Using tags toImprove Diversity of

Sparse Associative Memories

Stephen Larroque

with Ehsan Sedgh Gooya, Vincent Gripon, Dominique Pastor

An alternative to the size-diversity trade-off forneuromorphic devices

23rd March 2015COGNITIVE 2015

Long-term storage?

2

Long-term storage?

• Addressed memory models in computers :

3

Long-term storage?


4

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey

Long-term storage?


5

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey

Partial/noised query : 0000010_ ?

Long-term storage?


6

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey


Corruption ->

Long-term storage?


7

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey


Corruption ->

Long-term storage?


• Connectionist solutions :– Associative memories (Hebbian rule)– Deep neural networks (McCulloch-Pitts neurons)8

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey


Corruption ->

Associative memories

9


• Hopfield (1982)

10

(Courtesy ofR. Rojas)


• Hopfield (1982)

• Willshaw (non-holographic associative memory, 1969)

11



• Hopfield (1982)

• Willshaw (non-holographic associative memory, 1969)

• Cliques network (2011)

12


Cliques network, storing

• Hebbian rule

• Message ≡ clique

13

Cliques network, retrieval

14


15


16

Iterative !

Cliques network, storing more

17

000000000

000000000

000000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix


18

100000000

100000000

100000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix


19

100000000

101000000

101000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix


20

100000100

101000001

101000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix


21

100000101

101000101

101000100

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Size-diversity trade-off

22

100000101

101000101

101000100

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Problem : a non-learnt message is retrievable !

Fake memory :(network’s size toosmall for diversity)

Let’s color this graph!

23

200000304

102000403

103000400

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix


24

200000304

102000403

103000400

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix


26

200000304

102000403

103000400

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrixFake memory avoided by tag disambiguation

Performance

27

• Diversity/density for various tags limits against the error rate :

χ = 16, c = 8, L = 64, erasure rate = 0,5 (half of the query is erased)

Conclusion and future works• Tags : a viable alternative to the diversity-size trade-

off for fixed-size networks(e.g., neuromorphic devices)

• A tentative explanation of synapses heterogeneity : brain may use an affinity system to co-sustain synapses with similar parameters.« Neurons that fire together, wire together, and with a strong affinity. »

• Next :– Noisy scenario (unreliable tags)– pertinence of memories, variable resiliency : all items may

not need to be stored with equal resiliency. Try to refresh tags on access ? (« spacing effect » ?) 28

– Nonholographic associative memory, D. J. Willshaw, O. P. Buneman, and H. C. Longuet-Higgins, Nature, vol. 222(5197), June 1969, pp. 960–962

– Neural networks and physical systems with emergent collective computational abilities, J. J. Hopfield, Proceedings of the national academy of sciences, vol. 79, no. 8, 1982, pp. 2554–2558.

– Sparse neural networks with large learning diversity, V. Gripon and C. Berrou, Neural Networks, IEEE Transactions on, vol. 22, no. 7, 2011, pp. 1087–1096.

Thanks and a few references slideshare.net/LRQ3000

Vincent GRIPON Dominique PASTOR

ERC grant agreement n° 290901

EhsanSEDGH GOOYA

Thank you! slideshare.net/LRQ3000

Bonus Slides

Neural constraints• Energetic parsimony

• Material resources parsimony

• Noise robustness

• Simple processing rules (analog?)

=> Feed-forwoard ANNs : synaptic weights as floats are too sensitive(learning = adjust weights)and capacity ~ sub-linear in number of nodes.

35

(Aiello & Wheeler, 1995)

Cliques network

• Brain = information encoder

• Fully graphical model

• Associative, recurrent network with binary weights, integer output, capacity ~ n² :– Network : set of clusters– Cluster : set of fanals– Fanal : graph nodes

(microcolumn?)

36

(Gripon-Berrou Neural Network, 2011)

Capacity vs Diversity

• Diversity = number of messages possibly leant/stored

• Capacity = whole learnt information

• « From a cognitive point of view, it is better to learn (and possibly combine) 1000 messages of 10 characters than to learn 10 messages of 1000 characters »

37

(C. Berrou & V. Gripon, 2010)

Tagged network: colors as layers

38

Thrifty code

39

Disambiguation by tags

40

Disambiguation by tags - 2

41

Disambiguation by tags - 3

42

Analysis of error rate

43

• New error type : lost unit error


44



45



46



47


Lost forRed clique !


48


=> When a clique lose one node, because all edges have been overwritten by other tags of newer cliques, it becomes unretrievable.

=> only dependent on storage process !

•

Lost forRed clique !

Theoretical lost unit error

49

• With :

• Approximation : messages are i.i.d. variables

M = total messages ; c = clique orderχ = total graph parts ; L = units per part

Proba to overwrite one edgewhen storing one new clique

(1 chance over network’s size)

Clique size(repeat for all edges of each new message)

(need to overwrite all

edges of one unit to lose it)

What about other error types?

50

• Real error rate (red) against a composition of errors from each possible type (green) : lost unit error is a good predictor

What about other error types?

51

• Theoretical lost unit error (black) against real (blue)

Efficiency?

52

• Efficiency = B (amount of info stored) Q (material used)

• Clique network :

• Tagged network :

=> Tagged network use more material, proportionally to the number of tags !

Performance - 2

53

• Performance when accounting the efficiency :

THE END

Tagged network (colored clique network) COGNITIVE 2015 by Stephen Larroque

Science