HTM and SDRs for Newbies

NuPIC Bay Area Meetup

Mar, 25 2015

• HTM and SDRs for Newbies – Jeff Fohl ~20 mins• Open Source Community Updates – Matt Taylor ~10 mins• Nupic Demo ~30 mins

– Matt [ Application Demo ]– Subutai [ Research Demo ]– Chandan [ Walnut ]

HTM and SDRsfor

Newbies

Brain

Neocortex

Neocortical Layers

Neocortical Layers

Neuron

Dendrite

Synapses

Neocortical Hierarchy

Hierarchical Temporal Memory:

HTM

Hierarchical Temporal Memory is the model that describes how information is stored and processed in the brain.

• An HTM is a hierarchy of nearly identical regions of neurons.

• Regions learn, infer, and recall time-based sequences.

• Regions are constantly predicting the next inputs.

• Regions detect anomalous input.

Hierarchical

Temporal

MemoryIn order to remember sequences, a region needs a way to store this information. How is this done?

Sparse Distributed Representation:

SDR

SDRs are the fundamental data format of Hierarchical Temporal Memory.

How do we represent an SDR?It can be written as a vector with the following properties:• High dimensionality. It will usually contain at least 2,000

elements.• Binary. Each element in the vector is either “on” or “off”.• Sparse. Out of the total number of bits, only a small percentage

(usually 2%) will ever be active.• Distributed. The “on” bits will be spread across the vector.• Semantic. Each bit in the vector has meaning.

00010000000000100000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000100000000000000000010000000000000000000000000000000000000000000000000000000001000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000001000000000000001000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000000000000000000000000000001000000000100000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000001000000000000000000000000101000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000

This could be written out using 1s and 0s:

Or, perhaps using dots, for a visual representation:

If you are using so many bits, why not use them all? By using only 2% at a time, aren’t you wasting information?

Q:

You would be, if you were processing the information in the way a traditional computer processes information. But, this is different.

A:

Dense Representation

m

o

Sparse Representation

uppercaselowercaseconsonantvowelsofthardopenclosedserifsans serif

m

uppercaselowercaseconsonantvowelsofthardopenclosedserifsans serif

m

SDRs are ultra high capacity

The number of patterns that can be represented by a 2,048 bit SDR with 40 active bits:

1084

(that’s way more than the number of atoms in the observable universe)

Storing SDRs

A 2,000 bit SDR, with 40 active bits, means we only need to store the indices of those 40 – not the entire 2,000 bits.

7,29,47,96,101,134,139,157,180,201,234,245,267,273,291,299,304,356,358,363,389,404,435,444,453,467,481,482,495,505,603,667,689,703,766,781,834,875,909,945

SDR Subsampling

indices60107243575…

10,000 bits

200 active

indices131816862975444144614517486071468179915810,000

bits

10 active

SDR Unions

Thank you

To get more depth on this topic, visit Numenta’s YouTube channel