Stochastic Pooling Networks: On the interaction between redundancy, noise and lossy compression in biological neurons. Mark D. McDonnell University of.

Stochastic Pooling Networks:On the interaction between redundancy,noise and lossy compression in biological neurons.

Mark D. McDonnellUniversity of South Australia

Pierre-Olivier AmblardCNRS, Grenoble, France

Nigel G. StocksUniversity of Warwick, UK

6 July 2008

What is a Stochastic Pooling Network?

What do SPNs Model?

Emergent properties of SPNs

This talk is about understanding how biological neurons efficiently compress stimuli during coding.

Real signals are analog but nearly everything in modern electronics is digital. Why is this?

X

XCEO

R1

RN

R2

N

ii RR

1

x

x

xN

1

2

N

1 2

What mechanisms do biological sensory systems use to compress information during transduction?

There is lossy, and lossless compression… and maybe something in between: “loss-least!”

Information theory underpins fast and accurate communication. But it is also about selecting what information is worth storing and communicating.

A “Recipe” for Stochastic Pooling Networks

Ingredients• 1 information source• N > 1 independent sensors (compressive types)• N > 1 random noise sources

PreparationStep 1: For each sensor, measure the information

source mixed with one of the noise sources, and then compresses its measurement.

Step 2: Ensure the whole network produces a single measurement by pooling each sensor’s.

SPNs can model many sensor network or source coding scenarios

• Digital Beamforming Arrays– Sonar, radar, MIMO communications

• Digital signal processing– Noise reduction via coherent averaging after digitization in

ADCs.• Distributed/Decentralized sensor networks

– CEO problem– Multiaccess Communication Channels– Power constrained wireless sensor networks

• Biological neurons– Representation of analog stimuli by rate coding action

potentials– Quantal release of chemical neurotransmitters at synaptic

junctions– Maybe some ion channels?

SPNs can model many sensor network or source coding scenarios

• Reliability schemes in nano-electronics– Averaging and redundancy to overcome

parameter variations and noise [Ferran Martorell, Spain]

• Social networks– Subjective voting on a continuous variable

• Quantum optical communication using polarization detection of single photons?

• Coupled multistable dynamic systems?• Reconfigurable chaotic logic gates?

The neurons that code sounds immediately after transduction can be modelled as an SPN

Cochlea Inner Hair Cell

Auditory NerveSound

Ear

Information

The neurons that code sounds immediately after transduction can be modelled as an SPN

Innerhair cell

Basilarmembrane

Brain

Outerhair cell

Slide courtesy of Prof Tony Burkett, Uni. Melbourne

Pooling must occur “naturally,” without external intervention, e.g. adding, or superposition.

We will not have a pooling network otherwise!

We assume combining of the ingredients is left to physics: POOLING

g1 (.)

g2 (.)

gN(.)

1

2

N

x

y1 {0,1,..,M}

y2 {0,1,..,M}

yN {0,1,..,M}

P y = h(y1, ..,yN)

|y| << M

This is a surprising emergent property of SPNs

Pooling loses no (or negligible) information!

Assume the information source, x, is random.The mutual information, I(x,y) loosely measures how well, on average, the SPN output, y, provides a good estimate for x.

McDonnell, Stocks et. al., Fluct. Noise Lett. 5, L457-L468, 2005.

McDonnell, “Applying stochastic signal quantization theory to the robust digitization of noisy analog signals", Book chapter in Springer Verlag Complexity Series, In Press, 2008.

SPNs digitize (quantize) their input

McDonnell an Abbott, Proc. SPIE, 2006.

SPNs digitize (quantize) their input

SPNs reduce noise via coherent averaging… but not in a linear way!

• For small noise, performance is limited by compression: I(x,y) < log(1+MN)

• For large noise, performance is limited by “averaging”: I(x,y) < 0.5log(1+N SNR)

N=1,M=511

N=511,M=1

0.5log2(1+NSNR)

There are a number of differences in the final SNR achieved by linear analog averaging vs SPNs

Output SNR Analog SPN

Dependency on source distribution

no yes

Decoding

dependency

N/A yes

Scaling with input SNR

linear -linear for small SNRs,

-nonlinear for large SNRs

Scaling with N

(no. of times averaged)

proportional - Approx. proportional for small SNRs

-no effect for large SNRs

Scaling with M (bits) N/A -increases for small SNRs

-no effect for large SNRs

McDonnell, “Signal Estimation Via Averaging of Coarsely Quantised Signals,” Proc IEEE Information, Decision and Control, Adelaide, Australia, pp 100-105, 2005.

Probability of Error: binary detection.

[Zozor, Amblard and Duchene, Fluct. Noise Lett. 7, L39-L60, 2007]

Simulation of cochlear implant coding.

[Stocks et. al., Proc SPIE, 2007.]

Multiplicative Noise at the input to network nodes.[Nikitin, Stocks and Morse, Phys. Rev. E, 2007]

SSR measured by mutual information foradditive random noise.[Stocks, Phys. Rev. Lett., 2000]

Suprathreshold Stochastic Resonance in SPNs

*Also studies by others, e.g. Rousseau & Chapeau-Blondeau (2003), Hoch et al (2003), Martorell (2005)…

McDonnell, Stocks et. al., “Optimal information transmission in nonlinear arrays through suprathreshold stochastic resonance,” Phys. Lett. A 352, pp. 183-189, 2006.

Optimizing the nodes: more noise means more nodes that are identical

We have observed similar effects for optimized networks of Poisson neuron models, and binary detection networks.

There are many other surprising emergent properties

• Very noisy SPNs behave like analogue Gaussian channels [McDonnell, IEEE Aus. Comm Theory Wkshp, 2008]

• Very large SPNs behave like multiplicative noise channels [McDonnell and Stocks, Proc. SPIE. 2007]

• Optimal reconstruction depends only on the noise distribution and the number of sensors [McDonnell, Stocks and Abbott, Phys. Rev. E 75, Art. No. 061105.]

• Optimizing the noise distribution is like optimizing a neuron’s stimulus-response curve.

• Negative correlation provides improved MI.• The optimal stimulus is actually discrete!

Unsolved problems on SPNs in biology

• Do biological senses really use SPN principles?

• What mechanisms does the brain use to compress/reduce information

• Will the controlled use of random noise in cochlear implants improve the hearing of patients?

If healthy auditory neurons act like SPNs then bionic ears should stimulate them randomly!

Electrode arrayElectrode array

11stst turn turnofof

inner earinner earAuditoryAuditory

nervenervefibresfibres

Image courtesy of Cochlear Ltd, 2008

If healthy auditory neurons act like SPNs then bionic ears should stimulate them randomly!

© Australasian Science, 2008

“Stochastic beamforming coding strategy”,Morse, Holmes, Shulgin, Nikitin and Stocks, 2007

There are many unsolvedtheoretical problems on SPNs

• Can the clustering of nodes be predicted mathematically?

• What further complexities can be added to SPNs without changing the basic ingredients?

Redundancy allows noise reduction and simplicity

Lossy compression is required for efficiency

Random noise improvessub-optimal compression

Questions?

In summary, stochastic pooling networks are a versatile and surprising concept for achieving the twin goals of accuracy and efficiency.