A Hypermap Model for Multiple Sequence Processing Abel Nyamapfene 30 April 2007
Nov 01, 2014
A Hypermap Model for Multiple Sequence Processing
Abel Nyamapfene
30 April 2007
Research Motivation
I am investigating complex sequence processing
and Multiple Sequence Processing using an
Unsupervised Neural Network processing
paradigm based on the Hypermap Model by Kohonen
What is A Sequence?
• A sequence is defined as a finite set of pattern items:S: s1 – s2 - … - sn
where sj : j = 1, …, n is a component of the sequence and the length of the sequence is n.
Examples: In Language Processing
• Speech utterances• Action sequences• Gestural sequences
In Multimedia Processing • Video sequences • speech
Why Unsupervised Processing?
• Distributed sequence processing prone to catastrophic interference
• Requirement of teaching signal inappropriate for unsupervised processing applications
Issues in Complex Sequence Processing
• Definition:A sequence is complex if it contains repetitions of the samesubsequence like C- O - N -F-R- O - N -T, otherwise it is a simple
sequence
• Research Issue:In complex sequences the correct sequence component can only be
retrievedby knowing components prior to the current one.
How can this be done automatically in an unsupervised neural network
framework?
Issues in Multiple Sequence Processing
• Built for Single Sequence Processing Only:
Most existing sequential neural networks have no inbuilt
mechanism to distinguish between multiple sequences.
• Research Issue:
How can a sequential neural network learn multiple sequences
one after the other without catastrophic interference – the
phenomenon whereby the most recently learned ones erase the
previously learnt sequences.
The Hypermap Model (Kohonen, 1991)
Context domain selected using input context
vector
Best match from within the selected
context domain picked using input
pattern vector
Key Features• a self-organising map
• patterns occur in the context of other patterns.
• the most recent elements prior to an element, or some processed form of them, used as context.
Shortcomings• Can not recall a sequence using time-varying context• Can not handle multiple sequences
Barreto-Araujo Extended Hypermap Model(1)
a(t-1), y(t-1)
a(t), y(t)
LateralWeights
M
FeedforwardWeights W
z-1z-1z-1z-1 z--1
contextSensory stimuli
Barreto-Araujo Extended Hypermap Model(1)
Key Features:
• Lateral weights encode the temporal order of the sequences.
• The context weights encode sequence identity
• Sensor weights encode sequence item value
Successes:• Can recall entire sequences in correct temporal order • Can handle multiple sequence processing (Up to a point)
Shortcomings:• Model can only handle sequences with no repeating elements.
Barreto-Araujo Extended Hypermap Model(2)
z-1z-1z-1z-1
Fixed context
Time-varyingcontext
Sensorimotor stimuli
a(t-1), y(t-1)
a(t), y(t)
LateralWeights M
FeedforwardWeights W
z--1
Barreto-Araujo Extended Hypermap Model(2)
Key Features:
• Time-varying context vector act as element ID within a
sequence
• Fixed context vector to act as sequence identity vector
Successes:• Can handle both complex and multiple sequences
Shortcomings:• No mechanism to identify, anticipate and recall sequences using partial sequence data• Contextual processing of sequences rather limited
The Hypermap Model for Multiple Sequence Processing
Have Modified the Barreto-Araujo Model as Follows:
• Incorporated a short term memory mechanism to
dynamically encode the time-varying context of each sequence
item, making it possible to recall a stored sequence from its
constituent subsequences.
• Incorporated inhibitory links to enable competitive queuing
during context dependent recall of sequences.
Temporal Hypermap Neuron
Dj0 Dj1 Dj2 Dj(d-1)
Dj+10 Dj+11
Dj+20
Dj+1(d-2)
Dj+2(d-3)
Dj+d-10
Dj+d-21
(j-1)th Neuron (j+1)th Neuron
Pattern VectorContext Vector
Threshold unit
Delay units
Hebbian Link
Hebbian Link
InhibitoryLinks
The Competitive Queuing Scheme for Context- Based Recall
Have Modified the Barreto-Araujo Model as Follows:
• Incorporated a short term memory mechanism to
dynamically encode the time-varying context of each sequence
item, making it possible to recall a stored sequence from its
constituent subsequences.
• Incorporated inhibitory links to enable competitive queuing
during context dependent recall of sequences.
The Competitive Queuing Scheme for Context- Based Recall
• Context vector applied to the network and Winner Take All
mechanism activates all the target sequence neurons
• Inhibitory links ensure that only the first neuron is free to fire.
• Next sequence neuron fires on deactivation of first neuron
• Neuron activation and deactivation continues until entire
sequence is retrieved
• Scheme first proposed by Estes [17], and used by Rumelhart
and Norman [18] in their model of how skilled typists generate
transposition errors.
The Short Term Mechanism for Sequence Item Identification and Recall
• Pattern input Winning Neuron applies a pulse to its tapped delay line.
• Each tap on a delay line feeds into a threshold logic unit. • The inputs to each threshold logic unit are the output of the tap
position to which it is connected on its neuron’s delay line as well as all the simultaneously active tap positions on later neurons in the sequence.
• Threshold unit activation levels and output activations dependent on tap position and WTA ensures highest level threshold logic unit wins the competition.
• STM mechanism transforms network neurons into subsequence detectors which fire when associated subsequence entered into network, one item at a time.
Experimental Evaluation
Evaluation Criteria
• Sought to evaluate the network’s ability to store and recall– Complex sequences– Handle multiple sequences with high degree of overlap
• Sought to compare performance with other models usingPublicly available benchmark data set
Experimental Evaluation1: Evaluation Data
No Sequence No Sequence
1 Learning and Memory II 7 Time Series Prediction
2 Intelligent Control II 8Neural SystemsHardware
3 Pattern Recognition II 9 Image Processing
4 Hybrid Systems III 10Applications of NeuralNetworks to Power Systems
5Probabilistic Neural Networks and Radial Basis Functions
11 Supervised Learning
6Artificially Intelligent Neural Networks II
Network correctly recalls sequences through Context and when partial sequences applied to the network
Partial Sequence Recalled Sequence
Learning and Learning and Memory I1
Radial Radial Basis Functions
Pro No CHOICE due to conflict between sequences 5 and 10
Proc Processing
Time Time Series Prediction
Series Series Prediction
Intelligent No CHOICE due to conflict between sequences 2 and 6
Neural Networks and Neural Networks and Radial Basis Functions
cog cognition II
Artificially Artificially Intelligent Neural Networks II
Hybrid Hybrid Systems III
Case Study:Two-Word Child Language
“there cookie” instead of “there is a cookie”
“more juice” instead of “Can I have some more juice”
“baby gone” instead of “The baby has disappeared”
Two-Word Model Assumptions
• Ordering of two-word utterances not random but similar to adult speech word order (Gleitman et al, 1995)
• Two-word stage and one-word stage communicative intentions similar: e.g. naming, pointing out, state ownership, comment on actions etc (Brown, 1973)
• Leading us to the following two word stage Modelling Assumptions:– a multimodal speech environment as in one-word stage– consistent word ordering for each two-word utterance
Two-Word Model Simulation
• Model based on a Temporal Hypermap with: – Tri-modal Neurons with weights for word utterances,
perceptual entities, and conceptual relations
• Data: – 25 two-word child utterances from the Bloom’73 corpus
Perceptual Entity Vector
WordVector
Conceptual Relation Vector
Inhibitory Link
Z-1 Z-1
(j-1)th Neuron
jth
Neuron
(j+1)th Neuron
Threshold Logic Unit
Delay Line Element
Temporal Hypermap Segment
Discussion: Two-Word Model
• Temporal Hypermap encodes utterances as two-word sequences
• Utterance can be recalled by inputting a perceptual entity and conceptual relationship
• Network capable of utterance completion – Entering a unique first word leads to generation of entire two word utterance
Simulating Transition from One-Word to Two-Word Speech
From Saying
“cookie”
“more”
“down”
To Saying:
“there cookie”
“more juice”
“sit down”
One-Word to Two-Word Transition Model Assumptions
• From 18th month to 24th month child language undergoes gradual and steady transition from one-word to two-word speech (Ingram,1981; Flavell 1971)
• Transition is gradual, continuous and has no precise start or end point (Tomasell & Kruger, 1992)
• For the transition we assume for each communicative intention:– Transition probability increases with exposure (training)– Transition is non-reversible transition
Gated Multi-net Simulation of One-Word to Two-Word Transition
• Gated Multinet Architecture comprises: – a modified counterpropagation network for one word
utterances – a Temporal Hypermap for two-word sequences– Exposure-dependent inhibitory links from Temporal
Hypermap units to counterpropagation network to manage transition from one-word to two-word output
• Data: – 15 corresponding pairs of one-word and two-word child
utterances from the Bloom’73 corpus
One-Word to Two-Word Model
Static CPnetwork
TemporalHypermap
Inhibitory Link
Wordvector
Perceptual entityvector
Conceptual Relationship
vector
z-1z-1
Output Two-Word Utterances Plotted Against Number of Training Cycles
0.00
2.00
4.00
6.00
8.00
10.00
12.00
1 3 5 7 9 11 13 15 17 19
No. of Training Cycles
Tw
o W
ord u
tterances
P=0.001P=0.01P=0.1
Future Work:Majority of multimodal models of early child language are static:
Image (input)
Image (output) Label (output)
Label (input)
Plunkett et al model, 1992
perceptual input
Conceptual Relation input
One Word Output
Nyamapfene & Ahmad, 2007
• Spoken words can be viewed as phoneme sequences and/or syllable sequences
• Motherese directed at preverbal infants relies heavily on the exaggerated emphasis of temporal synchrony between gesture/ongoing actions and speech (Messer, 1978; Gogate et al, 2000; Zukow-Goldring, 2006)
But in, reality, the early child language environment is both multimodal and temporal:
So we intend to model early child language as comprising phonological word forms and perceptual inputs as temporal sequences
We Will Make These Modifications
Dj0 Dj1 Dj2
(j-1)th Neuron (j+1)th Neuron
Pattern VectorContext Vector
Threshold unit
Delay units
Hebbian Link
Hebbian Link
To includePattern Items
from other sequences
1
To IncludeFeedback from
Concurrent Sequences
2
Thank YouDiscussion and Questions
??!!