Connectionist Approaches to Language Acquisition Kim Plunkett avec Julien Mayor, Jon-Fan Hu and Les Cohen Oxford BabyLab and UT, Austin
Jan 15, 2016
Connectionist Approaches to Language Acquisition
Kim Plunkett avec
Julien Mayor, Jon-Fan Hu and Les CohenOxford BabyLab and UT, Austin
How to figure out the meaning of words…?
• Huge number of possible meanings
• Hierarchical level?
• Relation between objects?
Learning constraints?
e.g. whole object, taxonomic, …
The weak ‘taxonomic constraint’
Given a choice between a thematically and a taxonomic related object, language users will favour the taxonomic choice over the thematic choice
(Markman & Hutchinson 1984)
The strong ‘taxonomic constraint’
« When infants embark upon the process of lexical acquisition, they are initially biased to interpret a word applied to an object as referring to that object and to other members of its kind »
(Waxman & Markow 1995)
Important implication of the TC
From a single labelling event, infer that every object that belong to the same category is called with the same name
– Powerful communication tool– Refer to objects one has never seen
Controversy
• Specifically linguistic?• How about the shape bias?• Innate? Learnt?
Experimental status is unclear, e.g. Markman & Hutchison ’84 do not demonstrate the strong taxonomic constraint
Objectives
• We investigate how lexical organisation can use pre-lexical categorisation…
• We want to understand the conditions necessary for having good generalisation of labels to object of like kinds (taxonomic constraint)…
…within a modelling framework
The model Baby
• Early infancy; Baby gets experience with visual and acoustic environment
• Later infancy, joint attentional activities with care-giver become important:
=> she gets simultaneous presentations of objects and labels
The model of the model Baby!
• Early infancy: unimodal maps (SOMs) receive input from visual and acoustic world (objects and labels)
• Later infancy: object and their labels are presented at the same time. Synapses linking active neurones on both maps are reinforced
Speech perception development
• Initial sensitivity to speech… – prenatal exposure experiments (Mehler, Fifer &
Moon 2003), fetal voice recognition (Kisilevsky 2003)
• …to learning of language-specific sound patterns…– forward vs backward speech (Dehaene-Lambertz
et al. 2002)
• … and word segmentation – at 7-8m (Jusczyk & Aslin ‘95)
Visual stream development
• Basic structure of cortical maps innate but experience essential (Crair & al. 1998)
• Contribution of sensory experience to development of orientation selectivity:– Stripped environment, alteration of dist.
orientation preference (Blakemore & Cooper 2001)
– Environment deprivation (White & al. 2001 )
Modelling the unimodal perceptual development
We use Self-Organising Maps (SOMs, Kohonen 1984)– they achieve dimensionality reduction– they self-organise around topological maps– they work in an unsupervised way (ie
environment structures the maps)
Similar objects are mapped to neighbouring neurons
Visual input
Prototype Blurred version (2/3 of max)
Results reported on 20x20-sized images, 6 categories (cat, dog, cow, pig, sheep, bunny), 18 blurred images/prototype
Early stage of infant life; passive presentation of images & words
… ‘Dog’ … ‘Sheep’
Maps are structured, now we can build cross-modal associations
After maps are structured; joint attentional activities
‘Dog’
Hebbian learning is ‘switched on’ (with n BMUs)
Testing procedure
Label ?
whole image datasetDoes map structure supportgood generalisation?
Random assignment
Cat 1
Cat 2
Cat 3
Dog 1
Dog 2
Dog 3
Memorisation (one-to-one mapping)
Cat 1
Cat 2
Cat 3
Dog 1
Dog 2
Dog 3
Successful generalisation
Cat 1
Cat 2
Cat 3
Dog 1
Dog 2
Dog 3
Generalisation after a single labelling event
Random One-to-one Simulation
Cla
ssifi
catio
n su
cces
s (%
)
Role of # of pairings
Developmental aspect of generalisation
Role of map structure
Corr. of perceptual abilities with language (Tsao’04, Kuhl’05, …)
--- increasing map quality
Summary
• In a first phase, we feed SOMs with realistic input, visual and acoustic
• After maps are structured, we present a single word-object pair from a given class (e.g. dogs)
• Generalisation of the label to other images in the same class (other dogs) is successful
• We propose the taxonomic ‘constraint’ to be an emergent property of the network
Role of # of Hebbian links
Take-home messages• Map structure is critical for generalisation
(similarity measure)
• Good generalisation if many units are allowed to fire & wire together, even from single word-object presentation
• If we reduce the number of units that fire & wire together, we restrict generalisation; role for proper nouns?
Our neuro-computational account
• Taxonomic responding is an emergent property of the network, inevitable outcome if exposed to the right environment, non-language specific– Architectural constraint: cross-modal association
of two well-formed maps (early exposure to images and sounds)
– Algorithmical constraint: activity-dependent learning (Hebbian type)
• Mechanism for generalising associations between any formed categories
Labels facilitate CategorisationLabels are invitations to form categories
Waxman and Colleagues
Two Conditions
Labelling Condition No Word Condition
Novelty Preference in Labelling Condition but not in No Word Condition
Some Difficulties
• Familiarisation stimuli probably from categories familiar to the infant – not necessarily category formation but category activation
• Only one category is presented during familiarisation – category is not independently motivated by label
• “No Word” condition is not a silent condition• Infants show “Out of Category” novelty preferences in
the absence of labels• “No Word” condition is anomalous – perhaps
overshadowing is occuring: Sloutsky et al.
How to show that labels impact categories formation?
• Show that a new visual category has been formed• Show that the structure of labelling events influences
the structure of visual categories• Two possibilities
– use labels to motivate category formation in the absence of visual structure
– use labels to override existing visual categories to form a new category
– Requires two labels and/or two categories
Learning Novel Categories at 10-monthsYounger 1985
Familiarisation
Broad
1155 1515 2244 2424 4422 4242 5511 5151
Narrow
1122 1212 2211 2121 4455 4545 5544 5454
1111 3333 5555
Testing
Novelty PreferenceYounger 1985
• Evidence for One category in Broad Condition: 1111/5555 > 3333
• Evidence for Two categories in Narrow Condition: 3333 > 1111/5555
>
>
Impact of Labels on Categorisation in 10 month olds
• Replicate Younger’s Original Experiments (No labels or carrier phrases)– Experiments 1 and 2
• Familiarise with Narrow Condition in 3 different labelling conditions:– Experiments 3 – 5
– Two labels correlated with category membership– Two labels uncorrelated with category membership– One label for all stimuli
• Test for Novelty Preference– 24 Infants in each condition– Replication: 10 second familiarisation per trial – 4 test trials– Labelling: 6 second familiarisation per trial – 4 test trials– Testing conditions identical across all 5 experiments, i.e., in silence
Familiarisation Findings
• Hearing labels highlights attention to objects
• Infants track perceptual similarity of objects
Replication of Younger’s original findings(no label conditions)
Experiment 1Broad Condition
0
0.5
1
1.5
2
2.5
3
3.5
4
Looking Time (sec)
3333
5555/1111
*
Evidence for One category
Experiment 2Narrow Condition
0
0.5
1
1.5
2
2.5
3
3.5
4
Looking Time (sec)
*
Evidence for Two categories
Impact of labels on visual category structure – two categories
Experiment 32 Labels Correlated
0
0.5
1
1.5
2
2.5
Looking Time (sec)
*
Evidence for Two categories
Experiment 42 Labels Uncorrelated
0
0.5
1
1.5
2
2.5
Looking Time (sec)
No evidence for categories
Experiment 51 Label
0
0.5
1
1.5
2
2.5
Looking Time (sec)
*
Evidence for One category
Conclusions
• Infants compute the cross-modal statistical correlations between labels and visual objects during the categorisation process
• Labels can override dissimilarities between objects so that they are treated as being perceptually more similar.
• No evidence for label facilitation• No evidence for auditory dominance
Ongoing and Future Work
• Can labels divide as well as unite?
• Do words have a privileged status?
• The impact of perceptual similarity
• Integration with word learning