LANGUAGE ORIGINS SOCIETY UNIVERSITY OF NIJMEGEN 4-5 JULY 2003 Robin Allott LANGUAGE AND SPEECH AS MOTOR ACTIVITIES.

LANGUAGE ORIGINS SOCIETY

UNIVERSITY OF NIJMEGEN

4-5 JULY 2003

Robin Allott

http://www.percepp.demon.co.uk

LANGUAGE AND SPEECH AS MOTOR ACTIVITIES

ABSTRACT

For the last half-century study of the functioning of language has taken two separate forms: an abstract purely linguistic approach - and the application of the increasingly powerful techniques of neuroscience to the functioning of language in the brain ( PET, fMRI, MEG and ERP ).

The language capacity is a product of the evolution of the human brain and not a purely conventional and cultural phenomenon (the Saussurean concept).

The motor theory of language is that there is a direct relation between the functioning of speech and motor control generally, with language depending on pre-existing motor primitives together with the operation of motor equivalence.

Motor primitives are elementary action-units instantiated in the CNS which make possible the construction of complex action-sequences.

Motor equivalence means that the same motor program can be executed by different muscle/joint assemblies.

MOTOR THEORY OF LANGUAGE

The Motor Theory is a theory of the origin and functioning of language. The theory is that the structures of language (phonological, lexical and syntactic) were derived from and modelled on the pre-existing complex neural systems which had evolved for the control of body movement. Motor control at the neural level requires pre-set elementary units of action which can be integrated into more extended patterns of bodily action -- neural motor programs. Speech is essentially a motor activity (a stream of articulatory gestures). Language made use of the elementary pre-set units of motor action to produce equivalent phonological units (phonemic categories). The neural programs for individual words were constructed from the elementary units in the same way as motor programs for bodily action. The syntactic processes and structures of language were modelled on the motor ‘syntax’.

“Although listeners obviously cannot have kinesthetic feedback from someone else's articulation, they interpret what they hear by implicit motor- matching. ... actual movements of the organs of speech become unnecessary; the appropriate pattern of impulses within the central nervous system is enough.”

Hockett, C.F. 1987. Refurbishing our foundations: Elementary linguistics from an advanced point of view.

The presentation draws on recent research in neuroscience by Rizzolatti and his colleagues on mirror neurons, by Graziano on cortical action programs, by Mussa-Ivaldi and Decety on motor imagery, and by a number of other researchers on motor primitives and motor equivalence.

Also older material from Lotze, William James, Lashley, and Bernstein as well as books by Berthoz and Jeannerod.

The paper seeks to integrate the motor theory of language into neurological research on motor control, particularly recent research on motor programs, motor imagery, motor primitives and motor equivalence.

This talk is about the relation between the movements we make in speaking and the movements we make generally: of the limbs, the head, the body, of the face.

How do you move your arm? How do you move your tongue?

The outline of an explanation is beginning to emerge

MOTOR CONTROL

The aim is to look in broader terms at the progress made in thinking about the hierarchy, stages and phases in motor control which need to be established before the location or identification of the network connections becomes possible (if it ever will be.)

A great deal of research is attempting to discover where in the brain particular aspects of motor control are dealt with

And ultimately to discover the network connections which are operative in motor control.

This not the subject of this paper. The research in any case is not sufficiently advanced and is often controversial

The problem in motor control is that of the multiple degrees of freedom, the multiplicity of muscles involved in each movement, the complex relation between posture and movement, together with the need to integrate motor action with visual perception, to allow for dynamic factors, etc.

Ways in which the problem has been tackled:

“If mirror neurons form the basis for a “vocabulary” of actions, then, it is of great importance to understand how the words of this vocabulary may be combined with each other by the brain to span a repertoire of purposeful behaviors how these action goals may be translated into movements so that their concurrent activation lead to meaningful results.”[Mussa-Ivaldi 1999]

Theoretical and experimental studies converge on the concept that complex control problems may be solved by a combination of independent modules. From a neurophysiological perspective, these modules organize specific synergies of muscles. From a mechanical perspective, these modules generate force fields upon the controlled limbs. [Mussa-Ivaldi]

The present view:

[The motor program may form part of higher level action plans]

Execution of the motor program using motor primitives of muscle/joint organisation

Disinhibition of the motor program

Assembly of pre-existing motor routines into the motor program

Integration of the motor program with the body image and the environment image

Conversion of the motor image into a motor program

A motor image - a motor idea

We make any movement through the combined operation of:

MOTOR CONTROL

MOTOR IMAGE

MOTOR PROGRAM

MOTOR PROGRAMEXECUTE !

ENVIRONMENT

IMAGE

BODY

IMAGE

MOTOR PRIMITIVES

DISINHIBIT!

MOTOR ROUTINES

The typical subject matter of recent research has been concerned with movements of the hand, arm and foot - or of animal limbs: how the muscles and joints are organised for movement and flexing of the arm.

What is involved?

MOTOR IMAGERY

A validation and extension of the ideas of William James and Hermann Lotze is now provided through brain-scanning techniques.

The motor system not only executes actions but also internally represents them in terms of ‘motor ideas’.

Current research in motor imagery is focused on similarities between actual and imagined movements on a central and a peripheral level of the nervous system.

This gives a neurological realisation of what in evolutionary terms must have been an intimate intertwining of perception and action not as separate functions but as part of single system for effective behaviour in any creature's environment.

PET and fMRI scanning demonstrates that bodily action is preceded by a mental picturing of the proposed action. A perceptually-organised pattern is transduced into a motor program and executed by the changes in posture, changes in limb positions which constitute human action.

DECETY

Primary motor cortex activation during actual (right) and imagined (left) gesture with the right hand - fMRI

”Mirror" neurons discharge both when the monkey makes a particular action and when it observes another individual (monkey or human) making a similar action.

Transcranial magnetic stimulation and positron emission tomography (PET) experiments suggest that a mirror system for gesture recognition also exists in humans and includes Broca's area. Rizzolatti et al. 1999]

MIRROR NEURONS

Mirror neurons by linking brain systems for perception and action make possible the imitation of gesture and other bodily action.

MIRROR NEURONS

VISUOMOTOR NEURONS

AUDIOVISUAL NEURONS

MULTIFUNCTIONAL NEURONS

PERCEIVED MOTORPROGRAM

VISUOMOTOR NEURON

EXECUTE

ACTION

EYE

PREMOTOR

MOTOR PROGRAMS

The concept of the motor program has evolved from earlier understanding of motor action in terms of ‘schemas’ (Henry Head) and motor ‘patterns’.

The motor programs, patterns or schemas operate at different levels, or in parallel. At the highest level the motor program may be better described as an ‘action program’; at the lowest level there are motor sub-routines, motor elements.

Motor elements are combined in chains and in combination contingent on the interaction of feedback and central motor programs. [Marsden]

Brain motor commands are patterned in terms of movements rather than in terms of muscles. [Desmedt 1985]

Experimentally, the importance of preprogramming in the control of movement has been well established. [Hollerbach 1985]

Many aspects of motor behaviour, and particularly expressive motor behaviour, are found in new-born infants; the neural connections to support the behaviour must have been established before birth. This requires that the elementary motor subprograms, motor units, from which the more complex movements are constructed, must also be pre-wired.

Innate and acquired motor programs

A range of hardwired motor programmes is available within the immature nervous system [Marsden et al. 1985]

That this must be so is shown very clearly in the case of many animals.

MOTOR PRIMITIVES

“Our data provide strong, objective support for the conjecture made in the past by many other motor control researchers that a repertoire of motor primitives constitute fundamental building blocks of complex motions.

There is compelling evidence that early post-stroke recovered motions are composed of isolated segments. In 20 [patients recovering from a single cerebral vascular accident (stroke), we identified the apparent submovements that composed a continuous arm motion.”[Krebs et al. 1999]

Low-level primitives

The neural circuits in the spinal cord [of the frog] are organised into a number of distinct functional modules. The simultaneous stimulation of two sites [in the spinal cord] leads to the vectorial summation of the endpoints generated by each site separately. This linear behavior is quite remarkable.

“We regard these force fields as computational primitives that are used by the CNS for generating a rich grammar of motor behaviors - motor primitives in analogy with language primitives used to generate unlimited sentences out of a finite vocabulary of words.” [Mussa-Ivaldi & Bizzi 2000]

““If mirror neurons form the basis for a “vocabulary” of actions, then, it is of great importance to understand how the words of this vocabulary may be combined with each other by the brain to span a repertoire of purposeful behaviors how these action goals may be translated into movements so that their concurrent activation lead to meaningful results.”[Mussa-Ivaldi 1999]

Higher-level primitives

PENFIELD’S HOMUNCULUS

Stimulation of each cortical site in the right hemisphere evoked a different final posture of the left hand and arm.

2002 The Cortical Control of Movement Revisited. Michael S.A.Graziano, Charlotte S.R. Taylor, Tirin Moore, and Dylan F. Cooke, Department of Psychology, Princeton. Neuron, 36, 349–362, October 24, 2002.

Regardless of the starting position, stimulation caused the hand to move toward a specific final position.

"One possibility is that the mechanisms for speech were built on a preexisting mechanism for motor control" (Graziano et al.355)

MOTOR EQUIVALENCE

That the same movement can be executed by different effectors was called ‘motor equivalence’ by Lashley (1930); Bernstein described the concept of motor equivalence to achieve specific kinematic goals.[1967]

The motor image or motor idea floats free of the particular limb or particular muscle/joint assembly usually employed to execute the motor program.

The same pen-stroke can be realised by an infinite number of joint rotation patterns.

“I can write the letter A with my hand, with my foot, or even with my mouth; I could even make an A by walking on the beach.” [Berthoz 1997]

Subjects wrote their signature with their dominant index finger and ipsilateral big toe. fMRI showed that movement parameters for this movement are stored in secondary sensorimotor cortices of the dominant hand.

These areas can be accessed by the foot and are therefore functionally independent from the primary representation of the effector. [Rijntjies et al. 1999]

MOTOR EQUIVALENCE

SIGNATURE

MUSCLES OF HAND AND ARM

MUSCLES OF LEG AND FOOTMUSCLES OF BODY

WALKING ON BEACH

MOTOR PROGRAM MOTOR CORTEX

SIGNATURE IMAGE

The present results demonstrate the existence of motor equivalence in a combined upper and lower extremity task. [Marteniuk et al. 2000]

Dynamic cortical activity in the human brain reveals motor equivalence. [Kelso et al. 1998]

LANGUAGE AND SPEECH AS MOTOR ACTIVITIES

The results show the same type of alterations of the temporal organisation of speech as those characteristic for rapid alternating limb movements.

They support the view that the speech and skeletomuscular systems share common neural control modes despite fundamental biomechanical differences. [Volkmann et al. 1992]

Language is also a mode of action. Language and speech depend basically upon complex serial neuronal events (action programs).

A strict relation exists between language and motor control. [Gentilucci]

Comparing findings on the motor organisation of speech with the organization of voluntary movements about the elbow, we have found that the kinematic patterns for movements of the tongue dorsum were similar to those of voluntary flexion-extension movements about the elbow. [Ostry and Cooke 1987]

The task dynamic model we are using for speech was exactly the model used for controlling arm movements, with the articulatorsof the vocal tract simply substituted for those of the arm.

Such gestures not only can characterize the movements of the speech articulators but also can act as phonological primitives. [Browman and Goldstein 1991]

So what is the nature of the tongue and other articulatory components?

The motor control of the tongue is a section of the total system of motor control for all bodily movement (including arm and facial movement)

The tongue manipulates food, dilates the airway during inspiration, and shapes the sounds of speech. While performing these functions the tongue morphs through many complex shapes.

Tongue and other articulatory elements

TONGUE MUSCLES AND NEURAL CONTROL

Extrinsic muscles move the tongue about in the oral cavity; intrinsic muscles change the shape of the tongue. 4 extrinsic muscles: genioglossus pulls the tongue forward or protrudes it; hyoglossus pulls the tongue down and posterior or retracts and depresses; styloglossus retracts and elevates the tongue.

Intrinsic muscles: longitudinis linguae shapes the tongue for speech and mastication; transversus linguae compresses the sides of the tongue; verticalis linguae with fibres running suproinferiorly.

The tongue: a large collection of muscles covered by mucous membrane.

Tongue movements can be explained in terms of a small number of independent muscle groups, each corresponding to an elementary or ‘primitive’ movement, making use of a relatively small number of invariant muscle synergies. [Sanguineti 1997]

In dogs, the intrinsic muscles are composed of many neuromuscular compartments. The superior and inferior longitudinal muscles each had an average of 40 distinct muscle fascicles, each supplied by a nerve branch. Each of the transverse and vertical muscles is composed of over 140 separate muscles sheets and every sheet is innervated by a separate terminal nerve.[Mu & Sanders 1999]

[The human tongue is not likely to be less complicated in its muscles and neural control than the tongue of the dog.]

If motor primitives are necessary to avoid the degrees of freedom combinatorial explosion for movements of the arm or leg, it seems inevitable that they are even more acutely required for the overwhelming complexity of the the tongue musculature and the unlimited possibilities of movement and change of shape the tongue can assume in speech.

MOTOR EQUIVALENCE IN RELATION TO SPEECH

MOTOR EQUIVALENCE

A as in HAT

ARM MUSCLESARTICULATORY MUSCLES

MOTOR PROGRAM MOTOR CORTEX

SPEECHSOUND IMAGE

EQUIVALENCEOF

SPEECH AND MOTORELEMENTS

Motor equivalence can operate from speech to gesture or from gesture to speech.

It also seems likely that it can operate between other modalities and speech - or more precisely between motor programs for other modalities and motor programs for speech.

Motor equivalence is demonstrated most remarkably in the relation between speech and gesture.

Speech and gesture arise as interacting elements of a single system. [McNeill 1987]

Every articulatory program can be redirected (through motor equivalence) to produce an equivalent movement of the hand and arm.

Every gesture structured by a perceived object or action can be redirected to produce an equivalent articulatory action.

SPEECH

GESTURE

VISION

HEARING

MOTOR EQUIVALENCE

IMITATION

Motor equivalence can function between speech and perception and between perception and motor action because perception is also a motor activity. There are motor programs and motor primitives for vision.

These can be transferred by motor equivalence to other muscle/joint assemblies e.g. imitating the shape of a scanned object or a perceived

movement by a gesture.

Vision is a highly motoric activity >>>>>

NOTON AND STARK

Eye Movements and Visual Perception 1971

EYE MOVEMENTS of subject viewing photograph of a bust of Queen Nefertiti [from Yarbus]

Imitation is of central importance in the relation between gesture and speech, that is between bodily gesture and articulatory gesture.

The capacity to transfer visual patterning from what is seen to corresponding bodily action is innate.

Newly-born infants can imitate what they see >>>>

MELTZOFF

Imitation based on the neonate's capacity to represent visually and proprioceptively perceived information in a form common to both modalities. Observations in six newborns- one only 60 minutes old - suggest that the ability to use intermodal equivalences is innate [Meltzoff and Moore 1977]

From the motor processes associated with perception can be constructed articulatory gestures - word-programs related to what is perceived

Making specific the relation between what has been said about motor control generally and speech

and language

MOTOR CONTROL AND THE MOTOR THEORY OF LANGUAGE

The motor theory is that there is a direct relation between the functioning of speech and motor control generally, with language depending on pre-existing motor primitives coupled with the operation of motor equivalence.

Mirror neurons, motor primitives and motor imagery research fit closely with the motor theory of language origin and function.

The motor control of the tongue is a section of the total system of motor control for all bodily movement. Tongue movements can be explained in terms of a small number of independent muscle groups, each corresponding to an elementary or ‘primitive’ movement

Motor equivalence is demonstrated most remarkably in the relation between speech and gesture. Motor equivalence can operate from speech to gesture or from gesture to speech and from vision to speech

Arm movements - associated gesture - result from transferring the motor image of the speech-sound primitives or the motor programs for words to the arm by motor equivalence, in the same way as the motor program for writing one’s signature can be transferred by motor equivalence to writing one’s signature with one’s foot.

Individual speech sounds are MOTOR PRIMITIVES

Words formed from primitive speech-sound elements are MOTOR PROGRAMS

Movements of the arm are the MOTOR EQUIVALENTS of the speech primitives

Before we produce a sentence there is a MOTOR IMAGE of the sentence

A sentence is a high-level motor program or ACTION PLAN

SPEECH MOTOR CONTROL

MOTOR IMAGE

EVENT IMAGE

MOTOR PROGRAM

UTTERANCE!

ENVIRONMENT

IMAGE

BODY

IMAGE

MOTOR PRIMITIVESSPEECHSOUND

PRIMITIVES

DISINHIBIT!

MOTOR ROUTINESWORD SHAPES

A MR U T SI CC LU EL SA T O R Y

HE HIT ME

E H I M TMEHITHE

SENTENCE PROGRAMHEHITME

MEHITHE

E H I M T

Words and equivalent gestures