J.Vredenbregt and W.G.Koster - Research | Philips Bound... · J.Vredenbregt and W.G.Koster ... muscle stimulator for hemiplegie patients, Philips tech. Rev. 30,23-24, 1969. 74 J.VREDENBREGT

Philips tech. Rev. 32,73-78, 1971, No. 3/4 73

Analysis and synthesis of handwriting

J. Vredenbregt and W. G. Koster

Although the work described here will very likely be of interest to designers of equipmentfor reading handwriting automatically, the authors did not set out with this aim in mind.Their work is in fact part of a more general study being made at the Institute for Percep-tion Research (IPO) in Eindhoven on the motor system of the human body. The hand-writing process is used for studying the programming of muscle activity in relation to themovement produced.

IntroduetionEvery movement made by a human being is the

result of the precise coordination of a number ofmuscles. A movement carried out using only one jointis referred to as a single movement, one using morethan one joint as a composite movement. The writingprocess is an example of a composite movement. Theprocess is controlled by programmed activity of severalgroups of muscles, which produce the movements offorearm, hand and fingers. The process comprisesvirtually all aspects of the motor system of the humanbody. As a logical continuation of the studies onmuscle mechanics [1] started some years ago at the In-stitute for Perception Research (IPO), and in pursuanceofthe work of J. J. Denier van der Gon [2] and of J. S.MacDonald [3], it was therefore decided to embark on astudy of the production of handwriting [4]. It wasreasonable to expect that a study of the static anddynamic behaviour of the muscles during writingwould provide a better understanding not only of thewriting process itself but also of the rules underlyingthe programming of composite movement processes.This could be useful in developing power-controlledartificial limbs and muscle stimulators [5].

The methodology underlying this investigation ofmuscle activity is comparable with that used in thephonetic research at IPO. An instrument was construct-ed for analysing the phenomenon - in this case thewriting process - and a second instrument was builtwhich simulates the writing movements of the handand can be used to synthesize letter characters. Cursivehandwriting can be simulated and slight changes madein the shape of characters by simply changing the timeparameters. The success achieved in synthesizingcharacters is an indication that the hypothesis on whichthe instrument is based is not incorrect.

J. Vredenbregt and Ir. W. G. Koster are with the Institute forPerception Research, Eindhoven.

Analysis of the writing movementThe handwriting analyser we have constructed is an

instrument that records a displacement-time diagram,i.e. a curve that gives the position of the pen as afunction of time. In designing the analyser we regardedthe writing movement as being composed of twoseparate movements: one in the direction of writing- which we shall refer to as the x-direction - and onein the direction perpendicular to it - the y-direction.The x-movement is the result of a rotation of the handfrom the wrist or of the forearm from the elbow joint,or of a translational movement of the forearm. Themovement perpendicular (or slightly oblique) to thewriting direction is produced by the thumb, the indexfinger and the middle finger; sometimes it may beproduced by the whole forearm. In addition to thesetwo movements there is, of course, a third one, to putthe pen on to the paper and to lift it, but for the presentpurposes we can ignore this.

In analysing the writing movement we have mainlystudied the displacements in the y-direction, becausethey are larger than those in the x-direction and farless complicated. Preliminary studies had also shown

[1) J. Vredenbregt and W. G. Koster, Some aspects of musclemechanics ill vivo, IPO Annual Progress Report 1, 94-100,1966. J. Vredenbregt and W. G. Koster, Measurements onelectrical and mechanical activity of the elbow flexors, Bio-mechanics I, 1st Int. Seminar, Zurich 1967, pp. 102-105(Karger, Basle/New York 1968).

[2) J. J. Denier van der Gon and J. Ph. Thuring, Kybernetik 2,145, 1965.

[3) J. S. MacDonald, Quart. Progress Rep. M.l.T.-R.L.E. 76,210, 1965.

[4) See also: J. Vredenbregt and W. G. Koster, Analysis andsynthesis of handwriting, IPO Annual Progress Report 2,157-161, 1967; J. Vredenbregt, W. G. Koster and J. W. Kirch-hof, On the tolerances in the timing programme of synthesisedletters, ibid. 3, 95-97, 1968; W. G. Koster and J. Vredenbregt,Analysis and synthesis of handwriting, Biomechanics Il, 2ndInt. Seminar, Eindhoven 1969, pp. 77-82 (Karger, Basle/New York 1971).

[5) See for example H. J. van Leeuwen and J. Vredenbregt, Amuscle stimulator for hemiplegie patients, Philips tech. Rev.30,23-24, 1969.

74 J. VREDENBREGT and W. G. KOSTER Philips tech. Rev. 32, No. 3/4

that further investigation of movements in the x-direc-tion would not yield more information than the in-vestigation of movements in the y-direction.

Fig.1 givesa schematic viewofthe instrument used formechanically recording the y-coordinate of the move-ment of the pen at point 0 as a function of time. Theelectrical signal that is a measure of the displacementin the y-direction is the voltage across the strain gaugesQ, attached to one of the leaf springs S. Dependingon the displacement of the pen the leaf springs aredeflected to a greater or lesser extent by the disc R,thus varying the .resistance of the strain gauges.The construction of the instrument is kept as light as

possible to ensure that the mass of the moving partsdoes not unduly affect the recording. Displacement ofthe pen in the x-direction is possible because the detec-tion system for the y-coordinate just described can bemoved as a whole along a guide shaft A. The move-ments in the instrument take place practically withoutfriction because a cushion of air is maintained betweenthe block B and the shaft A, and also where the shaft Cpasses through a hole in the block B. To ensure thatblock B does not stick because of non-coaxial move-ment when it is moved along the shaft, the displacementfrom 0 in the x-direction is transmitted to B by meansof a thin nylon cord which passes over four rollers,i.e. twice around rollers PI and P2.

The frequency characteristic of the analysis systemis such that signals of the amplitudes normally en-countered in handwriting and up to 25 Hz in frequencyare reproduced without distortion. In the frequencyrange below 25 Hz there is no phase shift betweeninput and output signals. The frequencies encoun-tered in handwriting movements all lie below 15Hz.An example of the displacement-time diagram forthe y-direction of the letter a is shown in fig. 2.A fundamental question to which our analysis of

cursive handwriting could supply the answer reads:when different persons write the same character, is itpossible to recognize general laws, unconnected withthe person writing the character, from the recordeddisplacement-time diagrams? The results obtained sofar indicate that this might indeed be the case. Fig. 3shows five characters a written by five subjects withthe pen of the analysing instrument described above.Individual differences in the shape of the characterscan clearly be seen. Nevertheless, there is a certain sim-ilarity ofpattern in the displacement-time diagrams, inthe times taken to write the successive parts of a char-acter. On dividing the total writing time for the char-acter into intervals of similar direction, by marking thetime axis at the points where the y-movement reverses,it is found that the ratios between the successive inter-vals are about the .same for all subjects. This is also

G_B P,A

y

Î_Xo

Fig. 1. Instrument for recording the writing movement. 0 posi-tion of the pen. Movements in thè direction of writing (the x-direction) are transmitted by a nylon cord to the block B, whichcan move along the guide shaft A. Movements perpendicular tothe x-direction (the y-direction) are transmitted via the shaft C,to which a disc R is fixed, to deflect two leafsprings S to a greateror lesser extent. Q strain gauges, which provide a voltage that isnearly proportional to the deflection of the pen.

y

t

o 200 600ms400-t

Fig. 2. Displacement-time diagram showing the movements of thepen perpendicular to the x-direction during the writing of thecharacter a.

Fig. 3. The character a as written by five subjects, the writing timebeing divided into intervals tl-tso The intervals are marked by theinstants at which the y-component of the movement reversesdirection, as seen from the displacement-time diagrams. The ratiobetween these intervals is virtually constant for all five subjects.

Philips tech. Rev. 32, No. 3/4 HANDWRITING STUDIES 75

found to be the case for other characters. Thus, in spiteof individual differences in the shape of the character,this ratio is more characteristic of the character thanof the writer. The same constant ratio is found whetherthe subject writes a particular character quickly orslowly or large or small.

We shall now take a closer look at the way in whichskeletal muscles bring about such movements. Amuscle causing a movement in one direction is calledthe agonist, and that causing movement in the oppositedirection is called the antagonist. When an agonistcoritracts, the part ofthe body involved is set in motionand accelerated for as long as this muscle exerts aforce as a result of its activity. During this movementthe antagonist is passively stretched, which takes upenergy and thus offers resistance to the movement. Infast movements, moreover, the activities of the agonistand the antagonist are to some extent interdependent.When the muscular activity ceases, the movementcontinues for a time because ofthe characteristics ofthemuscle and the inertia of the mass of the part of thebody. When the antagonist contracts, the movementslows down and changes direction.

Whether and to what extent a muscle is active iseasily ascertained because muscular activity is associ-ated with electricity. The electrical effect is due to ion

o 60 200 290 420 560I I I I II I I I I

~f~

80 170 310 400

o 100 200 300 400 500-i

Fig. 4. Displacement-time diagrams for the movement perpen-dicular to the direction of writing, together with the associatedelectromyograms recorded during the writing of the character 0'.

exchange in the membranes surrounding 'the musclecells; and with surface electrodes the electrical signalscan be detected at the skin as a varying potentialdifference (electromyography).

Fig. 4 shows the displacement-time diagrams re-corded during the writing of the character a, togetherwith the associated electromyograms. The arrowonthe right of the electromyograms indicates the direc-tion of movement to which the various muscle activi-ties relate. A comparison between the displacement-time diagram and the associated muscle activitiesshows that the muscular activity indicated by the elec-tromyogram is usually of shorter duration than themovement it causes, and starts at the instants at whichthe movement must be started, slowed down or changedin direction. In "acquired" movements of this kind,agonist and antagonist are seldom active simultane-ously.

Results such as those described above suggest thefollowing hypothetical model for the writing process.Writing can be compared with the operation of amechanical system that possesses a certain inertia andis controlled by an excitation pattern of short periods ofunequal duration. The magnitude of the force, andhence the amplitude of the movement, depends moreon the duration than on the strength of the excitation.

Synthesis of written characters

tA handwriting simulator

To test this model of the writing process we havebuilt an electromechanical instrument that has thecharacteristics just described and also has the mechani-cal characteristics and limitations of the human hand(fig. 5). This instrument can indeed produce writing.

The energy is supplied by four d.c. electric motorswith starting and stopping characteristics closely re-sembling those of skeletal muscles. The motors aremechanically coupled in pairs, one pair for moving thestylus in the two x-directions, the other pair for themovement in the two y-directions. The effect of thiscoupling is that one motor in a pair acts as the "agonist"while the other acts as the "antagonist". When aparticular motor is actuated, its partner operates as agenerator, and this energy is dissipated in a resistor,which provides damping for the system (fig. 6). Thenon-actuated motor thus fulfils the role of the passivelystretched antagonist. The inertia required for the simu-lation is provided by the mass of the armature of themotor, the stylus and the other moving parts. Theoverall translation of hand and forearm along the lineis simulated by means of a lead-screw, and the stylus israised and lowered by means of an electromagnet.

76 J. VREDENBREGT and W. G. KOSTER Philips tech. Rev. 32, No. 3/4

responding to the amplitude of the supply voltage -is set such that the maximum velocity at which thestylus moves, both vertically and horizontally, isequal to the actual speed at which a hand-held penwould move.

The programme in which the motors are actuated ispreset by means of an electronic device [6] that deliversthe commands for the actuation pulses at the appro-priate instants. The device has various output channels,and the prograrnmed selection of a given output chan-nel determines which motor is actuated. The instant ofactuation can be varied in steps of I millisecond. It ispossible to make the periods of actuation of thevarious motors overlap.

Fig. 5. The handwriting simulator. Two mechanically coupled d.c. motors cause the pen tomove in the x-direction; the other coupled pair give the displacement in the y-direction.The whole system can be moved in the x-direction by means of a lead-screw. The pen is raisedand lowered by means of an electromagnet.

All four motors are supplied separately with voltagepulses of constant amplitude and variable duration inaccordance with a programme which is characteristicof the letter to be written. This is illustrated schemati-cally in fig. 7 for the character a. The exitation - cor-

v rlL- c::J4 -t

~R_

------ <;;;

Fig. 6. Waveform of the terminal voltages of a pair of mechani-cally coupled motors. The actuation pulses from both motors areshown with no shading (e.g. pulse i). The shaded area 2 showsthe waveform ofthe voltage generated when the motor continuesto run on under the influence of its inertial mass. If the othermotor is now actuated (pulse 3), the first motor starts to turn inthe opposite direction and a negative terminal voltage is generated(area 4). The energy generated in the shaded periods is dissipatedin a resistor, which provides damping for the system.

Timing programme

From the shape of a character and the way in whichit is produced when written by hand, the sequence inwhich the simulator motors have to be actuated maybe derived. Determining the timing and duration oftheactuation is a process of trial and error, with the"naturalness" of the character produced as the yard-

Philips tech. Rev. 32, No. 3/4 HANDWRITING STUDIES 77

v0, t

mf mD

500ms-- .. t

200 300 400o 100

/'a(...{_. , .Fig. 7. A timing programme for synthesizing the character a.Thesupply voltage for the four motors is plotted schematically in thevertical direction. The periods of excitation are indicated by theletters A to K, and an arrow indicates the direction in which theparticular motor causes the pen to move. The parts of thecharacter corresponding to the periods are shown below thetiming programme.

-

ing writing the direction of movement is reversed justafter the excitation is switched on, and that when theterminal voltage is switched off the inertia of thesystem causes the pen to continue moving for a shorttime. Fig. 9 shows various simulations of the charactera from fig, 3. It can be seen that the simulated charac-ters are barely distinguishable from the ones shown infig. 3.The differences that give handwriting its individ-uality can be simulated by making relatively smallchanges in the timing programme, i.e. by varying thestarting time and duration of the pulses actuating themotors.An alteration in the unit of time for the electronic

programming switch makes all components of the pro-.gramme proportionally longer or shorter, resulting' ina larger or smaller character with otherwise the sameshape. However, since the inertia of the system remainsunaltered, larger variation in timing will affect theshape of the character.

-

I ol~ "'""

)1 ~ ~L~[LCe_

I iy

t200 400 600ms 0

-t200o

Del

400 600 800ms 0-t

200 400 600 800ms-t

Fig. 8. Some characters produced by the simulator, with the associated displacement-timediagrams and the terminal voltages of the motors. In the shaded periods the motors act asgenerators.

Fig. 9. Simulations of four of the five characters a from fig. 3. The differences in shape areproduced by making slight changes in the timing programme of the simulator.

stick for judging the correctness of the programme.Using the knowledge gained from the analysis theprogrammer can soon acquire a certain skill.

Fig. 8 shows some characters written by the simulator,giving for each character the variation of the motor ter-minal voltages and the displacement-time diagrams ofthe pen. It can be seen that Iike muscular activity dur-

An interesting effect has been obtained by giving themotors a bias voltage. The conditions are then corn-parable with increased muscular tension, resulting in akind of cramped writing. The characters written bythemachine are no longer flowing but look somewhat[61. G. J. J. Moonen and C. Ai Lammers, A preset cascade

counter, IPO Annual Progress Report 1, 104-106, 1966.

78 HANDWRITING STUDIES Philip;' tech. Rev. 32, No. 3/4

240ms

Fig. 10. Variations of about20ms in the duration ofthe pulseor in the instant at which it isinitiated cause deformations inthe character. In the upperseries the starting point t of theexcitation period C (see fig. 7)was varied; the end point waskept constant at the same instantas in fig. 7 (275 ms). In the lowerseries the length of the period Cwas kept constant but the posi-tion along the time axis wasvaried. The time zl t is the dis-placement with respect to theposition in fig. 7 (t = 220 ms).

I I I , I

• l.

S_ ê q_ CL CL ~ tfr' VL

...,

I I I I I

180 200 220-t

I I I I

/L ~ ?L (,.-L- a_ Q Cv ()o{./

I I I I I

-40 -20 o

uncoordinated, as if they had been written by a personwith the faulty coordination of movement found insome slightly spastic subjects.A study of the effects of changes in the programme

on the legibility of the characters showed that thetolerance for a shift of a' period of excitation along thetime axis is at least twice the tolerance for an alterationin the lengt]: of that period. The most critical periodsin the' programme for the character a appear to be C,E and G. The periods C and E directly affect the loopof the character a, particularly at the point where itshould close and the downward stroke begins. Theperiod G mainly controls the length of this downwardstroke. As a rough guide, changes of 20 milliseconds inthe duration of the excitation will often be enough todeform a letter beyond recognition (fig. la).From the investigations described we conclude that

a natural-looking simulation of handwriting can beproduced by a system in which the mechanical para-meters are constant and only the time parameters of theactuating signal are varied. Some verification that thecharacters are produced in much the same way as inwriting appears from the resemblance between thedisplacement-time diagram of the simulation and that

20 40-L1t

60ms

of the movement of the hand, and also from theagreement between the pattern of the pulses actuatingthe simulator and the pattern of the electromyographicsignals recorded during handwriting.Since deviations of between 5 and 10 milliseconds in

the simulation are enough to cause perceptible differ-ences in the character, it is reasonable to assume thatmuscles are controlled with the same degree of ac-curacy.

Summary. The handwriting process was chosen as a topic ofstudy in the context of research on muscular activity in theexecution of composite movements. An instrument was builtwhich records displacement-time diagrams during the writing ofcharacters by hand. These diagrams correlate well with simul-taneously recorded electromyograms, which give a picture of themuscle acitivity. The writing process can be compared with theaction of a mechanical system that possesses a certain inertialmass and is controlled by constant excitation during a numberof periods of unequal duration. To test this model a simulatorhas been built which can produce written characters. It consistsof two pairs of d.c. motors which move a pen to and fro intwo orthogonal directions. The motors are actuated with constantvoltage pulses in accordance with a timing programme. Changesof 5 to 10 milliseconds in the programme characteristic of a partic-ular character cause changes in the character that correspond toindividual differences in handwriting. This makes it reasonable to .assume that muscles are controlled with comparable accuracy.