Problems of artificial system – human subject interaction reliability by Mirko Novák * ) Josef Faber * ) Zdeněk Votruba * ) Problems of artificial system.

Problems of artificial system – human Problems of artificial system – human subject interaction reliabilitysubject interaction reliability

bybyMirko NovákMirko Novák * *))

Josef FaberJosef Faber * *))

Zdeněk VotrubaZdeněk Votruba * *))

*) Prof. Dr. Mirko Novák, Prof. MUDr. Josef Faber, Prof. Zdeněk VotrubaJoint Laboratory of System Reliability,Department of Control Engineering and TelematicsFaculty of Transportation SciencesCzech Technical University, Prague, Konviktská 20, 11000 Prague 1, Czech RepublicInstitute of Computer Science,Academy of Sciences of the Czech Republic,Pod vodárenskou věží 2, 18207, Prague 8, Czech Republice-mail: [email protected], [email protected]

The problem of interaction reliability between artificial system and human being (operator, user), though extremely important, is still not solved satisfactorily, even the control of dominant part of complex and complicated systems is now realized by computer or at least by computer assistance.

The human operator, who has to interact with powerful, complicated and often also efficient artificial system (such as transport system, aircraft, express trains, large trucks, extensive power stations, security and defense systems etc.), is imposed to requirements on fast and correct reactions on very variable actual situations. Such exposition exists usually for considerably long time of his/her service.

The resulting high load of such operator´s brain and nervous system results necessarily to decrease of his/her vigilance and to degradation of his/her attention.

These demands on human operator ability are These demands on human operator ability are often jointed with often jointed with influence of some internal influence of some internal

and/or and/or external factorsexternal factors. . These are e.g. These are e.g.

the long length of service, psychical isolation in the course of service, operators actual mental and physical state, climatic conditions, quality of environment in the particular cock-

pit or control room, monotony of the scene or image, which

operator has to observe etc.

The endeavor of our investigations is at The endeavor of our investigations is at present the present the diagnosisdiagnosis of attention decrease of attention decrease and of the related and of the related reduction of the reaction reduction of the reaction speedspeed to unexpected emerging situation. to unexpected emerging situation.

Such attention degradation can also result in real micro-sleep. Both this states of operators brain,

      the long lasting attention decrease and       the short lasting micro-sleep, can be extremely dangerous and can result

not only in huge material and financial damages, but also to losses of human lives.

The The requirements on a human operatorrequirements on a human operator of of an artificial system can be concentrated in an artificial system can be concentrated in

the following main categories:the following main categories: a)requirements on attention level and

continuity,b)requirements on the speed of operator

reaction,c) requirements on the correctness of operator

decisions.Within all three above-mentioned categories of

the reliability of human operator - technical system interaction a correlation naturally exists.

A straightforward correlation exists between A straightforward correlation exists between attention level and speed of reactionattention level and speed of reaction. .

Operators commanding at Operators commanding at high level of attentionhigh level of attention usually also possess very usually also possess very fast reactionsfast reactions. .

On the other hand cases can appear, when fast, almost impulsive reaction may not be accompanied by very high level of the operator’s concentration and attention.

Some people can react fast also when their attention is shared by different objects (they have very fast reflexes).

In addition, a high level of attention in the majority of cases leads to a very high probability of correct decisions and vice versa - if somebody is not concentrating enough, there is a rather low probability that his/her decision will be correct.

On the other hand, in the case On the other hand, in the case of very fast of very fast reactionsreactions accompanied by a very accompanied by a very low levellow level of a of a human operator’s human operator’s attentionattention, the probability of an , the probability of an

incorrect decision can increase significantly. incorrect decision can increase significantly. This is typical for a so-called This is typical for a so-called surprise reactionsurprise reaction, , from which a transition to a from which a transition to a panic reactionpanic reaction can can

sometimes be observed. sometimes be observed. A A drop in the attention leveldrop in the attention level of a of a

particular human operator can be caused by particular human operator can be caused by various external or internal reasons; various external or internal reasons;

some of them have a general character; some of them have a general character; the intensity of others depends significantly on the intensity of others depends significantly on

the the operator’s individualityoperator’s individuality. .

Among the Among the general conditions causinggeneral conditions causing the the decrease of attentiondecrease of attention is: is:

– Extreme length of a particular operator’s service without breaks,

– Operator’s physical and mental exhaustion,– Monotonous scene which the operator has to observe for

a long time,– Extreme temperature in which the operator has to serve

(too high or too low),– Extreme humidity in which the operator has to serve (too

high or too low),– Extreme air pressure,– Air smell, dust density etc.– Matters leading the operator to concentrate on problems

other than from his/her main service.

The The level of a operator’s attentionlevel of a operator’s attention can be measured by the use of can be measured by the use of some real figure of merit some real figure of merit LLATAT, expressed by real numbers, e.g. by , expressed by real numbers, e.g. by

the inverse of his/her reaction time RT.the inverse of his/her reaction time RT.The The time developmenttime development of probands of probands LLATAT has usually very variable has usually very variable

character. In an ideal case, when only character. In an ideal case, when only the factor of length of his/her the factor of length of his/her operation service can be taken into accountoperation service can be taken into account it has the form, it has the form,

sketched in Fig. sketched in Fig. 11..

Fig. 1: The typical shape of probands level of attention when no other factors besides the length of service are considered.

In the region of In the region of shallow attention decreaseshallow attention decrease the probands the probands reaction reaction timetime is usually significantly is usually significantly prolongedprolonged (often 3 to 5 time). In addition (often 3 to 5 time). In addition to this to this the anglethe angle, in which he/she is able to react in adequate , in which he/she is able to react in adequate manner manner on visual stimulion visual stimuli is significantly is significantly narrowednarrowed and the and the probabilityprobability of his/her of his/her false reactionfalse reaction in much higher. More over, in much higher. More over, some probands can stay in such stage for considerably some probands can stay in such stage for considerably long timelong time. . They are still able to operate the respective system in principle, but They are still able to operate the respective system in principle, but the the probabilityprobability that they that they operate it in safe manneroperate it in safe manner is very is very loweredlowered. . Actually, they Actually, they operateoperate it it in dangerous formin dangerous form. . When the particular proband When the particular proband is so lucky and causes no accidentis so lucky and causes no accident in in the course of his/her system operation during the region of shallow the course of his/her system operation during the region of shallow attention decrease, he falls into the second region, where the attention decrease, he falls into the second region, where the level level ofof his/her his/her attention falls suddenly to zeroattention falls suddenly to zero. His/her eyes close and . His/her eyes close and he starts the period of deep he starts the period of deep micro-sleepmicro-sleep. Here his/her . Here his/her ability to ability to control and operate the respective system well is reduced almost to control and operate the respective system well is reduced almost to zerozero..

Nevertheless, there is known, that some skilled professional truck drivers can drive also being in this region – of course their abilities to control the car are reduced to some small set of basic functions, like to keep the car in straight direction and to keep the constant speed. They relax for short time on such pieces of highways, which they know well as straight and quiet. Problems appear, when some unexpected situation happens – usually with result of heavy accident, because they do not react adequate.

In In real situationreal situation, such ideal time-dependence of attention reduction in the , such ideal time-dependence of attention reduction in the course of long operator service appears only exceptionally. The particular course of long operator service appears only exceptionally. The particular operator attention decreases for some time. After that, operator attention decreases for some time. After that, it can be provoked it can be provoked

by some alarming stimulus or situationby some alarming stimulus or situation. This is . This is followed by another followed by another decreasedecrease, and so long. , and so long.

However, However, after several repetitions of such cyclesafter several repetitions of such cycles, the sensitivity of the tired , the sensitivity of the tired operator to provocative stimuli appears to be much lower and his/her operator to provocative stimuli appears to be much lower and his/her

average level of attention it falls down definitely (see Fig. average level of attention it falls down definitely (see Fig. 22). ).

Fig. 2. To the temporary improvements of operators attention level

Till known, no on-board applicable method for Till known, no on-board applicable method for direct measurementdirect measurement of the human subject of the human subject

reaction time RTreaction time RT is known. is known.

Almost all direct methods are based on measurement of the time interval, which appears between the time of stimulus appearance and the time of human subject final reaction (inclusive the time of his/her particular effector function).

Such methods of measurement are considerably well realizable in laboratory. In on-board praxis, such provocation is usually unwanted because it can have negative influence on human operator. Therefore, for practical purposes the indirect methods of actual values of RT estimation are to be developed.

For this purpose one has to find a set of suitable, on-board measurable parameters which can be used for identification of the attention decrease and of onset of the micro-sleep.

Among such parameters belong: Among such parameters belong: the electro-magnetic activity of brain, frequency of breath, frequency of hearth beats, eye movements, skin resistance, face grimaces etc. All these parameters have some advantages and also

disadvantages. We have chosen the EEG activity as the dominant significant parameter for our investigations.

The reason is that this is probably the only one The reason is that this is probably the only one source of data, from which the almost source of data, from which the almost

immediate and reliable information about the immediate and reliable information about the brain function can be mined.brain function can be mined.

After mining information from many EEG time-series recorded on several tenths of human subjects in our laboratory we can analyze the procedure of the respective operator vigilance decrease and attention degradation almost immediately and with quite acceptable reliability. As basis for such information mining we have used certain relations, which can be found among some components of EEG spectra. Here one has of course to take into account that the EEG time-series are of the quasi-periodical and quasi-stationary character in principle.

Fig. Fig. 33: Example of EEG time series.: Example of EEG time series.

Due the quasi-periodical and non-stationary character of these Due the quasi-periodical and non-stationary character of these time series the time series the standard methods of spectral analysis are of standard methods of spectral analysis are of

limited use herelimited use here only. We have applied a Gabor filtration based on only. We have applied a Gabor filtration based on the use of polynomial filter of the 50 degree considerably the use of polynomial filter of the 50 degree considerably

successfully. Example of resulting spectra is presented in Fig successfully. Example of resulting spectra is presented in Fig 44..

Fig. 4a: Rail-road driver in relaxation stage, RT = 494 ms. Fig. 4b: Rail-road driver in relaxation stage, RT = 810 ms.

The The careful analysescareful analyses of many such results allowed us to find of many such results allowed us to find such dependence like that shown schematically in Fig. such dependence like that shown schematically in Fig. 55

Such dependence are of Such dependence are of very individual charactervery individual character and differ and differ for each proband, but seems to for each proband, but seems to not change for long timenot change for long time of his of his

life.life.

Fig. 5: Example of dependence among dominant amplitudes of two Gabor spectral lines and and reaction time RT.

Beside this we search for much sharper analytical tool, Beside this we search for much sharper analytical tool, by the use of which we can mine from the EEG time-by the use of which we can mine from the EEG time-series the information on the actual level of probands series the information on the actual level of probands

attention. attention. One of my graduate students, One of my graduate students, Ing. Petr SvobodaIng. Petr Svoboda has has

tried to use for this purpose:tried to use for this purpose:analysis of the poles and zeros locations of Z-transfors analysis of the poles and zeros locations of Z-transfors

of some approximations of selected time-series of some approximations of selected time-series windows,windows,

analysis of time development of some parameters of analysis of time development of some parameters of differential equations describing the respective time-differential equations describing the respective time-

series.series.Example of his results, which I consider as very Example of his results, which I consider as very

interesting is shown in Fig. interesting is shown in Fig. 6 6 and and 77..

Fig. Fig. 6a 6a

Fig. Fig. 6b6b

Fig. Fig. 7 7

Having so some information about the particular Having so some information about the particular proband attention behavior, we can construct a proband attention behavior, we can construct a set of set of suitable attention predictors,suitable attention predictors, which give which give

us the information about the expected us the information about the expected development of the particular operator attention development of the particular operator attention

in the course of his/her service. in the course of his/her service. Such predictors are to be individually adjusted to each operator under

consideration. We expect that these predictors can work with acceptable prediction reliability (about 90%) for the prediction horizon about 2 – 3 minutes.

 We hope that on the base of this knowledge, in the course of further research the development of on board-applicable measuring set will be possible. By the use of this we hope to be able to warn the system operator and eventually also his/her supervisor, that the particular subject is losing his/her attention and starts to operate the respective system dangerously.

The principal scheme of such a warning system for micro-sleeps is The principal scheme of such a warning system for micro-sleeps is

shown in Fig shown in Fig 8.8.

Fig. 8: Scheme of micro-sleep warning system. The yellow line represents the way for eventual prevention.

References:References:[1]…Mišovič M., Malý V.: Reliability of Information Systems,

CATE, Brno, 1993[2]…Blackmore S.: The power of memes,

Scientific American, Vol. 283, October 2000, No. 4, 52-63[3]…Dawkins R.: The selfish gene,

Oxford University Press, 1976, 1989[4]…Blackmore S.: The meme machine,

Oxford University Press, 1999[5]…Dotlačil M., Kopřiva P.: Networks Reliability

Workshop 99, CTU, Prague, 1999, 41, 42[6]... Nilsson T., Nelson T.M., Carlson D.: Development of fatigue symptoms during simulated

driving Accid Anal Prev , 1997 July, 29(4), p. 479-488[7]... Marottoli R.A., Richardson E.D., Stowe M.H., Miller E.G., Brass L.M., Cooney L.M. Jr,

Tinetti M.E.: Development of a test battery to identify older drivers at risk for self-reported adverse driving events

J. Am. Geriatr. Soc.,1998 May, 46(5), p. 562-568[8]... Arnold P.K., Hartley L.R., Corry A., Hochstadt D., Penna F., Feyer A.M.: Hours of work,

and perceptions of fatigue among truck drivers Accid, Anal, Prev, 1997 July, 29 (4), p.471-477[9]… Novák M.: Theory of System Tolerances (in Czech), Academia, Prague, 1987

Control functions in micro-sleepControl functions in micro-sleep

How the EEG signals can be How the EEG signals can be measured ?measured ?

 Thalamo – cortical oscillation consists of extremly

complicated concert of very many almost parallel paths of electrical impulses travelling in membranes of axons connecting cortical and thalamic neurons.

These impulses have the potential of about 70 mV. However the resulting electromagnetic fiels is projected on the head surface with voltages about 10 – 30 V only.

These electric waves are of quasi-periodical and quasi-stationary character and its main spectral components appears in the range from 1 to 30 Hz.

Faber glasses, schemeFaber glasses, scheme

Faber glasses, laboratory modelFaber glasses, laboratory model

EEG mapsEEG maps

Attention curveAttention curve

Idealized EEG spectrumIdealized EEG spectrum

RT versus P of wrong responseRT versus P of wrong response

RT versus alpha, deltaRT versus alpha, delta

How the micro-sleep can be How the micro-sleep can be detected?detected?

By measuring of some significant human operator parameters (so called “markers”), e.g. breath temperature, eye movement, electro-cardiographic signals etc.

Among them the electroencephalographic (EEG) signals are most iportant.Why?EEG signals are the only marker which is immediately, primary producted my the

brain activity influencing vigility and sleep. All other markers concern the secondary factors.

EEG signals are produced immediately by thalamo-cortical oscillations in brain structure.