Intelligent SensorBased Road Vehicle DriverAssistance · There are two basic classes of sub-taskswhich make up the intended research -those which are concerned with ... transmitting

Intelligent Sensor Based Road Vehicle Driver AssistanceRay Jarvis

Intelligent Robotics Research CentreMonash UniversityWellington Road

Clayton, Victoria, 3800Australia

AbstractThis paper concerns research towards theprovision of a rich set of road vehicle driverassistance modes to reduce the stress and improvethe safety of negotiating both on-road and off­road terrain. These modes include navigationplanning, potential collision warnings, blind spotand rear traffic monitoring, backing assistance,drowsiness detection and night visionenhancement. The approach taken is partiallyrelated to adapting instrumentation fromautonomous and teleoperated robot navigationresearch; other aspects relate to human factorsconsiderations. Progress to date is reported indetail.

1. IntroductionRecent robotics conference and journal publications in thedomain of automated guided road vehicles [Thorpe,1997;Kimoto and Thorpe, 1997; Dickmans, 1993] wouldindicate that the current state of available technology is at asufficient level to support, in principle, fully autonomousnavigation, but perhaps not yet with sufficient reliability(in unpredictable circumstances) to overcome safetyrelated legal barriers to widespread acceptance, at least inthe realm of the public road systems. A practicalalternative, at least into the foreseeable future, is to retainthe human driver as the primary intelligent, controllingagent but to provide as much assistance as is reasonablypossible in the form of route planning facilities, warningsystems for potential collision or excessive speed,detection of the driver's diminishing ability to drive safely(perhaps related fatigue, drowsiness and/or alcohol or drugconsumption) and perhaps other supporting capabilitiessuch as automatic windscreen wiper and head lightoperations.

There are two basic classes of sub-tasks which makeup the intended research - those which are concerned withcritical and immediate safety issues and those which havethe potential to reduce stress in a way which mayindirectly improve safety, but would certainly makedriving more convenient and enjoyable and perhaps moreeconomical.

The first class requires real-time and reliably correctresponses to situations of imminent danger in a fonn whichpermits driver correction without panic and yet does notallow the driver to develop a false sense of security which

173

may lead to negligence. This class includes warning ofpotential or imminent collision from any direction butparticularly in relation to safe distance to a vehicle orobstacle in front, indication of the approach of anovertaking vehicle from behind or the side (particularly inthe door column 'blind spot' area) and alerting the driver toinappropriate steering and/or acceleration/breakingbehaviour that may be associated with sickness, druggedstate, inebriation or tiredness.

The second main class includes provision of trackingand mapping data along with optimal route planning and .spoken directions. These provisions would not normallybe regarded as urgent or critical but could still be ofparticular importance if they softened an otherwisepotentially stressful situation.

Drowsiness detection, whether it be by monitoring thefrequency and duration of eye closure, head nodding orbody slumping bridges the two classes, since early'detection could defuse a potentially critical situation but isnot as urgent as actual imminent collision warnings.

Perhaps the simplest way of providing details of theresearch is to first deal with conceptual sensor andmethodological issues and then with systemimplementations, operational issues and outcomes to date.

2. Concepts, Sensors and MethodologiesGlobal Positioning Systems (GPS) capable of specifyingthe position, orientation and velocity of a vehicle veryaccurately are commercially available but are currentlyvery expensive. However, relatively inexpensive systemscan provide 50 metres r.m.s. (root mean squared) positionaccuracy fixes which may be sufficient for road navigationadvice. Until recently, an intermediately expensive way ofachieving 10 metres r .m.s. accuracy was to utilise adifferential mode GPS system involving fixed stationstransmitting correction data to mobile units to compensatefor deliberately introduced clock errors (selectiveavailability). However, the recent 'turning off of the clockerror has permitted near differential GPS accuracycapabilities to be achieved with stand alone mobile units.The most accurate results require phase mode analysis ofthe carrier wave signals and is still quite an expensivealternative.

Three or more satellites of the GPS set of 24 (21 inoperation, 3 spares) must be in line-of-sight of the receiverfor a valid position fix. In built up areas, the likelihood ofnot meeting this requirement for short periods is quite

high. For continuity of tracking, backup is required; thiscan most easily be provided by odometry (measurementsof wheel rotation) which can return fairly accurate relativeposition and bearing information for short periods but issubject to accumulative errors over time due to wheelcontact variation, slippage and distortion as well as groundundulations. Combining differential GPS (DGPS) andodometry is an excellent solution to this problem. InertialNavigation Systems(INS) provide an alternative, but moreexpensive, back-up methodology.

Whilst GPS can retul11 bearing and velocity as well asposition it can only do so with acceptable accuracy whenthe vehicle is moving smoothly. To determine thedirection in which a robotic vehicle is pointing when it isstationary, a flux gate compass can be used. On land suchan instrument can be seriously perturbed throughdistortions of magnetic flux caused by ferric materials insurrounding structures, including those underground. Onthe other hand, an optical gyroscope of modest expensecan measure rotational velocity and hence turning extent(through integration) but is subject to drift over time;combining flux-gate-compass and optical gyroscopebearing determinations offers a solution to reliable bearingdetermination which is functionally similar to the

, GPS/odometry combination for position determination.Of course, differential odometry is also a source of

relative change of direction and thus provides anothermeans of bearing determination, but one which, like theuse of an optical gyroscope, suffers from accumulativeerrors over time. However, in driver assistance terms,bearing determination is not a critical issue whilst positiondetermination is as far as map route guidance is concerned.

Since many commercial GPS based car navigationsystems with detailed directions for the driver alreadyexist, there seems little point in replicating such. systems.aspart of this project. Thus, while such systems will beintegrated with other driver assistance components, themain emphasis will be on the other aspects such asimminent collision detection, hazardous driving indicationand drowsiness detection. Both open country off-road andon-road GPS systems will be incorporated, however, sincethese modes differ in some respects.

For potential collision detection, both passive andactive rangefinding are of value. For this project, ascanning active (time-of-flight) range finder capable ofproviding a set of range values at 1/2 degree intervals overa 180 degree sweep in· one second with an accuracy ofseveral centimetres up to a maximum range of 50 metres isused. Another source of range data will be provided bypassive stereopsis. Whilst it is possible to consider 3600

ranging, the forward component is most critical; thusactive and passive rangefinder systems are best deployedwith a forward aspect.

Erratic driving behaviour can be easily detected byinstrumenting the acceleration, braking and steeringmechanical systems with the appropriate linear androtational movement encoders, and setting up, throughexperimentation, tolerance limits for combinations of suchmovements outside of which unacceptably erratic (hencedangerous) behaviour is clearly evidenced. The attractionof this approach is that unacceptable and dangerousbehaviour can be detected objectively, whatever the cause

174

be illness, inebriation, drug ingestation, drowsiness ormechanical malfunction.

Relating measurable sensor quantities to danger of. collision is a crucial aspect of the work and involves

considerable conceptual and experimental development. Itwould seem best to avoid, if possible, subjective aspectssuch as mental state of the driver under various conditions,and concentrate more on the physics of the situation withregards to vehicle and driver behaviour in a variety of roadand off-road environments and the measurable controlactions taken by the driver.

Drowsiness detection is perhaps the most valuabledriver assistance component, especially for long solojourneys, since it is a preventative approach to correct anunacc.eptable driving behaviour which is known to be thecause ofmany fatalities and other serious accidents.

Whilst alcohol and/or drug consumption in relation todriving are recognised criminal activities, drowsiness whendriving does not incur the same societal outrage and thusmany respectable people indulge, knowingly or otherwise,in this dangerous activity. It is also the casethat the onsetof drowsiness may· not be consciously noted andappropriate action taken or that over-confidence lulls anindividual into a false sense of safety. In any case, areliable and unobtrusive/non-invasive means of earlydetection of drowsiness would be of considerabIeassistance to the average driver.

Placing targets on the forehead or eyelids of a driveror the requirement that the driver wear special glasses or aharness would allow a simple but probably unacceptabledrowsiness detection scheme to be devised. The challengeis to come up with more passive means of detection. It isintended that a number of alternative schemes, includingvisual monitoring of the driver's· eyes and face and otherparts of the body as well as fitted mechanical systemswould be researched. The early detection of eye closureand eyelid droop patterns, including frequency andduration of relevant movements by analysis of video datastreams is one of the more interesting approaches worthinvestigating in detail. Ultrasonic detection of headmovement from behind may well be another.

An alternative approach, which in some sense isinvasive but does not rely on the driver volunteering to donspecial equipment, is to require that certain alertness testsbe carried out at regular intervals. Failure of these testscould trigger graceful slowing down of and eventuallydisabling of the vehicle. Such a system could become astandard manufacturer's built-in feature of a modernvehicle.

When it comes to alerting a driver of impendingdanger or giving advice on appropriate routes etc. thequality and style of the reportage is very important. Whatis to be favoured is a informative mode, with gradations ofurgency apparent, which does not unduly distract thedriver from safely carrying out the appropriate response tothe report. A pleasant voice giving succinct informationwith the appropriate inflexions and volume plusassociated, perhaps partially redundant, visual displays,could provide a uniform way of alerting the driver toimminent danger as well as giving advice on favouredresponses. This research project will include studies ofthis important aspect of driver assistance.

Overall, this research project is intended to result inthe construction of an integrated scheme of driverassistance along the lines indicated above, embodied in aworking demonstration vehicle.

3. Implementation to DateSome of the ideas outlined above have now been tested ina demonstration vehicle [Figure 1]. The vehicle is aMitsubishi four wheel drive van; thus both on-road andoff-road experiments can be undertaken with it. Ageneralised inside view of the vehicle is shown in Figure 2.

Figure 1. Demonstration Vehicle

Figure 2. Inside View

Panoramic Viewing

The dome visible on the rear roof [See figure 3]. of thevehicle houses a .colour camera on pan/tilt mount (DirectedPerception Inc.) which is controlled by a joystick withineasy reach of the left hand of the driver. A ± 1800 pan anda ± 60° tilt are available. The pan/tilt unit can also bedriven under computer control, which option may proveuseful in the longer run. This camera can be used tomonitor· traffic conditions around the vehicle and, inparticular, can watch either for overtaking traffic whichmight ordinarily be.obscured behind 'the blind spot' or reartraffic. At night the camera is excellent for low lightvision. A colour LCD monitor for this camera is providednear the rear vision mirror [See Figure 4]. The driver isthus easily able to compare the limited rear view mirrorview of the traffic with the much more complete viewavailable from the camera.

175

Figure 3. Pan/Tilt Camera Dome

Figure 4. Pan/Tilt Camera View Near Rear Vision Mirror

Ultrasonic Rear Proximity Warning

On the rear of the vehicle, just above the bumper, twoultrasonic range sensors (Bosch) provide close proximitywarnings which can assist when backing and indicatingwhen a following vehicle is too close. Both audio andvisual warnings are provided.

Night Vision Camera

Inside the vehicle a camera with its own infra red lightsource for night vision (Sony) allows night visionenhancement for the driver. This might be particularlyuseful when driving on country roads where crossinganimals might prove a hazard..• The same monitor used forthe roof-mounted .camera can be used for night visioncamera viewing when selected.

Laser Rangefinder for Potential and ImminentWarnings

Under the front of the vehicle, just forward from thewheels, a scanning infra red rangefinder (Erwin .Sick) ismounted facing forward [Figure 5(a)]. It can return 360readings up to 50 metres at 1/2° intervals over 180°, asmentioned earlier. A down blast of air is provided in frontof a slit in its protective (from rocks etc.) enclosure ~o

deflect water droplets when driving on wet roads. ThISfacility has not yet been fully trialled as to effectiveness.

The rangefinder data is acquired by a Silicon Graphicsworkstation (02) on a serial line at 9600 baud and displayedon the workstation monitor [Figure 5(b)]. The viewingwindow on the monitor can be adjusted to correspond tothe frame of a video output facility provided, enabling therangefinder data to be viewed on a TV monitor mountedon the front ledge of the vehicle in a convenient locationfor the driver to view it without taking his/her eyes off theroad [See Figure 6]. Nine polygon sectors can beinteractively specified on the range data screen using themouse; these can be saved for recalling when needed.Intrusion into each sector is indicated on a disk of colouron the screen. .There are three red, three orange and threeyellow disks. The yellow disks would ordinarily be usedfor weak warnings for left, centre and right areas; theorange for medium level warnings and the red for criticalwarnings. The sizes of the disks can be adjusted at will.The inclusion of audio warning signals associated with thenine disks has been c~!Jsjdered but not et im lemented.

Figure 5.(a). Erwin Sick Rangefinder

Figure 5(b). Monitor View ofRange Data ProtectionZones and Warning Disks

176

Figure 6. Range and CARIN Navigation System Monitors

Passive Rangefinder 3D Monitoring

Commercial systems providing tear frame ratedisparity (inversre range) maps by applying stereopsismatching principles to the image data from two or morecameras are now readily available at modest cost. Sincethe laser rangefinder described in the previous sectionscans only in one place (in this cas.e horizontally forward)obstacles not in that plane (eg. overhanging tree branchesand constructions) remain undetected .and thus not warnedagainst. Using a stereopsis based area rather than planarscan could perhaps assist in these circumstances. TheTriclops three camera stereopsis system (Point GreyResearch) includes a demonstration program in whichimage details of all objects detected closer than somespecified distance can be shown with the remainder of thescene blacked out. It is intended that experiments becarried out with a Triclops system facing forward frominside the vehicle to see whether this capability is of anypractical use for obstacle detection. Unfortunately, thecamera base line distances are so small that accurateranging beyond several metres may not be possible andthus obstacles not detected by the laser rangefinder mightbe detected only too late by the passive rangefinder. Ifthese ~xperiments show promise, however, automaticalarms based on passive stereopsis can be easily built.

Map Based Navigation System

A comprehensive road navigation system (PhillipsCarin 520) has been provided to guide the driver alongfavourable routes to nominated road junction destinations.The system is map based using CD data covering relativelylarge areas of metropolitan and suburban regions, separateCDs for each capital city covered (in the Australiansystem). Map data can be viewed at a variety of scalesfrom tens of kilometres to hundreds of metres across thevideo' screen provided.

In addition to a map based route-planning capability,the system uses GPS, odometry and gyroscope data as asensor fusion mix to provide a highly reliable vehicletracking facility. The GPS provides a rough fix ofposition; this is refined using odometry and gyroscope data(both prone to accumulated error) which provides themeans of discovering a match in the map data in thevicinity of the GPS fix. Once tracking commences themap, odometry and gyroscope data can m~in~ain ve~icle

tracking even when GPS outages occur for lImIted perIods,

as in tunnels and in built-up areas. The system has' provenremarkably reliable and easy to use. At traffic hold-upsalternative routes can be requested. An even, soothingvoice provides advice on routing details including whenturns are to be anticipated, which lane to take and whichround-about exit to use. If an overshoot error is made bythe driver the voice will suggest a V-turn when possibleand will eventually suggest alternative ways to get back ontrack.

Ignoring advice simply leads to voice directions beingadapted to the current position of the vehicle and the mostappropriate road plan to reach the specified location fromthat position. Thus, if a home destination is provided, onecan explore unknown areas as far away as one likes, justignoring the instructions (or turning the voice off) untilready to go home, all the time viewing the tracking andmap data. Not being able to get lost takes a lot of stressout of driving in an unfamiliar place.

For off-road excursions a hard copy map GPS system(Azimuth/Pointer 2) can be used. This provides locationdata as a plotter position over the map but does not givepath planning advice. This type of planning could beprovided using methodologies devised earlier by the author[Jarvis, 1995] but the need for this is questionable.

As mentioned earlier, the removal of the deliberateclock error (which was previously introduced for securityreasons) now permits quite accurate GPS data (10 metresLm.s.) from a stand alone unit. This is particularlyvaluable for outback off-road navigation since stationaryGPS stations for differential mode support are generallylocated on coastlines to provide maritime navigationservice.

Drowsiness Detection

It was always accepted that drowsiness detection,whilst of utmost value to drivers, since many fatalaccidents (particularly on country roads or otherwisedeserted roads) can be attributed to fatigue and drowsiness(whether long activity, stress or substance abuse related),would be difficult to instrument if intrusive methods (suchas wearing special harnesses or glasses etc.) were to beavoided. Methods that came to mind includedmeasurement of skin resistance, hand pressure on thesteering wheel, steering wheel rotation patterns, brainwave monitoring, blink rate and dwell measurements, andposture change detection.

Some strong progress has been made in gaze directiondetection [Heinzmann and Zelinsky, 1998] and the authoris certainly keen to experiment with such systems in duecourse. Some of the other approaches are also worthy ofcareful assessment.

The notion of relating driver head posture to fatigueand drowsiness started to take shape as a relatively untriedidea worth following up, particularly if novel yet reliableand relatively unobtrusive instrumentation could providethe necessary data. At first it seemed as if monitoring theforward/backward movement of the drivers head using anultrasonic distance measuring device would be a goodapproach. On second thoughts such a system might provetoo simplistic, as only one measure would be the basis ofjudging whether drowsiness was setting in. After all,discovering a sleeping driver is not as useful as a warning

177

about that possibility in the near future.In a fit of minor inspiration the author thought of

usihg a Polehmus 6 degree of freedom tracker [See figure7] which uses electromagnetic fields to determine thelocation and pose of a small receiver (approximately 1cubic em in size) relative to the transmitter (approximately4x3x3 cm). This device is well known in the VirtualReality community for hand and head tracking.

Figure 7. Polhemus 6 Degree of Freedom Tracker

Attaching the receiver to a cloth cap worn by thesubject allowed some preliminary experiments to becarried out to determine whether drowsiness detection viahead pose variations could indeed be reliably detectedusing this instrument. If the experiments were promising(as they turned out to be) it was felt the receiver could besuspended behind the driver so that head contact against aslight elastic resistance would 'attach' the device to thedriver's head without much intrusion and littleinconvenience.The following experiment was carried out:1. For a medium length period (say 50 seconds), collect

all six degrees of head pose data over a range of smallmovements which are consistent with alert driving.

2 . Analyse the data arrays and calculate the mean andstandard deviation of each of the six degrees offreedom parameters (X, Y, Z, Yaw, Pitch, Roll).

3. Over a shorter period (say 10 seconds) collect thesame data. Again calculate the mean values.

4. Issue a 'doze warning' for any occurrence of a meanvalue from 3, above, differing by more than 3(arbitrary) standard deviations from the meanscalculated in 2, above.

5. Return to 3 and continuously repeat stages 3 and 4.It was found that slouching, peering forward, drooping thehead, deliberate stretching of the neck or movingunnaturally sideways all caused warnings that seemed tobe appropriate for the experiments carried out so far. The

elegan~ thing about this approach is that the 'calibration'phase of 2, above, will provide individual data for eachdriver and for each driving session. Assuming that thedriver is alert for 50 seconds at the beginning of a driveseems reasonable. More complex analysis can of coursebe included, such as tracking patterns of movementcharacteristics of alert and drowsy driving and makingcomparisons between them. This has not yet been done,but the first simple experiments have been very promisingand the author is convinced this is a good way to go on thequestion of drowsiness detection.

4. DiscussionThe pan/tilt controlled camera, the road navigation system,the laser rangefinder front collision warning and theultrasonic sensor rear collision warning systems have beentrialled extensively over the last year or so and haveproven to be both reliable and useful. The night visionenhancement system has only just been implemented butits usefulness is in some doubt. The drowsiness detectionsystem based on head pose variations is likely to be veryeffective but is difficult to trial in realistic situationswithout endangering the driver. Some laboratory basedexperimental test will need to be devised to properlymeasure the reliability of the system to actually testdrowsiness without a high occurrence of false alarmsrather than detect movements a subject might simply thinkcan be associated with drowsiness.

Instrumenting the steering wheel to detect· erratic usemay be of value in detecting dangerous driving behaviourthat might not be sleepiness related. Vehicle velocity and.acceleration monitoring could also assist. However, thesekinds of measurement have not yet been tried. Checking ifhead pose is consistent with steering direction may beanother way of detecting unacceptable driving behaviour,particularly that associated with steering drift.

Devising appropriate warning modes with a mixture ofaudio, visual and tactile cues without introducing eitherconfusion or distraction side effects will be a challenge inits own right. Another fascinating aspect of this work isthe realisation that increased human laxity in reaction tosafety device availability may neutralise the potentialgains. Coming to grips with this issue in a reliable waywill take some creative effort and good design andimplementation.

5. ConclusionsThis paper has described a number of modes of assistanceto a vehicle driver, which would either directly improve

178

driver safety or at least reduce stress and fatigue, which inturn might indirectly increase safety. The most criticaldirect safety features relate to imminent front collisionwarnings and drowsiness detection. However, mappingand navigation support plus panoramic viewing, nightvision enhancement and rear collision warnings are allcontributors· to. safe driving and stress reduced means ofgetting from one place to another or simply exploringunknown territory. The research so far has been promisingbut more work has yet to be done to reconciletechnological advances with human factor considerations.Cost, to date, has not been a serious consideration sinceconcept testing has been emphasised rather than marketpotential.

B,orrowing ideas and instrumentation from theresearch ~omain of intelligent robotics and especiallyautonomous and teleoperated robot navigation has been away off achieving a running start in this research.

References[Thorpe, 1997] C. Thorpe. Mixed Traffic and Automated

Highways, .Froc. International Conference on IntelligentRobots and Systems, Sept. 7-11, 1997, Grenoble, France,pp. 1011-1017.

[Kimoto and Thorpe, 1997] K. Kimoto and C. Thorpe.Map Building with Radar and Motion Sensors forAutomated Highway Vehicle Navigation, Proc.International Conference on Intelligent Robotics andSystems, Sept. 7-11, 1997, Grenoble, France, pp. 1721­1728.

[Dickmans, 1993] E. Dickmans. Bilocal Dynamic Visionfor Vehicle Control, The Sixth International Symposiumon Robotics Research post-conference proceedings~edited by Kanade, K. and Paul, R.), Oct. 2-5, 1993,Hidden Valley, Pennsylvania, U.S.A. pp. 125-133.

[Jarvis, 1994] R.A. Jarvis, On Distance Transform BasedCollision-Free Path Planning for Robot Navigation inKnown, Unknown and Time-Varying Environments,invited chapter for a book entitled 'Advanced MobileRobots' edited by Professor Yuan F. Zang WorldScientific Publishing Co. Pty. Ltd. 1994, pp. 3-31.

[Heinzmann and Zelinsky, 1998] J. Heinzmann .and A.Zelinsky. 3-D· Facial Pose and Gaze Point EstimationUsing a R.obust R.eal - Time Tracking Paradigm, Proc. ofthe International Conference on Automatic Face andGesture Recognition, 1998, pp. 142-147.