Session 48 Johan Karlsson

ALR-JKAR/Jan2011/Transportforum - 1 Copyright Autoliv Inc., All Rights Reserved

Fordonsstrategisk Forskning och Innovation FFI – D4SF

Driver Drowsiness and Distraction Detection bySensor Fusion

D4SFJohan Karlsson, AutolivTransportforum 2011



Overview

�Background�Goals

� Drowsiness detection� (Distraction detection)

�Method� Data collection� Training/optimization of classifier� Sensor fusion

�Results� Reference – ground truth� Improvement by (f)using parallel detectors



Driver Drowsiness detection

�Drowsy driving is a road safety problem- drowsiness contributing in 10-30% of accidents (Anund & Patten 2010)

�What can be done?� Commercial fleet traffic

� Fatigue Risk Management� Work time regulation� Detection and warning

� Privately owned vehicles� Detection and warning

�Detection?� Detection systems offered as option from several OEMs� So far, performance is far from ideal...



Target and Goals

�Different indicators exist- ’Physiology’ measures - blink duration etc. - Driving performance measures - lane keeping measures- Environment measures - time of day, traffic, road type

(previous sleep possible in commercial fleet vehicles??)

� Various indicators have different strengths and weaknesses

�Improve performance by fusing data from multiple indicators

� The fusion algorithm shall show an improvement in:- Overall performance- Reduced number of faulty detections- Increased number of correct detections

+ +

+

+

–

Specificity

–+Indicator C

+ ++ +Fusion

+–Indicator B

++Indicator A

AvailabilitySensitivity



� Data collection

� Relevant vehicle data

� Speed, lane position, SW angle, pedals etc.

� Video based gaze direction, eyelid opening, head pos

� KSS value every 5 minute

� EEG, EOG and EMG

� Video recordings (road scenery and cabin)

� In total: 43 drivers have completed 3 drives each

� Procedure: Each driver drove three times during one day (day, evening and night). Trip duration 80-90 minutes

Data collection�On-road tests were conducted with governmental approval (N2007/5326/TR) and ethical approval by Regional ethics approval board (EPN 142-07 T34-09).



Test Route

Road RV34Mostly 9 m width Driving lane width 3,75 m Speed limit - mostly 90 km/h

Numbers on map are Yearly day traffic volume in January 2002

We know of only a few similar studies performed on public roads

90 minute driving, approx 115 km distance

Rested safety driver –dual command



Ground Truth – KSS

+ Simple to collect+ Simple to understand – immediately ready for analysis

- Training needed for participants- Some offset for inexperienced participants?

KSS Description in Swedish Verbal description

1 extremt pigg extremely alert

2 mycket pigg very alert

3 pigg alert

4 ganska pigg rather alert

5 varken pigg eller sömnig neither alert nor sleepy

6 första tecknen på sömnighet some signs of sleepiness

7 sömnig, ej ansträngande vara vaken sleepy, but no effort to keep alert

8 sömnig, viss ansträngning vara vaken sleepy, some effort to keep alert

9 mycket sömnig, ansträngande vara vaken, kämpar mot sömnen

very sleepy, great effort to keep alert, fighting sleep



Blink duration (AS): Mean blink duration

Lane keeping variability (Lane): Variability in Steering wheel position or Lane Position. e.g. using Generic Variability Indicator (Sandberg 2008) .

Time-of-day (TPM): Expected drowsiness with regard to time of day (circadian rythm)

* Each indicators has several parameters that needs to be tuned for optimal performance

))1(( pxxz ii δδ −+−=

ki

N

ii zzw

NG ||)(

1

1∑

=

=

)()( 11)(

RRLL zR

zL

e

c

e

czw βαβα −−−− +

++

=

GVI (Sandberg 2008)

Short Blink Long Blink

400 ms

200 msOpen

Closed

OpeningClosed

Closing

Amplitude

Example indicators of driver sleepiness



Video examples

Video examples



� SVM (Support Vector Machine):

� Machine learning method using data from field tests to calculate “best fit” function between indicator values and ground truth (KSS rating scale)

� Indicator parameters optimized simultaneously with training of SVM

� Data sets for SVM training and validation are from separate drivers. Thus, validation is done on truly “never-before-seen” data.

Sensor fusion

Indicator A

Indi

cato

r B

Drowsy data

Alert data

Goal: Find the maximum margin hyperplane



Evaluation Method� Assuming a binary classification,

alert or drowsy� Performance is the mean value of

sensitivity and specificity� Performance is related to the

proportion of the time where the algorithm is correct

DrowsyNon-

Drowsy

Detect A(hit)

B(false hit)

Non-Detect

C(miss)

D(correct reject)

Sum A + C B + D

2

yspecificitysensitiviteperformanc

DB

Dyspecificit

CA

Aysensitivit

+=

+=

+=

KS

S

Ground truth

Alg

orith

m o

utpu

t

KSS = ground truth

Binary Algo output

Ground truth cutoff

Time



Example of results from sensor fusion

0.84 (0.89)0.76 (0.81)0.80 (0.85)Blink + Steer + Circ.

0.92 (0.88)0.68 (0.68)0.80 (0.78)Blink + Lane + Circ.

0.96 (0.95)0.36 (0.32)0.66 (0.64)Blink

0.83 (0.87)0.77 (0.79)0.80 (0.83)Blink + Circadian

SpecSensFitnessModel

First figure is training data performancesecond figure is test data performance � Decision every 1 minute

� KSS >= 7 � drowsy� KSS < 7 � alert



Fulfillment of goals

� The fusion algorithm shall show an improvement in:

- Improved performance true

- Increased number of correct detections true

- Reduced number of faulty detections (?)

Clearly improved overall performance

– Minor differences between different combinations



Summary

�Controlled experiment on public roads� 43 drivers so far�What is ideal performance?

� Method developed with focus on mathematical performance� Most important goal is to have relevant warnings

�More data is needed: � Different road types� Different conditions (weather, drive duration etc.)� Different driver types (age, cultural differences etc.)



Thank you for you attention!

Session 48 Johan Karlsson

Education

copyright autoliv

day day

autoliv transportforum

time of day

performance measures

variability lane

road tests

training of svm data