Planning of a field operational test on navigation systems: Implementation and evaluation of pre-studies providing background information for the data analysis, Barbara Metz, Stefanie

Folie 1European Conference on Human Centred Design for Intelligent Transport SystemsBerlin, April 29-30 2010

Planning of a field operational test on navigation systems:Planning of a field operational test on navigation systems:

Implementation and evaluation of pre-studies providing Implementation and evaluation of pre-studies providing

background information for the data analysisbackground information for the data analysis

Barbara Metz, Stefanie Schoch, Mariana Rakic & Friedemann Kuhn


BackgroundBackground

Presented research is part of the EC-funded project euroFOT:

In 2010 a field operational test will take place with the aim to evaluate the effect and usage of navigation systems

Driving without navigation system, with navigation systems provided by BMW & Daimler and with a mobile navigation system will be compared

General aim of euroFOT: Assess impact of investigated driver assistance systems on driving on a EU-level for a representative sample.

Two pre-studies will be presented which provide background information for planning and implementing the FOT-data analysis.


Study 1: Questionnaire surveyStudy 1: Questionnaire survey

Development of short questionnaire (5-10 minutes) Usage and evaluation of navigation

system Frequency of driving & unfamiliar routes Other driving relevant background

information

Conduction Nov / Dec 2008 and June / July 2009

Sample: N = 147 drivers at highway service area N = 95 drivers at parking lot in front of

supermarket 30 % female, mean age 47 year (sd = 16.5)


Study 1: ResultsStudy 1: Results

Of sample, 56% use / own navigation systems

Of users, 63% use a mobile device, 36% a built-in navigation system

There are relevant differences between owners and non-owners:– Owners of navigation systems have higher annual mileage– Owners of navigation systems drive more often on unfamiliar routes– Owners of navigation systems rate both system functions as more useful


Study 1: Relevance for the FOTStudy 1: Relevance for the FOT

Conduction of the FOT:

To enhance the likelihood of system effects, a sample should be recruited with high mileage drivers– Effect of navigation system probably

mostly on unfamiliar trips

– With higher mileage the likelihood of unfamiliar trips rises

– Low mileage drivers only rarely experience the most relevant use case (unfamiliar route) km[last 12 months]

% u

nfa

mili

ar

trip

s

0 20000 40000 60000 80000 100000

0

20

40

60

80

100

r = 0.4371; p = 0.0000

no navigation systemowners of navigation system

Analysis of the FOT-data (especially up-scaling of results for larger sample):

Results from FOT-sample can not be transferred linearly to larger sample For up-scaling, the likelihood of unfamiliar trips has to be considered


Study 2: Aims of the experimentStudy 2: Aims of the experiment

Part of pilot activities in euroFOT:

Test of data logging

Implementation of data base

Implementation of data analysis– Implementation and evaluation of algorithms for calculating derived measures

(e.g. to detect certain events)

Evaluation of planned data analysis– Can the planned analysis differentiate between driving with and without navigation

system?


Study 2: Experimental designStudy 2: Experimental design

Experiment with Daimler prototype vehicle for euroFOT Data logging is comparable to FOT

data Logging of CAN-data, GPS-

position and video data

Experimental design: In real traffic Defined circular route with N=10

different destinations Drivers are not familiar with the

destinations N = 10 drivers, 5 with navigation

system and 5 without navigation system

Experimenter protocols relevant events (e.g. turnarounds) during the drive

Duration 4-5 hours


Study 2: RouteStudy 2: Route


Study 2: Implementing derived variablesStudy 2: Implementing derived variables

For analysis of FOT-data, event detection algorithms are needed

Automated search for relevant events If possible, no extensive control through video analysis necessary

Example: Turnaround manoeuvre

Manoeuvre, that directly relates to way finding performance The experimenter protocolled turnarounds during the drive In the CAN-data a turnaround is coded whenever reversing and steering

wheel angle > 90° occur together within 60 seconds


Study 2: Comparing protocols & event Study 2: Comparing protocols & event detection algorithmdetection algorithm

wi th o u t wi th0 .2

0 .4

0 .6

0 .8

1 .0

1 .2

1 .4

1 .6

1 .8

me

an

(N

Tu

rnin

g p

roto

co

lle

d)

w i th o u t wi th0 .2

0 .4

0 .6

0 .8

1 .0

1 .2

1 .4

1 .6

1 .8

me

an

(N

Tu

rnin

g m

ea

su

red)

(Z=1.98, p<0.05) (Z=1.46 p=0.144)


Study 2: Analysis of errors in event Study 2: Analysis of errors in event detectiondetection

Missed events (based on protocols):

Turning e.g. at intersections where no reversing is necessary

Can not be differentiated based on CAN-data from turning off at intersections

False positive events (based on analysis of video data):

62 % of detected events are turnarounds

False positive are mostly parking manoeuvres (33%)T u rn in g m ea s u red - T u rn in g pro to c o l le d [N]

N D

est

ina

tion

s [N

tota

l = 9

7]

-3 -2 -1 0 1 2 30

5

1 0

1 5

2 0

2 5

3 0

3 5

wi th o u twi th


Study 2: Improvement of algorithmStudy 2: Improvement of algorithm

The evaluation of detected events helps to find several weakness of the used algorithm. For example:

Used algorithm was simple, but lead to events that endured about 60 seconds– Coded time window is too long (in 71% of events turning at next intersection is included)

2260 2280 2300 2320 2340

Time [ms]

-400

-200

0

200

400

stee

ring

angl

e [°

]

pre-turn

turnaroundreversing

post-turn

steering wheel(L) vehicle state(R) Turn around(R)


Study 2: Relevance for FOTStudy 2: Relevance for FOT

Experimental approach has several advantages for implementing the analysis for an FOT:

Because of the protocols, the algorithms for event detection can be evaluated regarding false positive and missed events

Weaknesses of the algorithms can be detected and improved In case no improvement is possible, the impact of errors can be

described for the data from the experiment The planned analysis can be tested regarding its ability to detect effects

of the tested system The power of an purely automated analysis can be compared to an

analysis relying on expert rating


ConclusionConclusion

It has been tried to combine the expensive approach of an FOT with less costly methods:

A final evaluation of the contribution of the pre-studies is not possible before the analysis of the FOT-data.

Questionnaire survey provides information for a larger sample

Relevant for sample recruitment and planned up-scaling of results

Experiment provides well described and controlled data

Relevant for implementing and testing the analysis


Thank you for your attention!

For further information please [email protected]

Planning of a field operational test on navigation systems: Implementation and evaluation of pre-studies providing background information for the data analysis, Barbara Metz, Stefanie

Documents

fotdata analysis

mobile navigation system

builtin navigation system

usage of navigation

navigation systems of

analysis of fot

eurofot data logging

data analysis barbara