15/05/2014 Information and Communications Technologies Policy Support Programme (the “ICT PSP”) Information Society and Media Directorate-General Grant agreement no.: 270906 Pilot type A D4.3 Intermediate results of the tests Version number: Version 1.3 Main author: NavCert GmbH, Germany Dissemination level: PU Lead contractor: ERTICO – ITS Europe Due date:
213
Embed
D v. u Intermediate results of the tests - HeERO · D v. u Intermediate results of the tests Version number: ... 7.8.2 METHODOLOGY OF DATA ANALYSIS ... 7.9.5 RESULTS DRIVE TEST ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
15/05/2014
Information and Communications Technologies Policy Support Programme (the “ICT PSP”) Information Society and Media Directorate-General Grant agreement no.: 270906 Pilot type A
D4.3 Intermediate results of the tests
Version number: Version 1.3
Main author: NavCert GmbH, Germany
Dissemination level: PU
Lead contractor: ERTICO – ITS Europe
Due date:
Delivery date:
Delivery date updated document
Page left intentionally blank
D4.3 Intermediate test results
15/05/2014 3 Version 1.3
Control sheet
Version history
Version Date Main author Summary of changes
0.1 05.08.2012 Stefan Götte First draft version
0.2 12.09.2012 Stefan Götte internal input
0.3 30.09.2012 Stefan Götte after review
0.4 15.10.2012 Monika Hentschinski Input from Germany
6.4 DATA COLLECTION ................................................................................................................................ 27
DATA COLLECTION PLAN ............................................................................................................... 27 6.4.2
6.5 RECORDING OF RESULTS ....................................................................................................................... 27
6.6 DATA ANALYSIS ..................................................................................................................................... 27
7.3 FINLAND ................................................................................................................................................ 85
7.7 ROMANIA ............................................................................................................................................ 133
PURPOSE AND SCOPE .................................................................................................................. 133 7.7.1
STRUCTURE OF DOCUMENT ........................................................................................................ 133 7.7.2
DESCRIPTION OF EQUIPMENT ..................................................................................................... 133 7.7.3
7.8 SWEDEN .............................................................................................................................................. 161
TABLE 21: LIST OF TESTED KPIS (CZ) ...................................................................................................................... 71
TABLE 22: LIST OF KPI RESULTS WITH THEIR STATISTICAL EVALUATION – IVS SHERLOG TRACELL (CZ) ............... 73
TABLE 23: AVERAGE TIMING VALUES OF SIGNIFICANT ECALL PHASES – IVS SHERLOG TRACELL (CZ) .................. 76
TABLE 24: OF KPI RESULTS WITH THEIR STATISTICAL EVALUATION - IVS TELEMATIX (CZ) ................................... 77
TABLE 25: AVERAGE TIMING OF SIGNIFICANT ECALL PHASES- IVS TELEMATIX (CZ) ............................................. 81
TABLE 26: KEY PERFORMANCE INDICATORS PLANNED FOR EVALUATION OF FINNISH ECALL ............................. 91
TABLE 27: KPIS MEASURED IN PHASE 1 IN FINNISH ECALL PILOT (FI) ................................................................... 92
TABLE 28: DESCRIPTION OF TEST ROUTE (FI) ........................................................................................................ 93
TABLE 29: SUMMARY OF TEST RESULTS FOR GECKO IVS PROTOTYPE (FI) ........................................................... 94
TABLE 30: SUMMARY OF TEST RESULTS FOR GECKO IVS PROTOTYPE (FI) ........................................................... 94
TABLE 31: SUMMARY OF ACTIVATED ECALLS, STARTED MSDS AND SUCCESSFUL MSDS, GECKO PROTOTYPE (FI)
Completed internal review of the Draft first-phase WP4 consolidated
results report
15 October, 2012
D4.3 Intermediate test results
15/05/2014 30 Version 1.3
Consolidated and reviewed First-phase WP4 consolidated results
report presented to ERTICO
31 October, 2012
6.9 Overview of KPIs To summarize the definitions done in D4.1, the following table gives an overview about which KPIs
are recommended to be tested by all pilot sites.
ID of KPI Name of KPI
recommended Remark
KPI_001a Number of automatically initiated eCalls
This KPI measures mainly coverage of GSM
network but not eCall specific aspects, as typically
vehicle is driving, no voice communication
KPI_001b Number of manually initiated eCalls
Y Describes number of “real” eCall scenarios with
vehicle not moving and voice communication
KPI_002a Success rate of completed eCalls using 112
Y It is recommended to use eCall flag for call
establishment with 112
KPI_002b Success rate of completed eCalls using long number
Only if eCall via 112 is not possible as eCall flag not
supported in member state or due to other
technical restrictions
KPI_003 Success rate of received MSDs
Y Measures exactly what differs eCall from 112 call
KPI_004 Success rate of correct MSDs
Y Measures proper en-/de-coding of MSD
KPI_005 Duration until MSD is presented in PSAP
Y Measures time until information is available to
operator
KPI_006 Success rate of established voice transmissions
Y Measures basics of eCall, MSD and voice
transmission
KPI_007a Duration of voice channel blocking
Y
Most important to minimize, as during this time
passengers in the vehicle do not know if eCall does
work or not
KPI_007b
Duration of voice channel blocking: automatic retransmission of MSD
If for some reason the transmission of MSD was
not successful, operator may require
retransmission of MSD, but voice communication
has been established already
KPI_008 Time for call establishment
Allows calibration between timer in PSAP and IVS
if for some KPIs only one and not the other is
D4.3 Intermediate test results
15/05/2014 31 Version 1.3
available; if all timers are available not required
KPI_009 Accuracy of position
Provides accuracy information of IVS system, This
will mainly be influenced by the later integration
into the vehicle and as such not high priority for
pilot phase
KPI_010 Number of usable satellites
Provides additional information on good or bad
accuracy
KPI_011 Geometric dilution of precision
Provides additional information on good or bad
accuracy
KPI_012 Time between successful positioning fixes
Typically within IVS not varying too much
KPI_013 Success rate of heading information
Y This value is calculated by IVS and is critical to
identify right side on highways
KPI_014 Success rate of VIN decoding without EUCARIS
It is critical that VIN is properly encoded and
transmitted via MSD. VIN decoding is not eCall
specific
KPI_015 Success rate of VIN decoding with EUCARIS
It is critical that VIN is properly encoded and
transmitted via MSD. EUCARIS is not eCall specific
KPI_016 Time for VIN decoding with EUCARIS
It is critical that VIN is properly encoded and
transmitted via MSD. EUCARIS is not eCall specific
KPI_017 Dispatch time of incident data to rescue forces
Typically this value might not be available as no
real PSAP is used but only “test PSAP”. In case of
unique alarms not enough values for statistical
analysis
KPI_018 Time to activate rescue forces transmissions
Typically this value might not be available as no
real PSAP is used but only “test PSAP”. In case of
unique alarms not enough values for statistical
analysis.
KPI_019 Dispatch time of incident data to TMC
Potential value added Service for eCall but critical
part is more when to provide information
KPI_020 Success rate of presented incident data in TMC
Potential value added Service for eCall but critical
part is more when to provide information
KPI_021 Number of
In case of errors, operator may request call back to
D4.3 Intermediate test results
15/05/2014 32 Version 1.3
successful call-backs vehicle; mainly question of proper
implementation of dormant eCalls (validated by
certification)
KPI_022 Success rate of call-backs
In case of errors, operator may request call back to
vehicle; mainly question of proper
implementation of dormant eCalls (validated by
certification)
KPI_023 GSM network Latency
Depends on network implementation and no
significance on eCall timers (units are measured in
seconds not fraction of seconds)
KPI_024 112 National network latency
Depends on network implementation and no
significance on eCall timers (units are measured in
seconds not fraction of seconds)
KPI_025 112 Operator reaction time
Depending on loads in PSAP, no influence to eCall
KPI_026
Time for acknowledgement of emergency services
Typically this value might not be available as no
real PSAP is used but only “test PSAP”. In case of
unique alarms not enough values for statistical
analysis
KPI_027 Total response time
Typically this value might not be available as no
real PSAP is used but only “test PSAP”. In case of
unique alarms not enough values for statistical
analysis
KPI_028a Number of cross-border tests
Y
Required tests and should be specified per
member site with which cross border was
performed
KPI_028b Number of interoperability tests
Y
Required tests and should be specified per
member site with which interoperability was
performed
Table 2: recommendations of specific KPIs
In Table 3 the testing and evaluation of recommended KPIs in phase 1 is illustrated. means the Red
KPI was recommended but not tested by this pilot site, green that a recommended KPI was tested
and yellow that this KPI was measured although it was not recommended.
In chapter 7 each Pilot sites gives reasons why recommended KPI hadn’t been evaluated in Phase I.
In the following, this information is consolidated on per KPI basis:
D4.3 Intermediate test results
15/05/2014 33 Version 1.3
HR CZ FI DE GR IT RO SE NL
KPI_001a
KPI_001b
KPI_002a
KPI_002b
KPI_003
KPI_004
KPI_005
KPI_006
KPI_007a
KPI_007b
KPI_008
KPI_009
KPI_010
KPI_011
KPI_012
KPI_013
KPI_014
KPI_015
KPI_016
KPI_017
KPI_018
KPI_019
KPI_020
KPI_021
D4.3 Intermediate test results
15/05/2014 34 Version 1.3
KPI_022
KPI_023
KPI_024
KPI_025
KPI_026
KPI_027
KPI_028
Table 3: Testing of recommended KPIs in phase 1
KPI1b: Not tested by SE. In Sweden nearly all eCalls were computer-generated (only a few were
manually-initiated). The number of computer-initiated eCalls is reported under KPI1a, although the
call was manually initiated. The number of manually-initiated eCalls (KPI_001b) is not reported. Only
a handful manually-initiated eCalls were performed.
KPI2a: Not tested in CZ, FI, DE, GR and NL: In Czech Republic the eCall tests are carried out with the
short number 162 instead of 112. In Finland, Germany, Greece and the Netherlands, the eCall flag
has not been implemented by any of the national mobile network operators and won’t be available
during the HeERO project. Therefore all tests are performed using the long number of the PSAPs.
KPI5: Not tested in HR and FI. The presentation time couldn’t be measured in Finland because no real
PSAP with user interface was available for the tests. Croatia has not planned to test KPI005, since,
given the technology and business environment of the Croatian eCall Pilot test-bed, the very same
results are to be achieved by testing KPI007a.
KPI6: Not tested in FI: At the time of test preparation there was no requirement for implementation
of the voice connection in the two IVS prototypes. Therefore voice connection related KPIs were not
tested in Phase 1. This KPI will be measured in the Phase 2 of HeERO in Finland
KPI7: Not tested in FI, GR and IT: In Finland this KPI will be tested in phase 2 due to the fact that the
implementation of voice connection was not available for the tests in Phase 1. In Italy, KPI 7 is not
measured because on IVSs side it is impossible to acquire low level mobile modem related data. In
phase 2 this might be revised as the KPI may be measured based on PSAP information only as well.
KPI13: Not tested in HR, FI, GR, DE, IT and SE: In Italy, KPI 13 has not been measured mainly because
it was not of interest to check GPS related information with IVSs still in prototype version. In
Germany, heading information was not evaluated during the first test period, but is part of the
second test phase. In Finland the eCall IVS devices were early prototypes and the focus was on eCall
specific new features during the first phase. KPI 13 is planned to be measured in the Phase 2 of
HeERO tests in Finland. In Sweden this KPI is not measured, as there is a vendor specific (Actia)
solution successfully used in Volvo On Call for many years with the evidence of providing correct
information on location. Croatia has not tested KPI013, since, the technology environment of the
D4.3 Intermediate test results
15/05/2014 35 Version 1.3
eCall Pilot test-bed and equipment, the Croatian eCall Pilot has not got the technical and operational
means for reliable testing of the KPI013.
KPI28: not systematically tested in all pilot sites. There were some tests which showed that
interoperability is possible, e.g. at the General Assembly in Zagreb or before the eCall Days in Berlin.
However not enough test cases were executed to measure KPIs during interoperability tests. This will
be done in the second phase by all pilot sites.
The following table gives an overview which part of the eCall-system was evaluated via a KPI in which
country as stated in D4.1 for the pilot sites. As such the overall findings have been derived only from
those pilot sites measuring the specific KPI as indicated below.
The most important KPIs have been measured by nearly all member states. This refers to the number
of established calls, the resulting success rate and the amount of time passed until a direct voice
communication between passenger and operator in the PSAP can take place. All other KPIs have
been measured by less than 50% of the participating pilot sites, so it might be difficult to derive
overall conclusions from the consolidated data valid for all pilot sites. If less than 3 pilot sites have
measured a KPI, this KPI was not consolidated on pan-European level.
ID o
f K
PI
Nam
e o
f K
PI
Member States, where KPI is evaluated
Cro
atia
Cze
ch R
ep
ub
lic
Fin
lan
d
Ger
man
y
Gre
ece
Ital
y
Ro
man
ia
Swe
de
n
The
Ne
the
rlan
ds
KPI_001a Number of automatically initiated eCalls
X X -- X -- 0 X X X
KPI_001b Number of manually initiated eCalls
X -- X X X X X X X
KPI_002a Success rate of completed eCalls using 112
X 0 -- 0 0 X X X 0
KPI_002b Success rate of completed eCalls using long number
X -- X X -- -- 0 (X) X
KPI_003 Success rate of received MSDs
X X X X -- X X X X
KPI_004 Success rate of correct MSDs X X 0 X X X X X X
KPI_005 Duration until MSD is presented in PSAP
-- X -- X X X X (X) X
KPI_006 Success rate of established voice transmissions
X X -- X X X X X X
KPI_007a Duration of voice channel blocking
X -- -- X -- -- X (X) (X)
KPI_007b Duration of voice channel blocking: automatic retransmission of MSD
-- (X) -- -- -- -- 0 0 --
D4.3 Intermediate test results
15/05/2014 36 Version 1.3
X Tested as planned (X) Not Tested in phase 1, committed as “will be tested if possibly”
0 Not tested in Phase 1, eventually scheduled for Phase 2 -- additionally tested
-- No commitment
In chapter 7 the inputs of the pilot sites are listed. For Greece the input had been delivered in February 2014 and had then been integrated in the document. The reason for missing statistical analyses is that only success rates had been tested (FI) or the evaluation was focussed on decreasing some of the observed failures (NL).
Collected
data
Statistical
evaluation
Recommendations
conclusions
KPI_008 Time for call establishment X X -- -- 0 X -- -- X
KPI_009 Accuracy of position 0 -- 0 -- -- -- -- X
KPI_010 Number of usable satellites 0 X -- -- -- -- -- -- --
KPI_011 Geometric dilution of precision
0 X -- -- -- -- -- -- --
KPI_012 Time between successful positioning fixes
0 (X) -- -- -- -- -- -- --
KPI_013 Success rate of heading information
-- -- -- 0 -- -- -- -- X
KPI_014 Success rate of VIN decoding without EUCARIS
X X 0 -- -- -- X -- --
KPI_015 Success rate of VIN decoding with EUCARIS
-- -- -- 0 -- 0 0 -- 0
KPI_016 Time for VIN decoding with EUCARIS
-- -- -- -- -- -- 0 -- 0
KPI_017 Dispatch time of incident data to rescue forces
0 X -- -- -- 0 0 -- --
KPI_018 Time to activate rescue forces
-- 0 -- -- -- -- 0 -- --
KPI_019 Dispatch time of incident data to TMC
-- (X) 0 -- -- -- 0 -- X
KPI_020 Success rate of presented incident data in TMC
-- (X) -- -- -- -- 0 -- X
KPI_021 Number of successful call-backs
-- 0 -- -- -- 0 X -- --
KPI_022 Success rate of call-backs -- 0 -- -- -- 0 X -- 0
KPI_023 GSM network latency -- X -- -- -- -- X -- --
KPI_024 112 National network latency -- (X) -- -- -- -- X -- --
KPI_025 112 Operator reaction time -- -- -- -- -- -- X -- --
KPI_026 Time for acknowledgement of emergency services
-- 0 -- -- -- -- 0 -- --
KPI_027 Total response time -- 0 -- -- -- -- 0 -- --
KPI_028 Number of cross-border tests -- (X) -- -- -- (X) 0 0 --
Table 4: KPIs to be evaluated within the Member States
D4.3 Intermediate test results
15/05/2014 37 Version 1.3
Croatia x x x
Czech x x x
Finland x (x) x
Germany x x x
Greece x (x) x
Italy x x x
Netherlands x (x) x
Romania x (x) x
Sweden x x x
Table 5: Status of received inputs
X Received as planned
(x) Not received in favoured quality or only in part
6.10 Consolidated KPIs
Automatically and manually initiated eCalls 6.10.1
Looking to the KPIs, the most relevant ones are: 02a and 02b, 03, 04, 05, 06, 07a, 08. They provide
information on reliability (success rate – 02a, 02b, 03, 04 and 06), additional waiting time compared
to a normal 112 call (07a), time passed prior to information available to call handler (05) and
comparison to call set up times (08). As the number of tests initiated per pilot site varies quite a lot,
this should be reflected in the statistical evaluation. For better possibilities of interpreting a weighted
mean has been introduced. The mean is weighted with the total number of measurements a pilot
site did however only if more than 3 pilot sites provided results. As the standard deviation is huge,
the values have no real significance. As one example we have a closer look to KPI 07a “Duration of
voice channel blocking” in seconds. The voice channel blocking time is the most important KPI for the
later success of eCall. This KPI indicates the additional time which the passenger in a vehicle is
waiting to be connected to the call handler compared with a standard 112 call. Therefore all
measures should be taken to assure that this time is as short as possible. This KPI has been measured
by 6 pilot sites. The mean is 8.0s; the weighted mean 7.5s, median 6.5s, minimum value is 5s and
maximum value is 15.8s after exclusion of outliers. The standard deviation is 3.7. Therefore the
analysis and conclusions have to be made based on the individual results per pilot sites and the
degree of similarity between different pilot sites.
D4.3 Intermediate test results
15/05/2014 38 Version 1.3
KP
I
Explanation Mean
Wei
ghte
d
mea
n2
Med
ian
Min Max
test
ing
cou
ntr
ies
Std dev.
1a Number of automatically initiated eCalls 1918.3 644 79 1069
7 8
3359.1
1b Number of manually initiated eCalls 771.5 286 60 3791 8 1180.
7
2a Success rate of completed eCalls using 112 [%]
91 93 94 79 98 4 7.0
2b Success rate of completed eCalls using long number[%]
76 71 70 57 100 6 16.4
3 Success rate of received MSDs [%] 89 81 97 71 100 9 11.5
PSAP modem change request implementation (better identification of incorrect MSD
parameters)
D4.3 Intermediate test results
15/05/2014 85 Version 1.3
7.3 Finland
Introduction 7.3.1
7.3.1.1 Background
This report is prepared as a part of HeERO project and it provides input for deliverable D4.3.
7.3.1.2 Objectives
The objective is to evaluate the results of the Finnish eCall pilot implemented in Phase 1 of HeERO
project. The main focus in the evaluation of the Finnish eCall pilot will be to evaluate the
successfulness of MSD transmission and the whole eCall session. The evaluation process will also
identify points for improvement and provide recommendations when necessary.
7.3.1.3 Methods
The Finnish eCall pilot realized during Phase 1 of HeERO is documented in chapter 2 (Finnish eCall
pilot system). The key performance indicators used for evaluation are presented in chapter 3 (eCall
testing – key performance indicators) and testing and analysis methods used in chapter 4 (eCall
testing – testing and analysis methods).
HeERO KPI measurement 7.3.2
7.3.2.1 Basic definitions
The point of time, where the IVS starts the process to get in contact with the PSAP is called “call
connection initiation”, the corresponding phase starting here is called “call establishment”.
The phase, where the transmission of the MSD happens is called “data transmission”.
The phase, where the voice communication happens is called “voice transmission”.
D4.3 Intermediate test results
15/05/2014 86 Version 1.3
Significant instants defined with respect to the module where the measurement takes place:
T0-IVS: IVS initiated the eCall (start of phase “call establishment”) – time of Call setup
T1-IVS: IVS starts the MSD transmission (start of phase “data transmission”) – time of Initiation message
T2-IVS: End of phase “data transmission” - time of HL-ACK
T0-PSAP: Initiated eCall is indicated at PSAP – Answer message
T1-PSAP: Start of MSD reception at PSAP - time of START message (SEND MSD request)
T2-PSAP: Start of phase “voice transmission”
T3-PSAP: Start of dispatching information about incident to emergency services
T4-PSAP: Start of dispatching information about incident to TMC
T3-ES: Start of confirmation about incident handling to PSAP
T4-ES: Start of dispatching rescue forces
T0-FIX: point in time when the call enters the 112 national network
Time instants measurement points:
IVS SS7
monitorin
g
PSAP Telco part PSAP ASW part
IVS PSAP TMCES
T0-IVS
T2-IVS
T1-IVS
T0-PSAP
T3-PSAP
T2-PSAP
T4-PSAP
T1-PSAP
T0-FIX
INIT msg
AL-ACK
SEND MSD
msg
Start MSD transmission
End MSD transmission
ES
T3-ESPSAP
TMC
Call establishment phase
Data transmission phase
Voice communication phase
PSAP eCall processing phase
Answer
msg
SETUP
eCall initiation
Confirm
Dispatch
Dispatch
D4.3 Intermediate test results
15/05/2014 87 Version 1.3
T0-
IVS
Log IVS :
SETUP msg time
CDR MOB
:
SETUP
msg
- -
T1-
IVS Log IVS :
INIT msg time (push req) -
PSAP modem log:
Ecall::onPsapCatchEve
nt
PSAP_CONTROL_SYNC
-
T2-
IVS Log IVS :
Time of HL ACK
-
PSAP modem log:
Ecall::onPsapCatchEve
nt
PSAP_SENDING_HLAC
K
-
T0-
PSA
P
CDR FIX :
ANM msg
SIP message log :
SIP/2.0 200 OK after
INVITE
PSAP modem log:
DateOfBirth
-
T1-
PSA
P Log IVS :
Time of SEND MSD reception -
PSAP modem log:
Ecall::onPsapCatchEve
nt
PSAP_SENDING_STAR
T
-
T2-
PSA
P - -
T server log :
message
EventEstablished
DM3TCTV.log :
MSG:IDM_PHONE_RINGI
NG
D4.3 Intermediate test results
15/05/2014 88 Version 1.3
T3-
PSA
P
- - - DM3TCTV.log :
IDM_SET_ID_UDALOSTI
T4-
PSA
P - - -
DM3TCTV.log :
MSG_TOBE_SENT_TO_T
MC
Finnish eCall pilot system 7.3.3
The following figure (Figure 46) outlines the Finnish eCall pilot system and its basic components (see
D2.3 Finnish pilot implementation plan).
Figure 46 HeERO Finnish pilot system architecture outline (FI)
The main parts of the system include:
eCall client simulator (eCall IVS)
PSAP simulator (eCall test bed)
ECall pilot system control and administrator’s UI.
eCall IVS
Software
modem
GPS
User interfaceMessage activation and configuration
eCall client simulatorPSAP
PSAP1
PSAP2
eCall testbed
Software
modemMSD
encodingMSD
extraction, decoding,
validation
Voice + MSD
ELS
ELS SOAP API
ELS
ELS SOAP API
Risk assessment by a
human user
PSAP2 API
MSD handling,
data complementing
and verification
PSAP1 API
call to test number
https
eCall testbed control UI
Control and configuration
Show result logs
www.ecall.fi/xyzwww.ecall.fi/xyz
Logs, configurations
PSAP2
PSAP simulator
Control and administrator’s UI
D4.3 Intermediate test results
15/05/2014 89 Version 1.3
7.3.3.1 ECall client simulator (IVS)
Two IVS prototypes were used in Phase 1 of HeERO in Finland. The first of them (Gecko) implements
only transmission of MSD but provides no voice connection (Figure 47). However, it has features for
automatic activation of MSD transmission. The second prototype used in Finland (Indagon) provides
both voice connection and transmission of MSD but no features for automatic activation of eCall
(Figure 48).
Figure 47: Gecko IVS prototype (FI)
Figure 48: Indagon IVS prototype (FI)
D4.3 Intermediate test results
15/05/2014 90 Version 1.3
Both prototypes have been photographed without cables used for power supply or data logging.
While the Gecko prototype has external GSM and GNSS antennas, the Indagon prototype has internal
antennas mounted within its plastic enclosure.
Both prototypes provide interfaces suitable for data logging at IVS side. However, the data logging
interfaces as well as the formats of log files are different for the two prototypes. Therefore, both
prototypes required their own tools for post processing of log files.
The test results are presented separately for each IVS prototype. Because the Gecko prototype
provides no voice connection to PSAP, KPI_002B (Table 8) is measured only for the Indagon
prototype.
7.3.3.2 PSAP simulator
The PSAP simulator part of the system consists of eCall test bed and eCall test bed control user
interface (Figure 46). These components were used in Phase 1 tests of the Finnish eCall pilot as a
PSAP simulator.
The eCall test bed is the eCall message receiver part of the system. It includes functionality for
handling incoming eCall phone calls. It receives and decodes eCall message data, includes interfaces
for PSAP1 and PSAP2 subsystems, provides logs for analysing results and includes facility for
configuring the operation of the system.
A landline phone number (other than 112) was configured for test bed to receive eCall phone calls.
The test bed used the standardized in-band modem (Version 10.0.0) implemented with the ANSI-C
reference code to extract MSD data from the call. The incoming MSD messages were assumed to be
encoded according to the standard CEN EN 15722 (eCall minimum set of data). The test bed decoded
and validated MSD messages. For analysing results, the eCall test bed provided data logging features.
ECall testing – key performance indicators 7.3.4
The following key performance indicators were planned to be used to evaluate the Finnish eCall pilot after the first round of tests carried out in HeERO (Table 26).
KPI Name of KPI Definition
KPI_001B Number of manually initiated eCalls
Every manual initiation of an eCall is counted up to get an overview of the total number of manually initiated eCalls.
KPI_002B Success rate of completed eCalls using long number
eCall success rate = successful eCalls / all initiated eCalls * 100 % Successful eCalls = initiated eCalls - failed eCalls General definition of successful eCall: Voice call path was established, MSD data transfer was done and MSD content was shown at operator’s desk. Initiated eCall: eCall triggered by IVS Failed eCall: Either no establishment of a voice path connection at all, or no stable connection at all, or no voice call possible or
D4.3 Intermediate test results
15/05/2014 91 Version 1.3
no MSD transmission or faulty MSD transmitted
KPI_003 Success rate of received MSDs
MSD success rate = successful MSDs / all initiated MSDs * 100 % Successful MSDs = initiated MSDs - failed MSDs General definition of successful MSD: Content is presented at operator’s desk in PSAP Initiated MSD: Start of MSD-transmission in push mode (comes from IVS) Failed MSD: No MSD data transmission or faulty transmission: voice call started without content of MSD is presented at operator’s desk in PSAP or MSD transmission is not successfully completed.
KPI_004 Success rate of correct MSDs
MSD correctness rate = correct MSDs / all received MSDs * 100 % correct MSDs = received MSDs - incorrect MSDs
KPI_014 Success rate of VIN decoding without EUCARIS
VIN success rate = correct reported information about vehicle by database / all requests at database * 100 %; “Correct” if provided VIN is identical and presented data fits to type of vehicle (interface to database is correctly implemented), otherwise “Incorrect”.
KPI_019 Dispatch time of incident data to TMC
Required time until incident data is presented = point of time of presentation of incident data at operator’s desk in TMC - point of time for IVS initiated the eCall
Table 26: Key performance indicators planned for evaluation of Finnish eCall
Limitation of measured key performance indicators
Because the Gecko IVS prototype had an automatic activation feature, KPI_001A (Number of
automatically initiated eCalls) had to be used for that prototype instead of KPI_001B. KPIs requiring a
voice connection could not be measured for the Gecko IVS prototype. This was caused by the fact
that the Gecko prototype implemented only the transmission of MSD but not voice connection.
All other KPIs in Table 8 except KPI_014 and KPI_019 were evaluated for the Phase 1 implementation
of the Finnish eCall pilot.. KPI_014 was not measured in Phase 1 tests because the Finnish PSAP
organization has indicated that they are not going to use the VIN number at least in current situation.
Some of the recommended KPIs (listed in ) could not be measured in the Phase 1 tests in Finland:
KPI_002a (Success rate of completed eCalls using 112): The eCall-flag has not been
implemented yet in mobile networks in Finland. It will be available in Finland only after the
HeERO project.
KPI_005 (Duration until MSD is presented in PSAP): There was no real PSAP (with user
interface) available for the tests in Finland. Therefore, presentation time could not be
measured.
KPI_006 (Success rate of established voice transmissions): At the time of test preparation
and selection of KPIs there was no information about implementation of the voice
connection in the two IVS prototypes. Therefore voice connection related KPIs were not
tested in Finland Phase 1. This KPI will be measured in the Phase 2 if HeERO tests in Finland.
KPI_007a (Duration of voice channel blocking): [The same as above]]
D4.3 Intermediate test results
15/05/2014 92 Version 1.3
KPI_013 (Success rate of heading information): In Finland the eCall IVS devices were early
prototypes and we wanted to concentrate on eCall specific new features. Positioning rated
optimization of the devices (e.g. GPS antennas etc.) was not in the production level.
Therefore, positioning related KPIs were not tested in Finland Phase 1. This KPI is planned to
be measured in the Phase 2 if HeERO tests in Finland.
KPIs measured for the two prototypes during tests carried out in 2012 are summarized in Table 27.
KPI IVS prototype
Gecko Indagon
KPI_001A – Number of automatically initiated eCalls X
KPI_001B – Number of manually initiated eCalls X
KPI_002B – Success rate of completed eCalls using long
number
X
KPI_003 – Success rate of received MSDs X X
KPI_004 – Success rate of correct MSDs X X Table 27: KPIs measured in Phase 1 in Finnish eCall pilot (FI)
ECall testing – testing and analysis methods 7.3.5
7.3.5.1 Overview
Tests were carried out by installing the IVS prototypes in a test vehicle and driving a test route (Table
28). The Gecko IVS prototype was activated automatically using the periodic automatic activation
feature provided by the prototype. The number of activations (KPI_001A or KPI_001B) was
documented separately for each route section.
The number of activated eCalls was documented manually for the Indagon prototype (KPI_001B) by
drawing a line each time the prototype was activated and making notes of the times each route
section was started or finished. KPI_003 and KPI_004 were then determined on the basis of notes
written during the test and log files provided by the test bed.
For the Gecko prototype, the number of activations was determined on the basis of a log file
collected on the IVS side, notes on the times each route section was started or finished and log files
provided by the eCall test bed.
For both prototypes, the number of successful MSD transmissions was determined on the basis of
test bed log files and the times each route section had been started or finished.
The successfulness of eCall session (KPI_002B) was determined by checking that a two-way voice
connection had been established and that the MSD was transmitted successfully. The voice
connection was checked manually by persons talking and listening on IVS and test bed side. The
times of voice calls were recorded during the tests, and corresponding modem sessions were then
identified among the test bed log files to see whether the MSDs related to the tested voice
D4.3 Intermediate test results
15/05/2014 93 Version 1.3
connections had been successfully received. The contents of those MSDs were not checked at this
stage.
The processing of test results is illustrated in Figure 49.
Figure 49: Processing of test results (FI)
7.3.5.2 Test route
A description of the test route used is provided in Table 28.
Route section Length / km Estimated driving time
Environment
1 Otaniemi-Helsinginkatu-Karhusaarentie
25,3 38 min Urban /suburban
2 Karhusaarentie-Hanko railway station
119,6 1 h 23 min Interurban road
3 Hanko railway station - Nummi-Pusula
103,9 1 h 39 min Regional and rural roads
Hanko railway station - Tenhola 48,2 50 min
Tenhola-Nummi-Pusula 55,7 49 min
4 Nummi-Pusula-Otaniemi 60,9 37 min Motorway VT1 and Ring Road I
Total 314,6 4 h 17 min Table 28: Description of test route (FI)
Test results 7.3.6
7.3.6.1 Gecko IVS prototype
The results for Gecko IVS prototype are presented in and summarized in Table 29 .
Gecko IVS
prototype
(USB output)
Indagon IVS
prototype
(Ethernet output)
Data logging to
a text file
(Laptop with
PuTTY)
Data logging to
a text file
(Laptop with
PuTTY)
Device-specific
postprocessing
of log f iles
Device-specific
postprocessing
of log f iles
Analysis of test
results
- Analysis with
IVS-specif ic
options
depending on
the capabilities
and limitations of
IVS prototypes
eCall testbed
(output as a text
f ile)
Presentation of
test results
- Presentation of
calculated KPIs
and other test
results
A B C D
A: IVS specif ic information
received via vendor-specif ic
interfaces
B: IVS specif ic information as a
text f ile
C: Events and other data
generated by IVS prototypes
expressed in a common f ile
format (plain text)
D: Values of selected KPIs
Notes
(notes made
during the test)
D4.3 Intermediate test results
15/05/2014 94 Version 1.3
Table 29: Summary of test results for Gecko IVS prototype (FI)
Table 30: Summary of test results for Gecko IVS prototype (FI)
Success rate of correct MSDs
The MSDs sent by the Gecko IVS prototype were received and decoded by the eCall test bed. All
MSDs received during the test had problems with latitude, longitude and timestamp fields. In other
words, the values decoded from MSDs were implausible. For this reason, the value of KPI_004
(Success rate of correct MSDs) was considered to be zero.
ID of test set:
Unit Result 1 Result 2 Result 3 Result 4
KPI_001a Number of automatically initiated eCalls - 22 94 144 47
KPI_001b Number of manually initiated eCalls - 0 0 0 0
KPI_002a Success rate of completed eCalls using 112 %
KPI_002b Success rate of completed eCalls using long number %
KPI_006 Success rate of established voice transmissions %
KPI_007 Duration of voice channel blocking s
KPI_007a
Duration of voice channel blocking:
automatic retransmission of MSD s
KPI_009 Accuracy of position m
KPI_010 Number of usable satellites -
KPI_011 Geometric dilution of precision -
KPI_012 Time between successful positioning fixes s
KPI_013 Success rate of heading information %
KPI_021 Number of successful call-backs -
KPI_022 Success rate of call-backs %
KPI_027 Total response time s
KPI_028 Number of cross-border tests -
KPI_003 Success rate of received MSDs % 100 % 97,50 % 93,94 % 86,67 %
KPI_004 Success rate of correct MSDs % 0 0 0 0
ID of test set: 1
Date: 13.-15.6.2012
Time:
Type of
initiation: a a a a
Roaming (y) (y) (y) (y)
Environment urban interurban rural motorway
Moving vehicle y y y y
No. of involved
vehicles in
incident 1 1 1 1
Name of KPI1
ID of
test set:
Result 1 Unit
KPI_001a Number of automatically initiated eCalls 307 -
KPI_001b Number of manually initiated eCalls 0 -
KPI_002a Success rate of completed eCalls using 112 %
KPI_002b Success rate of completed eCalls using long number %
KPI_003 Success rate of received MSDs 94,69 %
KPI_004 Success rate of correct MSDs 0,00 %
1 Name of KPI
IVS 1
PSAP 1
MNO 1
D4.3 Intermediate test results
15/05/2014 95 Version 1.3
Other observations during the test
The IVS activated 307 times in total during the test. However, only 226 calls were successfully
established between IVS and PSAP. This means that in 81 cases of 307 some kind of failure occurred
in call setup or network registration phase before the in-band modem could start transmitting the
MSD. These cases were observed on all route sections (Table 31).
Route section
1 2 3 4
Successful MSDs 17 78 93 26
Started MSDs (cases with call setup from IVS to test bed)
17 80 99 30
Activations with failed network registration or call setup
5 14 45 17
Automatic activations (KPI_001A) 22 94 144 47 Table 31: Summary of activated eCalls, started MSDs and successful MSDs, Gecko prototype (FI)
7.3.6.2 Indagon IVS prototype
Number of manually initiated eCalls (KPI_001B)
The number of manually initiated eCalls was determined on the basis of notes made during the test.
Success rate of completed eCalls using long number (KPI_002B)
Success rate of eCalls using long number was determined by checking the successfulness of both
voice connection and the MSD for a sample of all initiated eCalls made with the Indagon IVS
prototype. In this context, a successful MSD was defined as a MSD which had been successfully
received by the test bed. The contents of the MSDs were not checked at this stage.
Success rate of received MSDs (KPI_003)
The success rate of received MSDs was determined by dividing the number of MSDs successfully
received by the test bed by the number of times the test bed had started the reception of MSD using
the in-band modem.
Success rate of correct MSDs (KPI_004)
The correctness of the MSDs transmitted by the Indagon IVS prototype was checked by decoding
them in the eCall test bed and comparing the results against the MSD standard (EN15722). The
version number and message identifier fields decoded from the MSDs were zeros in all of the MSDs
transmitted by the Indagon prototype. At least some MSDs also had implausible latitude and
longitude values. This result suggests that the prototype does not set correct values for variables
D4.3 Intermediate test results
15/05/2014 96 Version 1.3
“MSD version” and “messageidentifier” nor has some other problems with encoding of MSD.
Therefore, the success rate of correct MSDs (KPI_003) was considered to be zero.
The test results for the Indagon IVS prototype are presented in Table 32 and summarized in Table 34.
Table 32:.Summary of test results for Indagon IVS prototype (FI)
ID o
f te
st s
et:
Un
itR
esu
lt 1
Re
sult
2R
esu
lt 3
Re
sult
4R
esu
lt 5
Re
sult
6R
esu
lt 7
Re
sult
8R
esu
lt 9
Re
sult
10
KP
I_00
1aN
um
be
r o
f au
tom
atic
ally
init
iate
d e
Cal
ls-
00
00
00
00
00
KP
I_00
1bN
um
be
r o
f m
anu
ally
init
iate
d e
Cal
ls-
3457
3321
3352
1012
5813
KP
I_00
2aSu
cce
ss r
ate
of
com
ple
ted
eC
alls
usi
ng
112
%
KP
I_00
2bSu
cce
ss r
ate
of
com
ple
ted
eC
alls
usi
ng
lon
g n
um
be
r%
56 %
--
--
-50
%67
%-
-
KP
I_00
6Su
cce
ss r
ate
of
est
abli
she
d v
oic
e t
ran
smis
sio
ns
%
KP
I_00
7D
ura
tio
n o
f vo
ice
ch
ann
el b
lock
ing
s
KP
I_00
7a
Du
rati
on
of
voic
e c
han
ne
l blo
ckin
g:
auto
mat
ic r
etr
ansm
issi
on
of
MSD
s
KP
I_00
9A
ccu
racy
of
po
siti
on
m
KP
I_01
0N
um
be
r o
f u
sab
le s
ate
llit
es
-
KP
I_01
1G
eo
me
tric
dil
uti
on
of
pre
cisi
on
-
KP
I_01
2Ti
me
be
twe
en
su
cce
ssfu
l po
siti
on
ing
fixe
ss
KP
I_01
3Su
cce
ss r
ate
of
he
adin
g in
form
atio
n%
KP
I_02
1N
um
be
r o
f su
cce
ssfu
l cal
l-b
acks
-
KP
I_02
2Su
cce
ss r
ate
of
call
-bac
ks%
KP
I_02
7To
tal r
esp
on
se t
ime
s
KP
I_02
8N
um
be
r o
f cr
oss
-bo
rde
r te
sts
-
KP
I_00
3Su
cce
ss r
ate
of
rece
ive
d M
SDs
%59
%47
,92
%30
,30
%42
,86
%48
,00
%50
,00
%55
,56
%66
,67
%43
,40
%46
,15
%
KP
I_00
4Su
cce
ss r
ate
of
corr
ect
MSD
s%
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
ID o
f te
st s
et:
2
Dat
e:
25.9
.201
225
.9.2
012
25.9
.201
225
.9.2
012
26.9
.201
226
.9.2
012
26.9
.201
226
.9.2
012
26.9
.201
226
.9.2
012
Tim
e:
Typ
e o
f
init
iati
on
:m
mm
mm
mm
mm
m
Ro
amin
gy
yy
yy
yy
yy
y
Envi
ron
me
nt
urb
anin
teru
rban
rura
lm
oto
rway
urb
anin
teru
rban
rura
lm
oto
rway
rura
lm
oto
rway
Mo
vin
g ve
hic
ley
yy
yy
yy
yy
y
No
. of
invo
lve
d
ve
hic
les
in
inci
de
nt
11
11
11
11
11
1N
ame
of
KP
I
D4.3 Intermediate test results
15/05/2014 97 Version 1.3
Table 33: Summary of test results for Indagon IVS prototype (FI)
Other observations during the test
The IVS prototype was activated 323 times in total during the test. However, only 290 calls were
successfully established between the IVS and the eCall test bed. This means that in 33 of 323 cases
some kind of failure occurred in network registration or call setup phase before the in-band modem
could start transmitting the MSD (Table 34).
Route section
1 2 3 4
Successful MSDs 31 45 38 23
Started MSDs (cases with call setup from IVS to test bed)
57 92 95 46
Activations with failed network registration or call setup
10 17 6 0
Manual activations (KPI_001B) 67 109 101 46 Table 34: Summary of activated eCalls, started MSDs and successful MSDs, Indagon prototype (FI)
The prototype had problems in obtaining position and time from GNSS. This was documented in the
log files generated by the prototype.
7.3.6.3 Calculation of confidence intervals for success rate of received MSDs
The results of the Finnish eCall pilot for year 2012 include the shares of successful MSDs of all
received MSDs (KPI_003). Because only a finite number of trials were carried out for each
combination of prototype and route section, the impact of random variation on the results has to be
estimated by calculating confidence intervals for MSD success rate.
When estimating confidence intervals for the MSD success rate, assumptions on the properties of
the system under analysis are needed. After that, suitable mathematical tools have to be employed
to solve the problem.
ID of
test set:
Result 2 Unit
KPI_001a Number of automatically initiated eCalls 0 -
KPI_001b Number of manually initiated eCalls 323 -
KPI_002a Success rate of completed eCalls using 112 %
KPI_002b Success rate of completed eCalls using long number 57 % %
KPI_003 Success rate of received MSDs 47,24 %
KPI_004 Success rate of correct MSDs 0,00 %
1 Name of KPI
IVS 2
PSAP 1
MNO 1
D4.3 Intermediate test results
15/05/2014 98 Version 1.3
When calculating confidence intervals for the success rate of received MSDs (KPI_003), it was
assumed that transmission of a MSD is in fact a Bernoulli trial which can result in either success or
failure. The successes or failures of individual MSDs were assumed to be independent of each other
and each MSD transmitted by the same prototype within the same route section (in the same
operating environment) was assumed to have equal probability of success or failure. When the
previous assumptions are both true and the experiment consists of a fixed number of trials which can
result in either success or failure, the number of successful MSDs transmitted by the same prototype
within the same route section follows binomial distribution.
Means for calculating confidence intervals for a binomially distributed variable can be found in
textbooks (Milton and Arnold 1995). In case of a binomially distributed random variable, confidence
intervals (with probability of 0.95) can be calculated with formula
√ ( )
(1), when and ( )
(Milton and Arnold 1995), where is the probability of success estimated on the basis of the sample
under analysis.
The confidence intervals calculated for KPI_003 using (1) and figures from Chapters 5.1 and 5.2 are
Estimation of confidence intervals successful: n x p > 5 and n x (1-p) > 5
yes yes yes yes
Table 36: Confidence intervals for success rate of received MSDs (KPI_003), Indagon prototype. (FI)
D4.3 Intermediate test results
15/05/2014 99 Version 1.3
Conclusions 7.3.7
Success rate of received MSDs
The tests have indicated that there is a large difference between prototypes in success rate of MSD
transmission. While the Gecko prototype was able to successfully transmit 94.69% of initiated MSDs,
the corresponding success rate was only 47.24% for the Indagon prototype. This difference has most
likely been caused by differences in hardware and software between the two prototypes. While the
94.69% success rate is reasonably close to the required success rate, 47.24% leaves much room for
improvement. In any case, the evaluated devices were prototypes under development, and the MSD
success rate can likely be improved by solving possible hardware-related problems and by updating
the software of the prototypes.
However, the figures mentioned above do not take into account the number of cases in which some
kind of failure occurs in the call setup phase. In these cases, no call is connected from IVS to test bed
and the in-band modem never starts the transmission of MSD. In case of the Gecko prototype, 81 of
307 calls attempted by the IVS (26.38%) were never connected to the test bed. In case of the Indagon
prototype, this was the result for 33 of the 323 calls attempted (10.22%). The cause behind the large
number of failures in call setup or network registration should be investigated in detail. The
difference between the two prototypes may be related to the differences in the status of mobile
network at the time of testing, the manner the test was carried out or some unknown factor which is
likely related to the mobile network or the SIM card used in the tests.
Success rate of completed eCalls
The success rate of completed eCalls was measured only for the Indagon IVS prototype. The success
rate of completed eCalls (57%) was limited by the relatively low success rate of received MSDs.
However, all MSDs for which the test bed sent acknowledgement to IVS were classified as successful
without looking at the contents. The most important measures to improve the success rate is to
ensure that the MSD is encoded and decoded correctly and to improve the MSD success rate for
example by solving possible hardware-related problems and updating the software of the prototype.
MSD encoding and decoding
Both of the tested prototypes had at least some inconsistencies or other problems in encoding of
MSD. This underlines the need to verify the conformance of the test bed MSD decoder to the
specification in EN15722 and to make the required changes to IVS prototypes to ensure correct
encoding of MSD.
Recommendations 7.3.8
The Indagon prototype used in the tests should be checked to detect possible hardware related
problems which may have affected the success rate of MSD transmission. The encoding of MSD
carried out by the prototypes should be investigated in detail to make sure that the MSD received by
the test bed complies with EN15722. It is also recommended that the firmware of the prototypes
should be updated to make sure that the best possible implementation of the in-band modem is
used in HeERO tests to be carried out in 2013. The causes of calls not connected by the mobile
D4.3 Intermediate test results
15/05/2014 100 Version 1.3
network should also be investigated in cooperation with the mobile network operator. Tests to be
carried out in 2013 should also cover a larger geographical area in Finland and involve several MNOs
instead of just one.
References 7.3.9
Götte, S. and Filjar, R 2012. KPIs, test specification and methodology, Final version. Deliverable D4.2 of HeERO. http://www.heero-pilot.eu/ressource/static/files/heero_wp4_d4-2-v1-1-_projectplace_112147_.pdf [accessed 1st October 2012] Milton, J. S. and Arnold, J. C. 1995. Introduction to probability and statistics. McGraw-Hill Book Co, Singapore.
Determination of KPIs out of eCall log files in the database 7.4.4
All logs of the performed eCalls are recorded within a database for later analysis. Out of this
database, all values for the defined KPIs can be determined as described in the following:
D4.3 Intermediate test results
15/05/2014 106 Version 1.3
KPI_001a – Number of automatically initiated eCalls: To determine this KPI, the logged parameter
concerning the activation is evaluated based on information from the PSAP log file.
KPI_001b – Number of manually initiated eCalls: To determine this KPI, the logged parameter
concerning the activation is evaluated based on information from the PSAP log file.
KPI_002b – Success rate of completed eCalls using long number: To determine this KPI, the PSAP
operator and test driver talked to each other at various occasions during the manual test session.
During the “emergency call” both parties filled in a test protocol describing the quality of voice
communication. Successfully completed eCalls are highlighted in the test protocol.
KPI_003 – Success rate of received MSDs: As every MSD is logged, each of them was checked for
invalid data. If – worst case - no MSD was received during the eCall, this was also noted. To
determine pass-fail criteria for this KPI, invalid MSDs were flagged and counted.
KPI_004 – Success rate of correct MSDs: As every MSD is logged on IVS side and on PSAP side, the
data pairs were compared. To evaluate this KPI, incorrect MSDs were flagged and counted.
KPI_005 – Duration until MSD is presented in PSAP: To assess this KPI, two timestamps were
evaluated, one in the IVS log file indicating the eCall activation and one in the PSAP log file indicating
that the MSD was received by the PSAP.
KPI_006 – Success rate of established voice transmissions: To assess this KPI, the PSAP operator and
test driver talked to each other a various occasions during the manual test session. During the
“emergency call” both parties filled out a test protocol concerning the quality of the voice
communication. Successful documented voice transmissions within eCall sessions are counted.
KPI_007 – Duration of voice channel blocking: To assess this KPI, two timestamps were evaluated,
one in the PSAP log file indicating the beginning of the log file and another one in the PSAP log file
indicating that the voice connection was established.
KPI_009 – Accuracy of position: To assess this KPI, the PSAP operator and test driver recorded the
vehicle’s position during the “emergency call”. In the post processing, the test protocols were
compared concerning the position information.
KPI_013 – Success rate of heading information: To assess this KPI, the heading parameter of the IVS log file was evaluated. In the post processing, the value of this parameter was compared with the direction of the road at the position logged by the IVS at the time of the eCall activation, assuming that the vehicle was parked in traffic direction.
Results of KPIs 7.4.5
KPI_001a Number of automatically initiated eCalls 10,697
KPI_001b Number of manually initiated eCalls 248
D4.3 Intermediate test results
15/05/2014 107 Version 1.3
KPI_002b Success rate of completed eCalls using long number [%] 68
KPI_003 Success rate of received MSDs [%] 72
KPI_004 Success rate of correct MSDs [%] 100
KPI_005 Duration until MSD is presented in PSAP [s] 9.9
KPI_006 Success rate of established voice transmissions [%] 92
KPI_007a Duration of voice channel blocking [s] 8.7
Table 41: results of the German KPIs
In the following the MSD Presentation Time (KPI 5) and Voice Channel Blocking Time (KPI 7a) are
evaluated in detail, because these are the most relevant parameters for acceptance of eCall.
Evaluation of MSD Presentation Time (KPI 5) 7.4.6
At first we show the distribution of all values (including outliers) of the two IVS.
Figure 53: Distribution of the MSD Presentation Time with different IVS (DE)
The variation of the values in the tests of Continental is wider and the focus seems to be higher than
in the tests with S1nn. The same shows Figure 53 where the relative histograms are plotted.
In the data basis of the next plots (relative histograms) the outliers are already removed.
M S D
IV S
[s]
S1n
n
Co
nti
nen
tal
0
1 0
2 0
3 0
4 0
5 0
D4.3 Intermediate test results
15/05/2014 108 Version 1.3
Figure 54: histograms of the MSD Presentation Time with different IVS (DE)
Relative histograms show the relative distribution of the data which are separated in classes. The left
histogram shows that the S1nn values are smaller than the MSD Presentation Time with an IVS by
Continental.
The figure shows like Table 42 that there are more measurements with Continental than with S1nn.
Therefore the dispersion is much higher (standard deviation of 2.3 s versus 0.8 s).
IVS S1nn Continental
Analysed values 65 112
Number of outliers 0 1
Number of values 65 111
Minimum 7 9
25% Percentile 8 10
Median 9 12
75% Percentile 9 14
Maximum 10 20
Mean 8.6 12.3
Std. Deviation 0.75 2.35
Std. Error of Mean 0.093 0.22
S 1 n n
M S D P re s e n ta t io n T im e [s ]
Re
lati
ve
fre
qu
en
cy
[%
]
7 8 910
0
1 0
2 0
3 0
4 0
5 0
C o n t in e n ta l
M S D P re s e n ta t io n T im e [s ]
Re
lati
ve
fre
qu
en
cy
[%
]
910
11
12
13
14
15
16
17
18
19
20
0
5
1 0
1 5
2 0
2 5
D4.3 Intermediate test results
15/05/2014 109 Version 1.3
Lower 95% CI of mean 8.4 11.9
Upper 95% CI of mean 8.7 12.8
Geometric mean 8.6 12.1
Skewness 0.16 0.81
Kurtosis -0.34 0.24
Table 42: statistical evaluation of MSD Presentation Time (DE)
Evaluation of Voice Channel Blocking Time (KPI 007a) 7.4.7
Figure 55: Distribution of the Voice Channel Blocking Time with different providers (DE)
The Voice Channel Blocking time with Continental takes longer than with S1nn. Figure 55 also shows
a wider distribution of the Continental values.
V C B
IV S
[s]
S1n
n
Co
nti
nen
tal
0
5
1 0
1 5
2 0
2 5
D4.3 Intermediate test results
15/05/2014 110 Version 1.3
Figure 56: histograms of Voice Channel Blocking Time with different IVS (DE)
IVS S1nn Continental
Analysed values 65 186
Number of outliers 0 5
Number of values 65 181
Minimum 5 7
25% Percentile 5 8
Median 6 9
75% Percentile 6 10
Maximum 8 15
Mean 5.9 9.2
Std. Deviation 0.82 1.62
Std. Error of Mean 0.10 0.12
Lower 95% CI of mean 5.8 9
Upper 95% CI of mean 6.2 9.4
Geometric mean 5.9 9.1
S 1 n n
V o ic e C h a n n e l B lo c k in g T im e [s ]
Re
lati
ve
fre
qu
en
cy
[%
]
5 6 7 8
0
1 0
2 0
3 0
4 0
5 0
C o n t in e n ta l
V o ic e C h a n n e l B lo c k in g T im e [s ]
Re
lati
ve
fre
qu
en
cy
[%
]
7 8 910
11
12
13
14
15
0
1 0
2 0
3 0
4 0
D4.3 Intermediate test results
15/05/2014 111 Version 1.3
Skewness 0.57 1.13
Kurtosis -0.15 1.28
Table 43: statistical evaluation of Voice Channel Blocking Time (DE)
The statistic values in Table 43 confirm the figures before. The standard deviation of Continental is
even larger than the whole distribution of the values.
Automatic test results 7.4.8
MSISDN
Total
no. of
calls
No. of
received
MSD
Success
probability
%
S1nn +4915734030788 1917 118 6.15
S1nn +491701737904 13 13 100
S1nn +491701748772 905 118 13.04
S1nn +491742685107 125 22 17.6
S1nn +4917692162809 317 93 29.34
Conti +4915152258969 2558 1913 74.78
Conti +4915205404588 2003 1245 62.16
Conti +4915737472190 1211 983 81.17
Conti +4917692190758 1896 1475 77.79
Table 44: automatic test results S1nn and Continental (DE)
For the automatic tests only the success rate has been determined as the success rate of both
systems is quite low.
The success probability of the S1nn tests is much too low and was caused by a synchronisation timer
issue.
The success probability of Continental is better but also not sufficient and needs to be improved for
Phase 2.
D4.3 Intermediate test results
15/05/2014 112 Version 1.3
Interoperability tests with CZ 7.4.9
In September and October 2012 interoperability tests with the Czech Republic were done, in which
75 Czech eCalls reached the German PSAP. In the following figure the most important KPIs are
evaluated for these calls.
Figure 57: histogram of KPI 5 with Czech IVS (DE)
In Figure 57 one can see that the majority of eCalls resulted in a MSD Presentation time between 5 to
10 s. An analysis of the calls in which this parameter is smaller showed, that in these cases no MSD
transmission took place. Calls in which KPI 5 is higher than 10 but less than 20 s were caused by
retransmission of MSD due to transmission errors (CRC). The calls with a value of larger than 20 s are
calls in which a retransmission of the MSD was requested after voice communication.
In Figure 58 one can see that the majority of eCalls resulted in a voice channel blocking time between
6 to 9 s. An analysis of the calls in which this parameter is smaller showed, that in these cases no
MSD transmission took place. Calls in which KPI 7 is higher than 9 but less than 18 s were caused by
retransmission of MSD due to transmission errors (CRC). The calls with a value of larger than 18 s are
calls in which a retransmission of the MSD was requested after voice communication. The overall
performance of the Czech IVS is better than the two IVS from German manufacturers resulting into a
slightly lower mean value for the 6 to 9 second class of performance.
Conclusions 7.4.10
To generate large amounts of data, automatic tests were performed during the test phase, but for
several reasons the success rate was quite low. Although the number of manually initiated eCalls was
much smaller, they provided much better results for evaluation. In addition to the scheduled tests,
cross border tests with the Czech Republic were performed.
The results of the various KPIs depend on which IVS is used. One IVS achieved for both KPI 5 and
KPI 7 better performance than the other. Same good performance was measured in the cross border
tests with the Czech Republic.
One reason for the bad performance is the missing of time stamps in the IVS logs which prevented
further evaluation of the calls.
Furthermore problems with the timers were identified which have been solved in the meantime.
Another finding is that it was possible to disturb the IVS in such a way, that normal operation was not
possible any longer. This happened for example when a busy number was dialled in the PSAP.
Although dialling 112 is not expected to get a busy tone, due to network errors a busy signal might
still be sent back to the IVS. Even if that is the case, the IVS should continue with standard behaviour
and for example try redialling the number.
Recommendations 7.4.11
In the next phase the log information provided by the IVS manufacturers should to be improved, so
that necessary information is provided with more reliability, but based on a comparable data
structure. The success rate should be increased to a level close to 100%.
It should be evaluated how the overall performance mainly of KPI 5 and KPI 7 can be increased.
As in Germany the MNOs do not support the eCall flag and will not implement it in 2013, the IVS
should be tested together with pilot sites which support the eCall flag like in the Czech Republic.
The analysis of the test results identified the necessity to increase timer t3 and t5 to allow higher
success rates of data transmission. To reduce disturbances on the line imposed by algorithms for
improvement of voice transmission, echo cancellation should optionally be used. The standard
should be modified accordingly.
D4.3 Intermediate test results
15/05/2014 114 Version 1.3
7.5 Greece The Greek Pilot Site has experienced significant difficulties since October 2011. The difficulties revolve
around the ability of the administration to complete the procurement process for the purchase of all
technical and logistical provision for the installation of Pan European eCall in Greece. The HeERO
management team have continued to support the pilot site as they continued to secure the necessary
permissions to secure a contractor to fulfil these tasks. The practical impact was that the Greek Pilot
Site has been unable to fulfil any of the operational and performance based activities for WP3 and
WP4 for HeERO 1. The results had been delivered in February 2014 and have been included in all
relevant chapters.
Methodology 7.5.1
On-road eCall tests in Greece were conducted in November-December 2013. We analysed log files
from a total of 1446 eCalls, manual and simulated automatic ones, equally distributed in Thessaloniki
and Athens and in several traffic environments. SIM cards by all three Greek MNOs were used; none
of the networks had the eCall discriminator software available. A long number was used. Two IVS
from Civitronic were used. The eCall PSAP was installed at the MINGR premises.
The following KPIs were evaluated in phase 1 of the Greek pilot tests.
KPI Description
1 KPI_001a Number of automatically initiated eCalls
2 KPI_001b Number of manually initiated eCalls
3 KPI_002b Success rate of completed eCalls using long number
4 KPI_003 Success rate of received MSDs
5 KPI_004 Success rate of correct MSDs
6 KPI_005 Duration until MSD is presented in PSAP
7 KPI_006 Success rate of established voice transmissions
8 KPI_014 Success rate of VIN decoding without EUCARIS
9 KPI_021 Number of successful call-backs
10 KPI_022 Success rate of call-backs
Table 45: KPIs measured in the Greek pilot tests (GR)
The following figure graphically depicts the number of tests performed in Athens and Thessaloniki
per day.
D4.3 Intermediate test results
15/05/2014 115 Version 1.3
Figure 59: Number of eCalls per day and region (GR)
To calculate the KPIs log files were automatically stored by the IVS and the PSAP. The data were
stored in an electronic file for each call, which was available at the end of the process for further
analysis. The data were introduced in relational database in an MS SQL Server in order to perform
the analysis and to extract the required KPIs.
Figure 60: Tables to extract the KPIs in an MS SQL server (GR)
Devices
id
IMEI
CallNumber
Base
IVSLog
ID
IVSID
TypeOfECall
NetworkRegistrationMode
Moment
PositionLat
PositionLon
PSAP
IMEI
IMSI
MSD
Network
Log1
Log2
Log3
Log4
Log5
Log6
Log7
Log8
Log9
Log10
Log11
Log12
Log13
Log14
Log15
Log16
Log17
Log18
Log19
Log20
Log21
Log22
Log23
Log24
Log25
Log26
Log27
Log28
Log29
KPI_14
KPIID
Text
CreatedDateTime
FullId
KPI_1A
KPIID
CallID
DateTime
ID
Message_Identifier
Control_automaticActivation
Control_testCall
Control_posCanBeTrusted
Control_vehicleType
VIN
vehPropStorageType_gasoline
vehPropStorageType_diesel
vehPropStorageType_compNat...
vehPropStorageType_liqPropGas
vehPropStorageType_electric
vehPropStorageType_hydrogen
Timestamp
VehLoc_latitude
VehLoc_longitude
Vehicle_direction
RecVehLoc_n_1_latitude
RecVehLoc_n_1_longitude
RecVehLoc_n_2_latitude
RecVehLoc_n_2_longitude
No_of_Passengers
Optional_additional_data
MSDReceived
CallerID
InbandID
Ticketnum
KPI_1B
KPIID
CallID
DateTime
ID
Message_Identifier
Control_automaticActivation
Control_testCall
Control_posCanBeTrusted
Control_vehicleType
VIN
vehPropStorageType_gasoline
vehPropStorageType_diesel
vehPropStorageType_compNat...
vehPropStorageType_liqPropGas
vehPropStorageType_electric
vehPropStorageType_hydrogen
Timestamp
VehLoc_latitude
VehLoc_longitude
Vehicle_direction
RecVehLoc_n_1_latitude
RecVehLoc_n_1_longitude
RecVehLoc_n_2_latitude
RecVehLoc_n_2_longitude
No_of_Passengers
Optional_additional_data
MSDReceived
CallerID
InbandID
Ticketnum
KPI_21_22
KPIID
FullId
recorder_local_time
Comment
KPI_2B
KPIID
CallID
DateTime
ID
Message_Identifier
Control_automaticActivation
Control_testCall
Control_posCanBeTrusted
Control_vehicleType
VIN
vehPropStorageType_gasoline
vehPropStorageType_diesel
vehPropStorageType_compNatGas
vehPropStorageType_liqPropGas
vehPropStorageType_electric
vehPropStorageType_hydrogen
Timestamp
VehLoc_latitude
VehLoc_longitude
Vehicle_direction
RecVehLoc_n_1_latitude
RecVehLoc_n_1_longitude
RecVehLoc_n_2_latitude
RecVehLoc_n_2_longitude
No_of_Passengers
Optional_additional_data
MSDReceived
CallerID
InbandID
Ticketnum
KPI_2BB
KPIID
FullId
recorder_local_time
Comment
KPI_3
KPIID
db_id
msd_record_local_time
msd_status
CallerID
Ticketnum
ticket_creation_time
KPI_4
KPIID
db_id
msd_record_local_time
ID
Message_Identifier
Control_automaticActivation
Control_testCall
Control_posCanBeTrusted
Control_vehicleType
VIN
vehPropStorageType_gasoline
vehPropStorageType_diesel
vehPropStorageType_compNat...
vehPropStorageType_liqPropGas
vehPropStorageType_electric
vehPropStorageType_hydrogen
ivs_local_time
VehLoc_latitude
VehLoc_longitude
Vehicle_direction
No_of_Passengers
Optional_additional_data
msd_status
CallerID
Ticketnum
ticket_creation_time
KPI_5
KPIID
FullId
Customer
ESN
EventDate
CreatedDate
AssignedDate
CallAuth
Closed
TicketState
TicketOwner
KPI_6
KPIID
FullId
recorder_local_time
Comment
KPI_7A
KPIID
FullId
Customer
ESN
EventDate
CreatedDate
AssignedDate
CallAuth
Closed
TicketState
TicketOwner
KPI_7B
KPIID
FullId
recorder_local_time
Comment
D4.3 Intermediate test results
15/05/2014 116 Version 1.3
Results 7.5.2
In total there were 1268 eCalls with simulated automatic initiation. The data for KPI_001a are shown
below.
Figure 61: Number of eCalls with simulated automatic initiation (KPI_001a) (GR)
Figure 62: Number of eCalls with simulated automatic initiation per MNO (KPI_001a) (GR)
In total there were 178 eCalls with manual initiation. The data for KPI_001b are shown below.
D4.3 Intermediate test results
15/05/2014 117 Version 1.3
Figure 63: Number of eCalls with manual initiation (KPI_001b) (GR)
Figure 64: Number of eCalls with manual initiation per MNO (KPI_001b) (GR)
The success rate of completed eCalls (KPI_002b) was 58.8% for Attica region and 65.8% for
Thessaloniki.
D4.3 Intermediate test results
15/05/2014 118 Version 1.3
Figure 65: Success rate of completed eCalls using long number (KPI_002b) (GR)
Figure 66: Success rate of completed eCalls using long number by MNO (KPI_002b) (GR)
The success rate of received MSDs (KPI_003) is 73.3% for Attica and 80.2% for Thessaloniki.
D4.3 Intermediate test results
15/05/2014 119 Version 1.3
Figure 67: Success rate of received MSDs (KPI_003) (GR)
Figure 68: Success rate of received MSDs per MNO (KPI_003) (GR)
The success rate of correct MSDs (KPI_004) was 79.5% for Attica and 81.6% for Thessaloniki.
D4.3 Intermediate test results
15/05/2014 120 Version 1.3
Figure 69: Success rate of correct MSDs (KPI_004) (GR)
Figure 70: Success rate of correct MSDs per MNO (KPI_004) (GR)
As regards the KPI_005, duration until MSD is presented in PSAP, there were inconsistencies in the
timestamps of the log files generated by the IVS and the PSAP, therefore this KPI could not be reliably
evaluated based on the log files. Because of the observed difficulties in calculating it there were
another 30 verification eCalls conducted at a later stage in Attica area. In these verification eCalls the
mean KPI_005 was 23 s with a standard deviation of 3 s.
The success rate of established voice transmissions (KPI_006) was 88% for Attica and 87.3% for
Thessaloniki.
D4.3 Intermediate test results
15/05/2014 121 Version 1.3
Figure 71: Success rate of established voice transmissions (KPI_006) (GR)
Figure 72: Success rate of established voice transmissions per provider (KPI_006) (GR)
VIN decoding without EUCARIS was successful in 736 of the 784 attempts (KPI_014). The success rate
is therefore very high. It is noted that decoding could not be implemented on some of the trial days;
this is why the success rate is around 94%.
D4.3 Intermediate test results
15/05/2014 122 Version 1.3
Figure 73: Success rate of VIN decoding without EUCARIS (KPI_014) (GR)
Figure 74: Success rate of VIN decoding without EUCARIS per MNO (KPI_014) (GR)
In total there were 259 successful call-backs from the PSAP to the IVS (KPI_021). The data about
KPI_021 are shown below:
D4.3 Intermediate test results
15/05/2014 123 Version 1.3
Figure 75: Number of successful call-backs (KPI_021) (GR)
Figure 76: Number of successful call-backs per MNO (KPI_021) (GR)
In total there were 330 attempts for call-backs, so the success rates of call-backs (KPI_022) was high.
D4.3 Intermediate test results
15/05/2014 124 Version 1.3
Figure 77: Success rate of call-backs (KPI_022) (GR)
Figure 78: Success rate of call-backs per MNO (KPI_022) (GR)
Conclusions 7.5.3
The KPI-002b was rather low, for both regions and for all three MNOs. This should be rather
due to low mobile network coverage at some locations, in which cases the eCall could not
be sent to the PSAP.
The success rate of received MSDs (KPI_003) was satisfactory for both regions and all three
MNOs.
The success rate of established voice transmissions (KPI_006) was satisfactory for both
regions and all three MNOs.
The success rate of correct MSDs (KPI_004) is a bit low. The main cause should be the low
GPS signal coverage in specific locations, which was clearly noted during the trials. Still it
was noted that the IVS failed to store log files in a significant number of eCalls. It has to be
checked further, if the IVS log files, which were used for calculating this KPI, were correct in
general.
D4.3 Intermediate test results
15/05/2014 125 Version 1.3
It was noted that the system was sensitive in bad weather conditions, as in cloudy days the
number of problematic eCalls was higher in the same area than in days with good weather
conditions. On such occasions, there were problems in voice connection; no voice was
transmitted to the PSAP or to the vehicle, and problems in the correct MSD transmission, in
which cases the PSAP operator had to request a new MSD.
Recommendations 7.5.4
Based on the results of first phase, the following must be further analysed in Phase 2:
Further analyse specific times in the eCall chain, so as to identify the reasons for delays.
Possibly perform eCalls with the eCall discriminator software and check how this affects
KPIs, especially those relevant to time.
Additional tests focusing on bad weather conditions and on locations with low GSM/GPS
signal and how they affect the KPIs.
Solve the synchronisation problem between IVS and PSAP, so that all time-related KPIs can
be reliably calculated from the log files.
Check the IVS logging and ensure that they are correct.
D4.3 Intermediate test results
15/05/2014 126 Version 1.3
7.6 Italy
Preliminary tests 7.6.1
Preliminary tests have been performed in order to evaluate the Inband modem communication and
the MSD transmission between IVS and Varese PSAP. For these tests a fixed long number for the
PSAP has been used.
From the list with recommended KPIs, KPI 7 has not been measured because on IVSs side it is
impossible to acquire low level mobile modem related data. KPI 13 has not been measured mainly
because it was not of our interest to check or verify GPS related information with IVSs still in
prototype version; the major Italian Pilot goal is to check and test the overall eCall chain.
89 manual calls received and with voice connection:
75 calls with MSD correctly received and decoded
14 calls without MSD
6 different IVS calling (2 Italian SIMs in roaming, 2 SIMs from UK, 2 SIMs from Luxemburg)
For these tests there wasn’t the possibility to register the log files from IVSs; for this reason the test
results are based on PSAP log data:
Average MSD reception time: 11.9 sec.
Min MSD reception time: 6 sec
Max reception MSD: 35 sec
Varese pilot tests 7.6.2
The eCall complete chain has been validated in Varese Area with 112 calls and eCall discriminator
management by the Italian Mobile Operator.
D4.3 Intermediate test results
15/05/2014 127 Version 1.3
Figure 79: Italian Pilot in Varese Area (IT)
Two different IVSs have been tested and their log files have been stored.
Manual activated calls: 38
MSD received correctly by PSAP: 33
Min PSAP MSD reception time: 5sec
Max PSAP MSD reception time: 18sec
The voice channel has been established between driver and PSAP operator as soon as MSD has been
received. In some cases the Operator couldn’t take in charge the call, probably due to other real
incoming call, and the MSD was transmitted again by the IVS. This problem will be analysed during
the forthcoming tests.
D4.3 Intermediate test results
15/05/2014 128 Version 1.3
Figure 80: PSAP log file (IT)
The following KPIs have been evaluated for IVS1 and IVS2:
D4.3 Intermediate test results
15/05/2014 129 Version 1.3
Figure 81: IVS1 distribution (IT)
Figure 82: IVS2 Distribution (IT)
Other tests have been performed in November and December; the previous IVSs were used together
with a third IVS. Again some of the analysed data are difficult to be interpreted, specially the success
ID of test set:
Result Unit Result Unit
KPI_001a Number of automatically initiated eCalls 0 - 0 -
KPI_001b Number of manually initiated eCalls 21 - 11 -
KPI_002a Success rate of completed eCalls using 112 % %
KPI_003 Success rate of received MSDs 100 % 100 %
KPI_004 Success rate of correct MSDs 80,95238 % 92,30769 %
KPI_005 Duration until MSD is presented in PSAP 8,411765 s 11,58333 s
KPI_006 Success rate of established voice transmissions 57,14286 % 100 %
KPI_008 Time for call establishment 3,238095 s 23,72727 s
KPI_015 Success rate of VIN decoding with EUCARIS - % - %
KPI_017 Dispatch time of incident data to rescue forces - % - %
KPI_021 Number of successful call-backs - - - -
KPI_022 Success rate of call-backs - % - %
KPI_028 Number of cross-border tests - - - -
Name of KPI
Combination of
IVS/MNO/PSAP:
Combination of
IVS/MNO/PSAP:
11/1/1 2/1/1
0
2
4
6
8
10
12
14
16
18
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
IVS1 - MSD Presentation Time
MSD Presentation Time
0
2
4
6
8
10
12
14
1 3 5 7 9 11 13 15 17 19 21
IVS1 - Establishment call time
Establishment call time
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8 9 10 11 12
IVS2 - MSD Presentation Time
Serie1
0
5
10
15
20
25
30
35
40
45
1 2 3 4 5 6 7 8 9 10 11
IVS2 - Establishment call time
Serie1
D4.3 Intermediate test results
15/05/2014 130 Version 1.3
rate of the completed eCall; this because the information logged in the IVSs state that the voice
channel hasn’t been correctly opened after MSD transmission, but is not possible to understand the
reason from the PSAP log where this information is not recorded yet. During the 2013 test campaign
more data will be recorded in log files.
Further analysis on test results will be performed before 2013 test campaign start in order to resolve
all pending issues on data interpretation.
The following KPIs have been evaluated for IVS1 and IVS3:
Figure 83: IVS1 Distribution (IT)
ID of test set:
Result Unit Result Unit
KPI_001a Number of automatically initiated eCalls 0 - 0 -
KPI_001b Number of manually initiated eCalls 33 - 13 -
KPI_002a Success rate of completed eCalls using 112 % %
KPI_003 Success rate of received MSDs 100 % 100 %
KPI_004 Success rate of correct MSDs 100 % 100 %
KPI_005 Duration until MSD is presented in PSAP 7,30303 s 8,307692 s
KPI_006 Success rate of established voice transmissions 75,75758 % 84,61538 %
KPI_008 Time for call establishment 3,714286 s 8,153846 s
KPI_015 Success rate of VIN decoding with EUCARIS - % - %
KPI_017 Dispatch time of incident data to rescue forces - % - %
KPI_021 Number of successful call-backs - - - -
KPI_022 Success rate of call-backs - % - %
KPI_028 Number of cross-border tests - - - -
Name of KPI
Combination of
IVS/MNO/PSAP:
Combination of
IVS/MNO/PSAP:
21/1/1 3/1/1
0
2
4
6
8
10
12
14
16
18
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
IVS1 - MSD Presentation Time
Serie1
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
1 3 5 7 9 11 13 15 17 19 21 23 25 27
IVS1 - Establishment call time
Serie1
D4.3 Intermediate test results
15/05/2014 131 Version 1.3
Figure 84: IVS3 Distribution (IT)
The following KPIs have been evaluated for IVS2:
Figure 85: IVS2 Distribution (IT)
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10 11 12 13
IVS3 - MSD Presentation Time
Serie1
7,4
7,6
7,8
8
8,2
8,4
8,6
8,8
9
9,2
1 2 3 4 5 6 7 8 9 10 11 12 13
IVS3 - Establishment call time
Serie1
ID of test set:
Result Unit
KPI_001a Number of automatically initiated eCalls 0 -
KPI_001b Number of manually initiated eCalls 32 -
KPI_002a Success rate of completed eCalls using 112 93,75 %
KPI_003 Success rate of received MSDs 90,90909 %
KPI_004 Success rate of correct MSDs 90,90909 %
KPI_005 Duration until MSD is presented in PSAP 7,606061 s
KPI_006 Success rate of established voice transmissions 96,875 %
KPI_008 Time for call establishment 24,53125 s
KPI_015 Success rate of VIN decoding with EUCARIS - %
KPI_017 Dispatch time of incident data to rescue forces - %
KPI_021 Number of successful call-backs - -
KPI_022 Success rate of call-backs - %
KPI_028 Number of cross-border tests - -
Name of KPI
Combination of
IVS/MNO/PSAP:
22/1/1
0
2
4
6
8
10
12
14
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
IVS2 - MSD Presentation Time
Serie1
21,5
22
22,5
23
23,5
24
24,5
25
25,5
26
26,5
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
IVS2 - Establishment call time
Serie1
D4.3 Intermediate test results
15/05/2014 132 Version 1.3
Conclusions 7.6.3
All performed tests indicated that the end to end emergency call has been correctly developed and
works properly; in order to calculate HeERO KPIs it is needed an improvement in log data
management. Before the 2013 test campaign it is necessary to improve this capability.
Recommendations 7.6.4
In some cases preliminary test results were difficult to analyse. It seems that not all the time
counters were aligned between IVSs and PSAP: KPI related to MSD displayed at PSAP operator desk is
dependent on the fact that operator is available to manage the call and display the data. This
because the information logged in the PSAP log file was not complete. A deeper analysis on PSAP log
files requirements shall be performed before the 2013 test campaign.
D4.3 Intermediate test results
15/05/2014 133 Version 1.3
7.7 Romania
Purpose and Scope 7.7.1
This document contains the operating eCall tests conducted in the Romanian pan European eCall
pilot and presents a report of results from Phase 1 testing.
This report presents the results of the evaluation of the reduced set of the Romanian eCall Pilot
experimental and real data.
Structure of Document 7.7.2
The report consists of the following:
- Table with the measured KPI’s during the tests
- Short description of the conditions in which the tests were made
- Values measured on each KPI on a test session
- Time series diagrams of the values of relevant KPIs
- Conclusions regarding the technical aspects identified in the tests
- Conclusions regarding the operational aspects identified in the tests
Description of Equipment 7.7.3
The equipment that were used in the eCall Phase 1 testing and validation in Romania eCall Pilot is
described in the table.
Component Description
T-IVS Manufacturer Rohde&Schwarz Topex Commercial name of the product NA Hardware revision 0.320 Software revision 1.0 C-IVS Manufacturer Civitronic Commercial name of the product Ubiq eCall IVS Hardware revision NA Software revision NA MODEM PSAP Manufacturer Rohde&Schwarz Topex Commercial name of the product Qutex eCall Hardware revision pgtexF Software revision 4.3.88
Table 46: equipment used in the Romanian pilot site (RO)
Measured KPIs 7.7.4
ID of KPI Name of KPI
D4.3 Intermediate test results
15/05/2014 134 Version 1.3
The only recommended KPIs that weren't measured in the Romanian pilot site are KPI_028a (Number of cross-border tests) and KPI_028b (Number of interoperability tests). KPI_028a wasn't measured due to the fact that none of the neighbouring countries have a working eCall system or are participating in HeERO, therefore making it impossible for doing cross-borders tests. KPI_028b wasn't measured because interoperability tests weren't planned for the first operational phase, but it will be measured in the second operational phase.
KPI_001a Number of automatically initiated eCalls KPI_001b Number of manually initiated eCalls KPI_002a Success rate of completed eCalls using 112 KPI_002b Success rate of completed eCalls using long number KPI_003 Success rate of received MSDs KPI_004 Success rate of correct MSDs KPI_005 Duration until MSD is presented in PSAP KPI_006 Success rate of established voice transmissions KPI_007a Duration of voice channel blocking KPI_007b Duration of voice channel blocking: automatic retransmission of MSD KPI_008 Time for call establishment KPI_009 Accuracy of position KPI_013 Success rate of heading information KPI_014 Success rate of VIN decoding without EUCARIS KPI_015 Success rate of VIN decoding with EUCARIS KPI_016 Time for VIN decoding with EUCARIS KPI_019 Dispatch time of incident data to TMC KPI_020 Success rate of presented incident data in TMC KPI_021 Number of successful call-backs KPI_022 Success rate of call-backs KPI_023 GSM network latency KPI_024 112 national network latency KPI_025 112 operator reaction time
Table 47: KPIs measured in the Romanian pilot site (RO)
D4.3 Intermediate test results
15/05/2014 135 Version 1.3
Figure 86: Timestamps measured in the Romanian pilot site (RO)
D4.3 Intermediate test results
15/05/2014 136 Version 1.3
Testing and Validation Scenarios Description 7.7.5
7.7.5.1 Test sessions – Laboratory tests
We tested all hardware and software for deployment using laboratory environment to ensure that
eCall integration doesn’t affect the existing 112 applications. All laboratory tests were made in a
training environment which is identical with the live system. We used only T-IVS because it permits
to modify MSD field’s value and the B-number used.
L1 to L5 means a laboratory test session.
Code No of IVS units involved
No of IVS units in roaming
eCall initiation
No of tests made
No of tests measured
L1 5 0 A >100 4
L2 5 0 A >100 8
L3 5 0 A >100 11
L4 5 0 A >100 28
L5 5 0 A >100 64 Table 48: overview of laboratory tests (RO)
7.7.5.2 Test sessions – Real life tests
Tests were made using both Civitronic IVS (C-IVS) equipped with RDS - SIM card because the RDS
network is supporting the eCall flag and Topex IVS (T-IVS) equipped with VODAFONE, ORANGE,
COSMOTE, RDS – SIM cards. There were generated both manual and automatic eCalls by setting the
eCall flag to Automatic eCall or Manual eCall. We have tried to cover different period times (morning,
afternoon, evening, night) in 11 different counties (from 42 of Romania), covering mountain, hills,
plane, highways, national roads and rural roads.
R1 to R12 means a real environment test session.
Code IVS used No of IVS units involved
No of IVS units in roaming
eCall initiation
No of tests made
No of tests measured
R1 T-IVS 5 0 A/M >100 32
R2 T-IVS 3 0 A/M >100 32
R3 T-IVS 5 0 A/M >100 32
R4 T-IVS 5 0 A/M >100 32
R5 T-IVS 3 0 A/M >100 32
R6 T-IVS 4 0 A/M >100 32
R7 T-IVS 2 0 A/M >100 32
R8 C-IVS 2 0 A/M >100 26
R9 C-IVS 2 0 A/M >100 55
R10 C-IVS 2 0 A/M >100 15
R11 C-IVS 2 0 A/M >100 20
R12 C-IVS 2 0 A/M >100 50
D4.3 Intermediate test results
15/05/2014 137 Version 1.3
R13 C-IVS 2 0 A/M >100 27
R14 C-IVS 2 0 A/M >100 35
R15 C-IVS 2 0 A/M >100 40
R16 T-IVS/ C-IVS
2 0 A/M >25 0
R17 T-IVS/ C-IVS
2 0 A/M >25 0
R18 T-IVS/ C-IVS
2 0 A/M >25 0
Table 49: overview of field tests (RO)
The test R16, R17 and R18 were made on a test cell in MNO Orange for testing eCall flag during
September - October 2012. We didn’t measure KPI’s.
7.7.5.3 Test sessions – Traffic Management Centre interface
Tests were done for the interface between the 112 PSAP and the Traffic Management Centre. Tests
were done for different scenarios:
Code Scenario
T1 Synchronizing the open cases between the TMC and 112 PSAP T2 Updating the cases presented in the TMC: creating new cases, modifying existing cases,
closing old cases T3 Positioning of the case on the map in the TMC application T4 Sending the correct data through the web interface
Table 50: overview of TMC tests (RO)
The time needed for the data to reach the TMC from the moment it was sent from the 112 PSAP was
measured. The average time for receiving data in the TMC was 23 s out of 96 tests, with a minimum
value of 10 s and a maximum value of 55 s.
Test Session Analysis 7.7.6
7.7.6.1 Laboratory sessions
a. Session L1 with T-IVS
In this session we focus on testing the PSAP modem interface with MSD decoder application. Value
measured was independent from the GSM network used because we generate MSD data directly
from PSAP modem.
KPI
measured
Test 1 Test 2 Test 3 Test 4
Value units Value units Value units Value units
KPI_007 9.4 s 10.7 s 8.5 s 9 s
Table 51: Romanian results for KPI_007 during session L1
D4.3 Intermediate test results
15/05/2014 138 Version 1.3
Figure 87: results for KPI_007 during session L1 (RO)
b. Session L2 with T-IVS
In this session we focus on testing the GSM and national network latency. Average value is made on
2 calls/ operator.
KPI
measured
RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_023 2.2 s 3 s 1.9 s 2.5 s
KPI_024 1.1 s 0.8 s 1.2 s 1 s
9,4
10,7
8,5 9
Test1 Test2 Test3 Test4
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 139 Version 1.3
Figure 88: results for KPI_023 during session L2 (RO)
Figure 89 results for KPI_024 during session L2 (RO)
c. Session L3 with T-IVS
In this session we analysed the per cent of MSD received versus correct MSD received.
KPI
measured
RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_003 98 % 97 % 100 % 98 %
KPI_004 96 % 93 % 95 % 90 %
2,2
3
1,9
2,5
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
1,1
0,8
1,2
1
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 140 Version 1.3
Table 52: results for KPI_003 and KPI_004 during session L3 (RO)
d. Session L4 with T-IVS
In this session we made a number of 32 eCalls (KPI_001a = 14, KPI_001b = 14).
KPI
measured
RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002b 100 % 100 % 100 % 100 %
KPI_003 97 % 95 % 99 % 98 %
KPI_004 82 % 80 % 73 % 81 %
KPI_005 24.2 s 26 s 23.9 s 25 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007a 14.3 s 15 s 13.8 s 15.7 s
KPI_008 4 s 4.5 s 5 s 5.8 s
Table 53: results for KPI_003 and KPI_004 during session L4 (RO)
e. Session L5 with T-IVS
In this session we made a number of 64 eCalls (KPI_001a = 44, KPI_001b = 20) and we also tested the
call-back function (KPI_021=12).
KPI
measured
RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002b 100 % 100 % 100 % 100 %
KPI_003 99 % 97 % 100 % 98 %
KPI_004 94 % 93 % 84 % 86 %
KPI_005 25.2 s 23.4 s 25.9 s 21.2 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007a 14.4 s 15 s 13.7 s 16.2 s
KPI_008 3.3 s 4.1 s 4 s 3.8 s
KPI_023 2.1 s 2.8 s 3.1 s 1.9 s
D4.3 Intermediate test results
15/05/2014 141 Version 1.3
KPI_024 1.2 s 1.2 s 1.8 s 1.9 s
Table 54: results for KPI_003 and KPI_004 during session L5 (RO)
7.7.6.2 Real life sessions
a. Session R1 with T-IVS
In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long
number and SIM cards from different GSM operators. We made 12 call-backs with success rate of
100%. KPI_007 and KPI_008 are presented also in diagram.
KPI measured
RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002a 100 % 0 % 0 % 0 %
KPI_002b 0 % 100 % 100 % 100 %
KPI_003 100 % 100 % 100 % 100 %
KPI_004 100 % 100 % 100 % 100 %
KPI_005 23.35 s 24.14 s 25.28 s 27 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007 14.3 s 15.9 s 15.3 s 14 s
KPI_008 2.78 s 3.42 s 3.57 s 4.35 s
KPI_013 100 % 100 % 100 % 100 %
KPI_014 100 % 100 % 100 % 100 %
KPI_022 100 % 100 % 100 % 100 %
KPI_023 1.85 s 3.14 s 2.42 s 3.42 s
KPI_024 1.28 s 1.28 s 1.28 s 1.5 s
KPI_025 3.85 s 3.42 s 3.57 s 2.14 s
Table 55: test results during session R1 (RO)
14,3
15,9
15,3
14
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 142 Version 1.3
Figure 90: results for KPI_007 during session R1 (RO)
Figure 91: results for KPI_008 during session R1 (RO)
b. Session R2 with T-IVS
In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long
number and SIM cards from different GSM operators. We made 12 call-backs with success rate of
100%. KPI_007 and KPI_008 are presented also in diagram.
KPI measured RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002a 100 % 0 % 0 % 0 %
KPI_002b 0 % 100 % 100 % 100 %
KPI_003 100 % 100 % 100 % 100 %
KPI_004 100 % 100 % 100 % 100 %
KPI_005 23.42 s 24.28 s 25.42 s 24.85 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007 14.9 s 16.6 s 16.3 s 16.1 s
KPI_008 2.85 s 3.42 s 3.28 s 3.42 s
KPI_013 100 % 100 % 100 % 100 %
KPI_014 100 % 100 % 100 % 100 %
KPI_022 100 % 100 % 100 % 100 %
KPI_023 1.78 s 3.57 s 2.42 s 3.57 s
KPI_024 1.28 s 1.28 s 1.28 s 1.14 s
KPI_025 3.57 s 3.42 s 3 s 3.42 s
Table 56: test results during session R2 (RO)
2,78
3,42 3,57
4,35
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 143 Version 1.3
Figure 92: results for KPI_007 during session R2 (RO)
Figure 93: results for KPI_008 during session R2 (RO)
c. Session R3 with T-IVS
In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long
number and SIM cards from different GSM operators. We made 15 call-backs with success rate of
100%. KPI_005 and KPI_007 are presented also in diagram.
KPI measured
RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average Units
KPI_002a 100 % 0 % 0 % 0 %
14,9
16,6
16,3 16,1
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
2,85
3,42 3,28
3,42
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 144 Version 1.3
KPI_002b 0 % 100 % 100 % 100 %
KPI_003 100 % 100 % 100 % 100 %
KPI_004 100 % 100 % 100 % 100 %
KPI_005 23.42 s 25 s 26 s 26.14 S
KPI_006 100 % 100 % 100 % 100 %
KPI_007 16.8 s 17 s 15 s 17 S
KPI_008 2.85 s 3.57 s 3.28 s 4.57 S
KPI_013 100 % 100 % 100 % 100 %
KPI_014 100 % 100 % 100 % 100 %
KPI_022 100 % 100 % 100 % 100 %
KPI_023 1.85 s 3.57 s 2.42 s 3.28 S
KPI_024 1.28 s 1.14 s 1.28 s 1.14 S
KPI_025 3.57 s 3.14 s 3 s 2.14 S
Table 57: test results during session R3 (RO)
Figure 94: results for KPI_005 during session R3 (RO)
23,42
25
26 26,14
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 145 Version 1.3
Figure 95 results for KPI_007 during session R3 (RO)
d. Session R4 with T-IVS
In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long
number and SIM cards from different GSM operators. We made 8 call-backs with success rate of
100%. KPI_005 is presented also in diagram.
KPI measured RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002b 0 % 100 % 100 % 100 %
KPI_003 100 % 100 % 100 % 100 %
KPI_004 100 % 100 % 100 % 100 %
KPI_005 22.42 s 25.71 s 25.42 s 27 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007 14.9 s 16.6 s 16.4 s 16.3 s
KPI_008 2.85 s 4 s 3.35 s 4.35 s
KPI_013 100 % 100 % 100 % 100 %
KPI_014 100 % 100 % 100 % 100 %
KPI_021 2 - 2 - 2 - 2 -
KPI_022 100 % 100 % 100 % 100 %
KPI_023 2.21 s 3.85 s 3.35 s 3.42 s
KPI_024 1.5 s 1.42 s 1.28 s 1.5 s
KPI_025 2.07 s 1.85 s 2.21 s 2.14 s
Table 58: Romanian test results during session R4
16,8 17
15
17
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 146 Version 1.3
Figure 96: results for KPI_005 during session R4 (RO)
e. Session R5 with T-IVS
In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long
number and SIM cards from different GSM operators. We made 12 call-backs with success rate of
100%. KPI_005 are presented also in diagram.
KPI measured RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002b 0 % 100 % 100 % 100 %
KPI_003 100 % 100 % 100 % 100 %
KPI_004 100 % 100 % 100 % 100 %
KPI_005 23.71 s 25.42 s 25.42 s 26.71 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007 14.8 s 16.5 s 16 s 16 s
KPI_008 3.42 s 4.14 s 3.14 s 4.21 s
KPI_013 100 % 100 % 100 % 100 %
KPI_014 100 % 100 % 100 % 100 %
KPI_022 100 % 100 % 100 % 100 %
KPI_023 2.57 s 3.42 s 3.71 s 3.57 s
KPI_024 1.57 s 1.21 s 1.28 s 1.21 s
KPI_025 1.92 s 1.28 s 1.71 s 1.57 s
Table 59: test results during session R5 (RO)
22,42
25,71 25,42 27
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 147 Version 1.3
Figure 97: results for KPI_005 during session R5 (RO)
f. Session R6 with T-IVS
In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long
number and SIM cards from different GSM operators. We made 4 call-backs with success rate of
100%. KPI_005 are presented also in diagram.
KPI measured
RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002b 0 % 100 % 100 % 100 %
KPI_003 100 % 100 % 100 % 100 %
KPI_004 100 % 100 % 100 % 100 %
KPI_005 23.57 s 24.57 s 25.57 s 26.42 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007 15.1 s 16.4 s 16.5 s 16.4 s
KPI_008 3.64 s 4.14 s 3.57 s 4.21 s
KPI_013 100 % 100 % 100 % 100 %
KPI_014 100 % 100 % 100 % 100 %
KPI_022 100 % 100 % 100 % 100 %
KPI_023 2.71 s 3.57 s 3.92 s 3.57 s
KPI_024 1.28 s 1.21 s 1.21 s 1.21 s
KPI_025 1.42 s 1.57 s 2 s 1.42 s
Table 60: test results during session R6 (RO)
23,71
25,42 25,42
26,71
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 148 Version 1.3
Figure 98: results for KPI_005 during session R6 (RO)
g. Session R7 with T-IVS
In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long
number and SIM cards from different GSM operators. We made 5 call-backs with success rate of
100%. KPI_005 are presented also in diagram.
KPI measured RDS ORANGE COSMOTE VODAFONE
Average units Average units Average units Average units
KPI_002b 0 % 100 % 100 % 100 %
KPI_003 100 % 100 % 100 % 100 %
KPI_004 100 % 100 % 100 % 100 %
KPI_005 23.57 s 24.57 s 19.28 s 27.57 s
KPI_006 100 % 100 % 100 % 100 %
KPI_007 15.1 s 16.4 s 16.5 s 16.4 s
KPI_008 3.64 s 4.14 s 4.71 s 4.42 s
KPI_013 100 % 100 % 100 % 100 %
KPI_014 100 % 100 % 100 % 100 %
KPI_022 100 % 100 % 100 % 100 %
KPI_023 2.71 s 3.57 s 3.42 s 3.14 s
KPI_024 1.28 s 1.21 s 1.28 s 1.14 s
KPI_025 1.42 s 1.57 s 5.14 s 2.14 s
Table 61: test results during session R7 (RO)
23,57
24,57
25,57
26,42
RDS ORANGE COSMOTE VODAFONE
kpi v
alu
e m
easu
red
(se
c)
D4.3 Intermediate test results
15/05/2014 149 Version 1.3
Figure 99 results for KPI_005 during session R7 (RO)
[10] Table of critical values of the F-distribution: NIS
D4.3 Intermediate test results
15/05/2014 181 Version 1.3
[11] Hughes, I G and T P A Hase, Measurements and their Uncertainties: A Practical Guide to Modern
Error Analysis, Oxford University Press, Inc., New York, NY, 2010, ISBN 978-0199566334.
[12] Maindonald, J and W J Brown, Data Analysis and Graphics Using R - an Example-Based Approach
(3rd edition), Cambridge University Press, Cambridge, UK, 2010, ISBN 978-0521762939.
[13] Ott, R L and M Longnecker, An Introduction to Statistical Methods and Data Analysis (5th ed),
Duxbury, Thomson Learning, Inc., Pacific Grove, CA, 2000, ISBN 978-0534251222.
[14] R Development Core Team (2010). R: A language and environment for statistical computing. R
Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, available at:
http://www.R-project.org, accessed on 1 June 2011.
[15] Wikipedia, Cochran's C-test, available at: http://bit.ly/OWuldq.
[16] Wikipedia, Grubbs' test for outliers, available at: http://bit.ly/NluMe6.
[17] National Institute of Standards and Technology (NIST), Detection of Outliers, NIST/SEMATECH e-
Handbook of Statistical Methods, 2012, available at: http://1.usa.gov/RXP1QU
D4.3 Intermediate test results
15/05/2014 182 Version 1.3
7.9 The Netherlands
In General 7.9.1
The Netherlands is participating in the European HeERO-project. Commission by the authorities,
Grontmij has executed several drive tests to evaluate 4 IVS-systems and 3 MNO’s. In order to
appreciate the companies participated in these tests, the results are made anonymous. This prevents
that the companies will be judged on their prototypes and gives them the opportunity to improve
their products.
The anonymous IVS 1 and 2 is the same product from the same manufacturer. This IVS is doubled
tested because two prototypes were available and the test system accommodated an extra IVS.
Work package 4.2 describes which KPI should be tested by each participating country. Next table
shows which KPIs were included in the drive tests.
KPI_001a: Number of automatically initiated eCalls Tested
KPI_001n: Number of manually initiated eCalls Tested
KPI_002a: Success rate of completed eCalls using 112 Not tested, during the test the MNOs
were not equipped to use to eCall-flag
KPI_002b: Success rate of completed eCalls long number Tested
KPI_003: Success rate of received MSDs Tested
KPI_004: Success rate of correct MSDs Tested
KPI_005: Duration until MSD is presented in PSAP Tested
KPI_006: Success rate of established voice transmission Tested
KPI_007a: Duration of voice channel blocking Tested
KPI_008: Time for call establishment Tested
KPI_009: Accuracy of position Tested
KPI_013: Success rate of heading information Tested
KPI_015: Success rate of VIN decoding with EUCARIS Not tested, EUCARIS not part of the
Dutch test setup
KPI_016: Time for decoding VIN with EUCARIS Not tested, EUCARIS not part of the
Dutch test setup
D4.3 Intermediate test results
15/05/2014 183 Version 1.3
KPI_019: Dispatch time of incident data to TMC Tested
KPI_020: Success rate of presented incident data in TMC Tested
KPI_022: Success rate of call-backs Not tested, already tested by J. Van
Hattem (Dutch HeERO (static)testing,
February 23/24 2012)
Table 83: Included KPIs in the drive tests (NL)
Not all of the recommended KPI´s The Netherlands have committed to test could be tested. This is caused by the fact that no MNO has already implemented the eCall flag. As a result the Dutch pilot could only perform the tests by using a long number instead of using the number 112. Only if a MNO implements the eCall Flag in 2013, The Netherlands can test KPI 002a (Success rate of completed eCalls) and KPI_028a (number of cross border tests). The Netherlands have started with interoperability tests (KPI_028b) and will perform more of these in Phase 2.
Methodology 7.9.2
7.9.2.1 Drive tests
This report describes the ‘drive test’ that was conducted as part of the HeERO-NL project. A drive test
is a test in which a specific route is driven, during which several eCalls are triggered. It is almost
impossible to execute such a test reliably without using specific test equipment. Therefor
Rijkswaterstaat has developed a specific test device that is able to do automatic triggering of
(currently) at maximum four (4) in-vehicle systems (IVSs). This device is assembled inside a ski-box
that can be mounted onto a car. In the remainder of this document this test device will be referred
to as ‘Dutch First Automated eCall Test Setup’ (D-FACTS)
Figure 122: the ski-box attached to a test-vehicle (left) and D-FACTS inside the ski box (left) (NL)
D-FACTS are capable of automatically triggering both ‘manual’ eCalls and ‘automatic’ eCalls (provided
the IVSs offer a triggering for both variants). The triggering can be location based, i.e. triggers for
D4.3 Intermediate test results
15/05/2014 184 Version 1.3
specific locations, or time based: a trigger each X minutes. In this specific drive test eCalls were
triggered each minute, resulting in four eCalls per minute.
The eCalls were, when received at the PSAP, routed to a test setup which received the MSD and
automatically switched to a voice connection. After this the call was disconnected by the PSAP side.
In order to measure the correctness of the GPS-position determined by each IVS, the vehicle was
equipped with a separate commercial available GPS-logger. This system logged the GPS-position
every 10 meters.
Three mobile network operators (MNOs) participate in the test. As the drive test was carried out
over several days, each day different combinations of IVSs and MNOs were tested.
Day 1 Day 2 Day 3 Day 4
IVS 1 MNO 2 MNO 2 MNO 3 MNO 3
IVS 2 MNO 2 MNO 1 MNO 1 MNO 1
IVS 3 MNO 1 MNO 2 MNO 3 MNO 2
IVS 4 MNO 1 MNO 2 MNO 2 MNO 2
Table 84: Combinations of MNOs and IVSs tested during the drive test (NL)
The drive test took place on four days: August 31, September 4, 5 and 7. The driver of the vehicle
drove a predetermined route through the Rotterdam-Rijnmond region in the Netherlands (Figure
123). This route was designed so that specific situations would be encountered that might have an
effect on the GPS accuracy (urban canyon) or on the mobile reception:
Low mobile coverage (lage dekking)
Rotterdam Port (haven)
Rotterdam-The Hague Airport (vliegveld)
Rotterdam city centre (stedelijk gebied)
High voltage cables and pylons (hoogspanning)
Tunnel (tunnel)
D4.3 Intermediate test results
15/05/2014 185 Version 1.3
Figure 123: the route of the drive test (NL)
7.9.2.2 Data collection and processing
The systems active during the drive test logged the important timestamps and the sent and received
data. In Table 85, per system the relevant data from the loggings are described.
System Logging
D-FACTS Telephone number, time eCall triggering per IVS,
PSAP Telephone number, CISID, time eCall reception, time back to speech, time voice
connection, time end eCall, MSD
TMC CISID, time MSD received, MSD
GPS-
logger
Time log, GPS-location
Table 85: Relevant logging per system (NL)
To evaluate individual eCalls, and to calculate the KPIs, the loggings from the systems had to be
matched. As soon as an eCall is received in the PSAP system a unique ID is tagged to it which can be
found in the other systems (emergency services and traffic management centre) as well. That ID can
D4.3 Intermediate test results
15/05/2014 186 Version 1.3
however not be used to match the eCall triggering (D-FACTS) with receiving the call in the PSAP
system.
It is possible to identify the IVS by its phone number, so that in combination with the timestamp of
triggering the eCall should suffice to match the loggings from D-FACTS, PSAP and the GPS-logger.
Unfortunately it turned out that the PSAP system had not been synchronised correctly. This meant
that the timestamp of triggering the eCall (D-FACTS) could not be related to the timestamp the eCall
was received by the PSAP. To overcome this deficit a time window of one minute was used to match
the individual eCalls between D-FACTS and PSAP.
After matching all loggings were combined to one dataset.
Figure 124: The process of linking the loggings of the systems (NL)
There are two possibilities if an eCall triggered by D-FACTS cannot be matched with logging in the
PSAP system. Either the eCall is not received by the PSAP (e.g. no mobile network coverage), or the
previous eCall was not ended. D-FACTS have no knowledge about calls in progress on the IVSs. So it
can trigger a new eCall while the IVS is still in communication with the PSAP. Conformant to the
specification the IVS will ignore the triggering, but D-FACTS will log it anyhow. Such situations have
been filtered from the loggings.
7.9.2.3 Outliers
To detect mismatches of the loggings in the dataset, or to detect error in loggings themselves,
statistical analysis for outliers is required. An outlier is a value in a dataset which differs greatly from
the other values in the same dataset, that the assumption is that it is an error and not a part of the
distribution of the dataset.
For every IVS, outliers are detected using the Turkey’s Outlier Filter. The quartiles (Q1 and Q3) of the
dataset are used to define a lower and upper limit. Values outside those limits are assumed to be an
outlier.
D4.3 Intermediate test results
15/05/2014 187 Version 1.3
lower limit = Q1 – 1, 5 * (Q3 – Q1)
upper limit = Q1 + 1, 5 * (Q3 – Q1)
Description of KPI’s 7.9.3
Several KPIs have been defined within work package 4.2 of the HeERO project. Not all are relevant
for the national situations, so each participating country will perform a subset of these. In the Dutch
HeERO project the KPIs described in this chapter have been calculated.
7.9.3.1 KPI_001a: Number of automatically initiated eCalls
This KPI measures the total number of ‘automatic’ eCalls. This simply is the number of ‘automatic’
eCalls that has been triggered by D-FACTS and is present in the dataset (not filtered).
7.9.3.2 KPI_001b: Number of manually initiated eCalls
This KPI measures the total number of ‘manual’ initiated eCalls. This simply is the number of ‘manual’
eCalls that has been triggered by D-FACTS and is present in the dataset (not filtered)
7.9.3.3 KPI_002b: Success rate of completed eCalls using long number
This KPI describes the relationship between the numbers of initiated eCalls versus the number of
successful eCalls while the long number of a PSAP is used as a telephone number for the emergency
call.
An eCall is considered to be successful when the voice connection between the IVS and PSAP is
established, and the received MSD in the PSAP contains a valid GPS-location, direction and VIN-
number.
7.9.3.4 KPI_003: Success rate of received MSDs
This KPI describes the relationship between the numbers of initiated eCalls versus the number of
received MSD’s in the PSAP. A MSD is considered as received when a valid GPS-location, the vehicle
direction or the VIN-number is presented in the PSAP.
7.9.3.5 KPI_004: Success rate of correct MSDs
This KPI describes the relationship between the numbers of initiated eCalls versus the number of
correctly received MSD’s in the PSAP. A MSD is considered as correctly received when a valid GPS-
location, the vehicle direction and the VIN-number are presented in the PSAP.
7.9.3.6 KPI_005: Duration until MSD is presented in PSAP
This KPI describes the duration from the initiation (automatically or manually) of an eCall to the
presentation of the MSD content in the PSAP.
Only eCalls with a correct MSD are included in this calculation.
7.9.3.7 KPI_006: Success rate of established voice transmissions
This KPI describes the relation between the number of initiated voice transmissions versus the
number of successfully established voice transmissions between the vehicle and the PSAP.
D4.3 Intermediate test results
15/05/2014 188 Version 1.3
A voice transmission is initiated when the PSAP finishes the data transmission and starts the voice
transmission. A voice transmission is considered to be successful when the eCall is received in and
accepted by the voice taker application
7.9.3.8 KPI_007a: Duration of voice channel blocking
This KPI represents the time the transmission of the MSD blocks the voice channel. The time the
voice channel is blocked is defined as the moment from the eCall reception in the PSAP until the
moment the eCall is send to the voice taker application.
Only eCalls with a correct MSD are included in this calculation.
7.9.3.9 KPI_008: Time for call establishment
This KPI represents the time necessary for the eCall setup. The time needed for the eCall setup is
defined as the moment from the eCall imitation at the IVS until the moment the eCall is received at
the PSAP.
7.9.3.10 KPI_009: Accuracy of position
This KPI describes the difference between the reported position by the IVS in the MSD and the actual
position of the vehicle (GPS-logger). As described in section 2.2, the time of the eCall imitation is
matched with the GPS-logger to determine the actual position of the vehicle.
7.9.3.11 KPI_013: Success rate of heading information
This KPI describes the accuracy of the heading information of the vehicle reported in the MSD. The
actual heading information of the vehicle is calculated using the two most relevant points (based on
the timestamp) of the GPS-logger. These points never differ more than 10 meters and offer a good
basis to calculate the heading. The heading information in the MSD is correct if the absolute
deviation from the calculation heading is no more than 37.5 degrees.
7.9.3.12 KPI_019: Dispatch time of incident data to TMC
This KPI refers to the time it takes to inform the TMC operators after the initiation of the eCall. This
KPI is calculated using the moment the eCall is initiated by the IVS and the moment the MSD is
received in the TMC-application.
7.9.3.13 KPI_020: Success rate of presented incident data in TMC
This KPI describes the number of times the incident data received in the TMC is accurate and
received in an acceptable time (1 minute). This KPI is calculated by comparing the MSD in the PSAP
with the incident data received by the TMC, and the time it takes from initialising the eCall and the
reception of the data in the TMC.
Data selection 7.9.4
To calculate the in chapter 3 presented KPI’s, the data from the loggings needs to be analysed. This
chapter describes what data is used, which subsets are created and how outliers are detected.
D4.3 Intermediate test results
15/05/2014 189 Version 1.3
7.9.4.1 Data used
Figure 125 shows what data from the different loggings is used for calculating the KPIs.
Figure 125: Overview of the data used for analysing the KPIs (NL)
7.9.4.2 Subsets
In Figure 126 the different subsets for calculating the KPI’s are presented.
Telefone
number
KPI_001a
Number of automatic eCalls
KPI_001b
Number of manual eCalls
Telefone
number
KPI_002b
Success rate completed eCalls
KPI_001a /
KPI_001b
Time voice
connection
GPS-location
Direction
VIN
KPI_003
Success rate received MSDs
KPI_004
Success rate correct MSDs
KPI_005
Duration MSD presented in PSAP
KPI_006
Success rate established voice
connections
KPI_007a
Duration voice channel blocking
KPI_008
Time for call establishment
KPI_009
Accuracy of position
KPI_013
Success rate of heading information
KPI_019
Dispatch time of incident data to TMC
KPI_020
Success rate presented incident data
TMC
KPI_001a /
KPI_001b
GPS-location
Direction
VIN
KPI_001a /
KPI_001b
GPS-location
Direction
VIN
KPI_001a /
KPI_001b
Time eCall
triggering
Time back to
speechMSD
KPI_001a /
KPI_001b
Time back to
speech
Time voice
connection
KPI_004Time eCall
reception
Time back to
speech
KPI_001a /
KPI_001b
Time eCall
triggering
Time eCall
reception
KPI_004 GPS position GPS position
Direction GPS positionKPI_004
KPI_004Time eCall
triggering
Time incident
data reception
KPI_019
GPS-location
Direction
VIN
GPS-location
Direction
VIN
Log
D-FACTSKPI
Log
PSAP
Log
GPS-logger
Log
TMC
D4.3 Intermediate test results
15/05/2014 190 Version 1.3
During the drive tests, D-FACTS registered the initialisation of 6901 eCalls. 5378 eCalls (78%) of these
can be matched with an eCall in the PSAP-logging. The remaining eCalls are not received by PSAP.
This can be explained by multiple causes:
A new eCall is initialised by D-FACTS, while the previous eCall is not finished yet. According the
specifications of the IVSs, the new eCall is not initialised by the IVS and therefore not received by
PSAP.
During the drive tests, multiple times an IVS crashed. If D-FACTS initialised a new eCall while the
IVS was crashed, the new eCall is not initialised by the IVS and therefore not received by PSAP.
There were some technical issues regarding the PSAP. Due to these issues, not all eCalls are
received by the PSAP.
In the analysis of the Dutch test setup, only the eCalls received by the PSAP and matched to D-FACTS
are taken into account. The remaining initialised eCalls by D-FACTS are subtracted from the data.
All initialised eCalls received by the PSAP started the transmission of the MSD. Unfortunately, not all
MSD’s were received by the PSAP. 4836 eCalls were received (90%), and 4351 eCalls were correct
(81%). From the correct eCalls the accuracy of GPS-location and the heading direction are calculated.
The completed MSD’s (including the incorrect ones) are used to determine the success rate of
establishing the voice connection from the vehicle to the call taker application. The same subset is
used to calculate the success rate of transferring the data to the TMC.
D4.3 Intermediate test results
15/05/2014 191 Version 1.3
Figure 126: Overview of the subsets used for analysing the KPI’s (NL)
7.9.4.3 Outliers
Some KPI’s are not arbitrary (yes or no), but refer for example to a time interval. For these KPI’s it is
important to detect outliers (section 2.3), in order to get a reliable analysis of the data. As described,
outliers are detected using the Turkey’s Outlier Filter.
Using the Turkey’s Outlier Filer, outliers are visualised with a boxplot (Appendix A). The median
together with the first and third quarter of the data are displayed with a box. 50% of all data is
located within this box. The outliers are displayed with dots above and under the limits of the filter.
Outliers are caused by different reasons. For example a mismatch of the loggings or a time
synchronisation error in the log files results in very high or low values. Using the outlier filter these
values are subtracted from the data.
Results drive test 7.9.5
In this chapter the results of the KPI’s are presented.
Initiated eCalls D-FACTS
6901
Matches D-FACTS <> PSAP
5378
MSD transmissions started
5378
MSD transmissions
completed
4836
KPI_001a automatic eCalls:
1587 (30%)
KPI_001b manual eCalls:
3791 (70%)
Containing:
- VIN: 4836
- Valid GPS: 4770
- Direction: 4351
- VIN & GPS & Direction: 4351
KPI_003 received MSD’s:
4836 (90%)
KPI_004 correct MSD’s:
4351 (81%)
KPI_009 accuracy of position:
131 m
KPI_013 Success rate of
heading information:
73%
Accepted eCalls call taker
application
4299
Containing:
- VIN: 4299
- Valid GPS: 4245
- Direction: 3837
- VIN & GPS & Direction: 3836
KPI_002b Completed eCalls:
3836 (71%)
KPI_006 Established voice
connections:
4299 (89%)
Received eCalls TMC
4352
Within 1 minute: 3505
MSD content correct: 3757
KPI_020 Successful incidentdata in
TMC:
3505 (81%)
D4.3 Intermediate test results
15/05/2014 192 Version 1.3
7.9.5.1 Overall results
Table 86 summarizes the overall results of the four days of driving tests. A total of 5.378 eCalls were
generated with four IVSs, three MNOs and one PSAP. 71% of those calls were “completed”, which
means that the MSD was received, its contents was presented to the PSAP operator and a voice path
was established. 19% of the received MSDs were incorrect, mostly due to incorrect location and/or
heading.
It takes, on average, 17 s starting from the initiation of the eCall until the moment the phone of the
PSAP-operator is ringing, 12 s to establish the call and 5 s to send the MSD.
ID of
test set:
Name of KPI
All IVSs
All MNOs
PSAP
Result Unit
KPI_001a Number of automatically initiated eCalls 1587 -
KPI_001b Number of manually initiated eCalls 3791 -
KPI_002b Success rate of completed eCalls using long number 71 %
KPI_003 Success rate of received MSDs 90 %
KPI_004 Success rate of correct MSDs 81 %
KPI_005 Duration until MSD is presented in PSAP 17 s
KPI_006 Success rate of established voice transmissions 89 %
KPI_007a Duration of voice channel blocking 5 s
KPI_008 Time for call establishment 12 s
KPI_009 Accuracy of position 131 m
KPI_013 Success rate of heading information 73 %
KPI_019 Dispatch time of incident data to TMC 15 s
KPI_020 Success rate of presented incident data in TMC 81 %
Table 86: Overall results of the drive test(NL)
D4.3 Intermediate test results
15/05/2014 193 Version 1.3
7.9.5.2 Results per IVS
Table 87 shows the results of the drive test per IVS. The IVSs score differently on the KPIs. IVS 1 and 2
have the highest success rate, while IVS 3 is the fastest. In general IVS 4 scores worst: it has a
relatively low success rate and it takes long before a call is established.
The GPS-accuracy of IVS 3 is not good. According to the specs, an IVS is supposed to save the GPS-
location at the moment the eCall is initialised. When the MSD is sent, which is a few seconds later
(after call setup (KPI_008) and modem synchronisation), that saved GPS-location should be included
in the MSD. IVS 3 sends the GPS-location of the vehicle at the moment the MSD is send to the PSAP.
If an accident is mimicked this behaviour is less noticeable, as the vehicle isn’t moving at that time.
The drive test however is conducted from a moving vehicle, which makes this error shown more
prominently.
The duration to establish an eCall differs per IVS (6-21 s). When the call is established the voice
channel is blocked (for sending the MSD) for the same amount of time (5-6 s) which indicates that
the speed of sending the MSD is the same for the IVSs.
IVS 4 has a low success rate for completing the eCalls (41%). This is mostly caused by the low success
rate of transmitting the MSD: 43% of the MSDs are not received by the PSAP. Compared to other IVSs
this percentage is significant lower, which indicates that this problem is caused by the IVS and not by
the PSAP or MNOs.
ID of
test set: IVS 1 IVS 2 IVS 3 IVS 4
All MNOs All MNOs All MNOs All MNOs
PSAP PSAP PSAP PSAP
Result Unit Result Unit Result Unit Result Unit
KPI_001a 843 - 744 - 0 - 0 -
KPI_001b 728 - 895 - 1278 - 890 -
KPI_002b 83 % 82 % 65 % 41 %
KPI_003 95 % 96 % 99 % 57 %
KPI_004 93 % 94 % 65 % 56 %
KPI_005 17 s 17 s 11 s 26 s
KPI_006 89 % 86 % 98 % 74 %
KPI_007a 5 s 5 s 5 s 6 s
D4.3 Intermediate test results
15/05/2014 194 Version 1.3
KPI_008 12 s 11 s 6 s 21 s
KPI_009 6 m 6 m 598 m 12 m
KPI_013 73 % 72 % 70 % 84 %
KPI_019 15 s 15 s 9 s 25 s
KPI_020 81 % 81 % 82 % 77 %
Table 87: Results of the drive test per IVS (NL)
7.9.5.3 Results per MNO
Table 88 shows the results of the drive tests per MNO. The most relevant KPI’s regarding the MNOs
are KPI_002b (success rate completed eCalls) and KPI_008 (time for call establishment).
Compared to the other MNOs, MNO 2 has a lower success rate of completed eCalls. This is partly
caused by success rate of transmitting MSD from the IVS to the PSAP (KPI_004) and partly by the
success rate of established voice transmissions (KPI_006).
By average, the time for the call establishment (KPI_008) is more or less the same for all MNOs.
ID of
test set: All IVSs All IVSs All IVSs
MNO 1 MNO 2 MNO 3
PSAP PSAP PSAP
Result Unit Result Unit Result Unit
KPI_001a 328 - 416 - 843 -
KPI_001b 1432 - 2051 - 308 -
KPI_002b 77 % 63 % 81 %
KPI_003 96 % 83 % 95 %
KPI_004 89 % 74 % 83 %
KPI_005 16 s 18 s 16 s
KPI_006 87 % 86 % 98 %
D4.3 Intermediate test results
15/05/2014 195 Version 1.3
KPI_007a 5 s 5 s 5 s
KPI_008 11 s 13 s 10 s
KPI_009 123 m 146 m 118 m
KPI_013 71 % 75 % 75 %
KPI_019 15 s 16 s 14 s
KPI_020 80 % 82 % 78 %
Table 88: Results of the drive test per MNO for all IVSs (NL)
The results in Table 88 might be influenced by distribution of the MNOs over the IVSs and days.
Section 7.9.5.2 learns that the IVSs perform differently on the KPI’s. If one IVS is overrepresented in
the testing of a MNO, the results may be influenced by this IVS and therefore the comparison with
other MNOs becomes invalid.
Table 89 shows the distribution of the MNOs over the IVSs. As IVS 1 and 2 are of the same type and
perform the same on the KPIs (section 7.9.5.2), table 12 shows the results per MNO only for IVS 1
and 2.
Table 90 shows there are some minor differences between the investigated MNOs. The time to
establish an eCall is equal for all MNOs. However, MNO 1 and 2 have a lower success rate on
establishing a voice connection between the vehicle and the PSAP compared to MNO 3. This results
in a lower success rate of completed eCalls.
IVS 1 IVS 2 IVS 3 IVS 4
MNO 1 0 1233 381 156
MNO 2 427 416 589 734
MNO 3 843 0 308 0
Table 89: Number of tests distributed per MNO and per IVS (NL)
ID of
test set: IVS 1+2 IVS 1+2 IVS 1+2
MNO 1 MNO 2 MNO 3
PSAP PSAP PSAP
D4.3 Intermediate test results
15/05/2014 196 Version 1.3
Result Unit Result Unit Result Unit
KPI_001a 328 - 416 - 843 -
KPI_001b 895 - 728 - 0 -
KPI_002b 78 % 81 % 90 %
KPI_003 96 % 97 % 93 %
KPI_004 94 % 94 % 93 %
KPI_005 17 s 17 s 17 s
KPI_006 82 % 86 % 97 %
KPI_007a 5 s 5 s 5 s
KPI_008 11 s 12 s 11 s
KPI_009 6 m 6 m 6 m
KPI_013 70 % 75 % 74 %
KPI_019 15 s 16 s 15 s
KPI_020 87 % 74 % 82 %
Table 90: Results of the drive test per MNO for IVS 1 and 2(NL)
7.9.5.4 Results per area-type
The route of the drive test was designed for encountering different areas with different
characteristics. Table 91 shows the results of the drive tests per area type.
The eCall chain performs differently in different circumstances. The success rate of establishing voice
transmission between the vehicle and the PSAP (KPI_006) is influenced by the area the vehicle is
driving in. Remarkable: the success rate is the highest in an area with low mobile coverage and the
lowest in city centre and near the airport.
Also the GPS-accuracy (KPI_009) depends on the area type. In the city centre and near the airport the
accuracy is much higher than in the other area. This is caused by the speed of the vehicle: IVS 3
reports the position of the vehicle when this MSD is send, and not the position when the eCall is
initialised. The lower the speed of the vehicle, the closer these positions is. This results in a better
accuracy of the GPS-location.
Regular Port Low coverage City centre Airport
ID of All IVSs All IVSs All IVSs All IVSs All IVSs
D4.3 Intermediate test results
15/05/2014 197 Version 1.3
test set:
All MNOs All MNOs All MNOs All MNOs All MNOs
PSAP PSAP PSAP PSAP PSAP
Result Unit Result Result Result Unit Result Unit Result Unit
KPI_001a 957 - 225 - 72 - 242 - 37 -
KPI_001b 2438 - 480 - 160 - 515 - 70 -
KPI_002b 72 % 76 % 83 % 63 % 66 %
KPI_003 90 % 90 % 93 % 90 % 93 %
KPI_004 81 % 82 % 84 % 78 % 81 %
KPI_005 17 s 16 s 16 s 18 s 17 s
KPI_006 89 % 93 % 100 % 83 % 84 %
KPI_007a 5 s 5 s 5 s 5 s 5 s
KPI_008 12 s 11 s 11 s 12 s 12 s
KPI_009 150 m 142 m 159 m 23 m 44 m
KPI_013 78 % 72 % 67 % 58 % 59 %
KPI_019 15 s 15 s 15 s 16 s 15 s
KPI_020 82 % 98 % 51 % 83 % 62 %
Table 91: Results of the drive test per area-type (NL)
During the drive tests also several objects (tunnels, high voltage cables) were encountered, and on
fixed locations the test vehicle stopped for several minutes. The results from these locations are
given in Table 92.
In a tunnel, the accuracy GPS location and the heading direction is lower than in regular situations.
Also the number of correct send MSDs is lower, resulting in a lower success rate of completed eCalls.
High voltage cables have no significant influence on the eCalls. Only the accuracy of the heading
information is lower, which can be explained that one stop-location was close to a high voltage cable.
Most likely, this lower accuracy is caused by this particular stop.
D4.3 Intermediate test results
15/05/2014 198 Version 1.3
Stops have both a positive and a negative influence on the eCalls. The GPS accuracy is higher, while
the accuracy of the heading information is lower. Also less correct MSDs are sent to the PSAP.
Regular Tunnel
High voltage
cables Stop
ID of
test set: IVS 1 IVS 2 IVS 3 IVS 4
All MNOs All MNOs All MNOs All MNOs
PSAP PSAP PSAP PSAP
Result Unit Result Unit Result Unit Result Unit
KPI_001a 1347 - 17 - 99 - 120 -
KPI_001b 3257 - 43 - 220 - 251 -
KPI_002b 72 % 52 % 73 % 63 %
KPI_003 90 % 87 % 94 % 94 %
KPI_004 82 % 57 % 80 % 73 %
KPI_005 17 s 17 s 16 s 17 s
KPI_006 89 % 92 % 92 % 88 %
KPI_007a 5 s 6 s 5 s 5 s
KPI_008 12 s 12 s 11 s 11 s
KPI_009 135 m 188 m 137 m 43 m
KPI_013 76 % 62 % 61 % 36 %
KPI_019 15 s 16 s 14 s 15 s
KPI_020 83 % 100 % 91 % 70 %
Table 92: Results of the drive test per encountered object (NL)
Conclusions 7.9.6
The main conclusion from the first round of drive tests is that eCall seems technically feasible but the
results aren’t satisfactory yet. For an emergency system the results are below par:
D4.3 Intermediate test results
15/05/2014 199 Version 1.3
The variance in performance between the different IVSs is substantially although the used
IVSs are supposed to be according to standards. This implies that the standards might not be
specific enough
IVSs that behave differently will be a menace for PSAP operators when they have to assess
the incoming emergency-calls
The variance in performance between the different MNOs is less significant.
Recommendations 7.9.7
Testing is only useful when all manufacturers have implemented the standards correctly; our
research shows that this may not be the case.
Insight in every part of the eCall chain is necessary in order to be able to realise and interpret
test results.
Every test method has its own presumptions on the functioning of the systems that are being
tested. One should make all these assumptions explicit.
The analysis of the test results implies that some standards need to be adjusted (change of
additional data definitions, timestamp definitions, a newly asked MSD should contain new
information)
Good understanding of the problems occurred while testing needs more analytical tools then
presently available.
Test results have led to changes in modems and application. A 2nd round of drive-tests and
scenario tests is needed, with a larger number of calls.
Interoperability testing will show whether the disappointing performance is merely a Dutch
issue or that there will be implications for other pilot countries as well.
The implication of the results in the HeERO pilot sites should be aggregated and discussed on
European level.
A distinction should be made between problems specifically related to eCall and more
general problems (like i.e. bad coverage) not related to eCall. These are not the scope of the
eCall project.
The definition of additional data in the standard MSD needs to be changed to make it usable.
Those will also take away the risk that future implementations of the “spare” room in the
standard MSD could have negative impact on the deciphering of the standard MSD.
The technical implementation of the first additional data within the present standard now
used by HGV eCall asks for more is the active involvement of other HeERO members.
D4.3 Intermediate test results
15/05/2014 200 Version 1.3
Appendix A 7.9.8
IVS 1 IVS 2 IVS 3 IVS 4
Measurements 1467 1551 890 499
Outliers 137 149 129 26
D4.3 Intermediate test results
15/05/2014 201 Version 1.3
IVS 1 IVS 2 IVS 3 IVS 4
Measurements 1467 1551 891 500
Outliers 140 149 82 17
D4.3 Intermediate test results
15/05/2014 202 Version 1.3
IVS 1 IVS 2 IVS 3 IVS 4
Measurements 1587 1658 1407 999
Outliers 122 278 102 83
D4.3 Intermediate test results
15/05/2014 203 Version 1.3
D4.3 Intermediate test results
15/05/2014 204 Version 1.3
Table 93: Outliers KPI_009 (NL)
IVS 1 IVS 2 IVS 3 IVS 4
Measurements 1413 1500 885 470
Outliers 88 225 71 5
IVS 1 IVS 2 IVS 3 IVS 4
Measurements 1181 1260 383 723
Outliers 163 157 33 60
D4.3 Intermediate test results
15/05/2014 205 Version 1.3
8 Reference [1] W.G. Cochran, The distribution of the largest of a set of estimated variances as a fraction of their
total, Annals of Human Genetics (London), 11(1), 47–52 (January 1941).
[2] D.L. Massart, B.G.M. Vandeginste, L.M.C. Buydens, S. de Jong, P.J. Lewi, J. Smeyers-Verbeke,
Handbook of Chemometrics and Qualimetrics: Part A, Elsevier, Amsterdam, The Netherlands, 1997,
ISBN 0-444-89724-0.
[3] P. Konieczka, J. Namieśnik, Quality Assurance and Quality Control in the Analytical Chemical
Laboratory – A Practical Approach, CRC Press, Boca Raton, Florida, 2009, ISBN 978-1-4200-8270-8.
[4] J.K. Taylor, Quality Assurance of Chemical Measurements, 4th printing, Lewis Publishers, Chelsea,
Michigan, 1988, ISBN 0-87371-097-5.
[5] W. Horwitz, Harmonized protocol for the design and interpretation of collaborative studies,
Trends in Analytical Chemistry, 7(4), 118–120 (April 1988).
[6] ISO Standard 5725–2:1994, “Accuracy (trueness and precision) of measurement methods and
results – Part 2: Basic method for the determination of repeatability and reproducibility of a standard
measurement method”, International Organization for Standardization, Geneva, Switzerland, 1994,
available at: http://bit.ly/P2AwhJ.
[7] R. Moore, Mathematics Department, Macquarie University, Sydney, Australia, 1999, available at:
http://bit.ly/QGUS1X.
[8] R.U.E. 't Lam, Scrutiny of variance results for outliers: Cochran's test optimized, Analytica Chimica