Universit` a degli Studi di Genova Consiglio Nazionale delle Ricerche Good Experimental Methodologies and Optimal Design Of Experiments in Marine Robotics Eleonora Saggini Dottorato in Matematica e Applicazioni XXVIII ciclo Eleonora Saggini DoE in marine robotics 4th November 2015 1 / 31
33
Embed
Good Experimental Methodologies and Optimal Design … · Good Experimental Methodologies and Optimal Design Of Experiments in Marine Robotics ... marine robots. Need of de nition
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Universita degli Studi di Genova Consiglio Nazionale delle Ricerche
Good Experimental Methodologiesand Optimal Design Of Experiments
in Marine Robotics
Eleonora Saggini
Dottorato in Matematica e ApplicazioniXXVIII ciclo
Eleonora Saggini DoE in marine robotics 4th November 2015 1 / 31
Autonomous Marine Robots
Figure: Marius AUVFigure: Charlie USV
Figure: Autosub AUV
Long term objective: Integration of robots within civilian scenarios in everyday lifeIssues: Lack of regulation laws
Good Experimental Methodologies in order to
reproduce experiments
assess robot performances
compare robot behaviours
Eleonora Saggini DoE in marine robotics 4th November 2015 2 / 31
Operational objective
Develope a software tool and a protocol for performance assessment of autonomousmarine robots. Need of
definition of replicable experiments
automatic selection and execution of replicable experiments
online or postprocessing performance evaluation
Issues for experiments
achievement of repeatability including environmental conditions and replicate initialconditions (difficulties to drive a UMV in the pre-defined, starting position andspeed)
time requirements, high cost of data acquisition and data sharing
where and when to evaluate performance (line following task and maneouvringphases: turn, path-approach, transient or overshoot, steady-state)
Eleonora Saggini DoE in marine robotics 4th November 2015 3 / 31
Repeatability VS Reproducibility
Repeatability concerns the fact that a single result is not sufficient to ensure thesuccess of an experiment
Reproducibility (or replicability) is the possibility to verify, in an independent way,the results of a given experiment.
chemistry, physics,...: to reproduce the same results of a previous experimentmarine robotics: the term replicability is applied to all controllable parameters of theexperiments and not to its results.The obtained results can be easily compared as well as the performance of two or moreplatforms (dependence on non controllable conditions e.g. environmental)
Eleonora Saggini DoE in marine robotics 4th November 2015 4 / 31
Eleonora Saggini DoE in marine robotics 4th November 2015 9 / 31
The crossing criterion [with Laura Torrente]
1 computation of an algebraic curve f = 0 that approximates the points in R within atolerance ε1 > 0.Low Polynomial Degree (LPA) algorithm
its total degree has to be bounded by the smallest total degree amongst the totaldegrees of all polynomials vanishing at all points in Rthe zero-locus of f lies close to the points of R by less than the tolerance ε1 (thedistance is induced by the given norm)
2 identification of the points in V far from the reference path f = 0 for more than atolerance ε2 > 0.Crossing Cell algorithm (CA)For each p ∈ V
|f (p)| > B1(f , p, ε2) f = 0 does not cross Bε2 (p) → red|f (p)| < B2(f , p, ε2,R) → crosses Bε2 (p) → blackthe crossing problem remains undecided → green
If ε2 < 1: approximated bounds B ′1,B′2 (ACA)
Eleonora Saggini DoE in marine robotics 4th November 2015 10 / 31
The crossing criterion [with Laura Torrente]
1 computation of an algebraic curve f = 0 that approximates the points in R within atolerance ε1 > 0.Low Polynomial Degree (LPA) algorithm
its total degree has to be bounded by the smallest total degree amongst the totaldegrees of all polynomials vanishing at all points in Rthe zero-locus of f lies close to the points of R by less than the tolerance ε1 (thedistance is induced by the given norm)
2 identification of the points in V far from the reference path f = 0 for more than atolerance ε2 > 0.Crossing Cell algorithm (CA)For each p ∈ V
|f (p)| > B1(f , p, ε2) f = 0 does not cross Bε2 (p) → red|f (p)| < B2(f , p, ε2,R) → crosses Bε2 (p) → blackthe crossing problem remains undecided → green
If ε2 < 1: approximated bounds B ′1,B′2 (ACA)
Eleonora Saggini DoE in marine robotics 4th November 2015 10 / 31
Eleonora Saggini DoE in marine robotics 4th November 2015 17 / 31
Experiments
TURN
TURN
BACKWARD PATH
INITIALPOSITION
FORWARD PATH
TURN
TURN
BACKWARD PATH
FORWARD PATH
TURN
BACKWARD PATH
FORWARD PATH
run 1 run 2 run n
.........
.........
An experiment is definedby n runs, correspondingto n different paths thatare executed sequentiallyin time
Runs are independentideally
The path is followed backand forth (at least onerepetition)
Repeatability is achievedthrough a suitable turningmanoeuvre
Performance is notmeasured during the turnphases
Eleonora Saggini DoE in marine robotics 4th November 2015 18 / 31
DeepRuler: Design Execute and Evaluate Path-following Experiments forRobotic Unmanned vehicLes and Repeatability [Sorbara et al, 2015]
A software for both for running experiments on a simulator or during a field trial to
guide the user in the definition of an experiment
conduct and supervise the execution of the experiment
Achieve these two aims
with a step-by-step windowed configurator: receives in input parameters that specifyan experiment (working area, specific path, choice of telemetry and of metrics...)and produces a file reusable to repeat the experiment
by sequencing the experimental phases, collecting the telemetry, computing themetric
Eleonora Saggini DoE in marine robotics 4th November 2015 19 / 31
DeepRuler
Main features of DeepRuler are:
open source
modular: for path generation, telemetry collection, metrics computation
portable1 written in C++ using well-known library. At the moment compiled and tested under
Linux amd64/i386 and Windows 64/32 bit2 at the moment two communication to/from robot: ROS and boost::serialization
Eleonora Saggini DoE in marine robotics 4th November 2015 20 / 31
Measurement process for the simulator of Charlie USV and sinusoidal paths
Eleonora Saggini DoE in marine robotics 4th November 2015 22 / 31
1st batch: paired sample t-test, back and forth
DFH,1 and DB
H,1 → p = 0.8024; DFA,1 and DB
A,1 → p = 0.4466
5 10 15 20 25 30
0.6
0.5
0.4
0.3
0.2
0.1
0
0.1
0.2
0.3
batch 1: Hausodrff
5 10 15 20 25 30
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
0.02
0.04
batch 1: Area
2nd batch: paired sample t-test, back and forth
DFH,2 and DB
H,2 → p = 0.2378; DFA,2 and DB
A,2 → p = 0.2559
5 10 15 20 25 30
0
2
4
6
8
10
12
batch 2: Hausdorff
5 10 15 20 25 30
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8batch 2: Area
Eleonora Saggini DoE in marine robotics 4th November 2015 23 / 31
Simulation study: same batch, different order
Paired samples t-test for the 2 sets of data
DH,1 and DH,2 → p = 0.4652;
DA,1 and DA,2 → p = 0.7879
0 5 10 15 20 25 30 35−6
−5
−4
−3
−2
−1
0
1batch 1 e 2: Hausdorff
0 5 10 15 20 25 30 35−0.8
−0.7
−0.6
−0.5
−0.4
−0.3
−0.2
−0.1
0
0.1batch 1 e 2: Area
Eleonora Saggini DoE in marine robotics 4th November 2015 24 / 31
Design of Experiments: adaptive procedure
Worst performance
1 Space filling designLHSone generator lattice...
2 Performance index computation
3 Optimal local designfor maximizing the performance
4 Kriging re-estimation
Best prediction
1 Space filling designLHSone generator lattice...
2 Performance index computation
3 Kriging estimation
4 Optimal local designfor minimizing the error
MSE of the predictorw.r.t. values computed from a fullfactorial design(simulation/benchmark)
5 Kriging re-estimation
Eleonora Saggini DoE in marine robotics 4th November 2015 25 / 31
Definition of path following “complexity”?
Sinusoidal path-followingy = θ1 sin(πθ2
Lx)
↓
Maximum curvature inxMC = −θ2/4− L/2
10 20 30 40 501
2
3
4
5
6
7
8
9
10
11
θ1
(amplitude)
θ 2(h
alf−
perio
ds)
Absolutecvaluecofcthecmaximumccurvature
c 0
1
2
3
4
5
6
0 10 20 30 40 500
1
2
3
4
5
6
7
8
9
10
θ1
(amplitude)
θ 2(h
alf−
perio
ds)
Area distance
0
1
2
3
4
5
6
Eleonora Saggini DoE in marine robotics 4th November 2015 26 / 31
Latin hypercube sampling + Central composite
Worst performance
0 10 20 30 40 500
1
2
3
4
5
6
7
8
9
10
θ1
(amplitude)
θ 2(h
alf−
perio
ds)
LHSC+CCCC(maxCarea)
Best prediction
0 10 20 30 40 500
1
2
3
4
5
6
7
8
9
10
θ1
(amplitude)
θ 2(h
alf−
perio
ds)
LHSC+CCCC(maxCmse)
Eleonora Saggini DoE in marine robotics 4th November 2015 27 / 31
Kriging prediction
Worst performance Best prediction
Eleonora Saggini DoE in marine robotics 4th November 2015 28 / 31
Papers
E Saggini, E Zereik, M Bibuli, A Ranieri, G Bruzzone, M Caccia, E Riccomagno,Evaluation and Comparison of Navigation Guidance and Control Systems for2D/Surface Path-Following, Annual Reviews in Control, available online athttp://dx.doi.org/10.1016/j.arcontrol.2015.08.006.
A Sorbara, A Ranieri, E Saggini, E Zereik, M Bibuli, G Bruzzone, M Caccia, Testingthe Waters: Design of Replicable Experiments for Performance Assessment of MarineRobotic Platforms, IEEE Robotics and Automation Magazine 22(3), 2015, 62–71.
E Saggini, L Torrente, M Bibuli, G Bruzzone, M Caccia, E Zereik Assessingpath-following performance for Unmanned Marine Vehicles with algorithms fromNumerical Commutative Algebra, Proceedings of the 22nd Mediterranean Conferenceon Control and Automation (MED’14), June 16–19, Palermo, Italy, 2014.
E Saggini, E Zereik, M Bibuli, G Bruzzone, M Caccia, Performance Indices forEvaluation and Comparison of Unmanned Marine Vehicles’ Guidance Systems,Proceedings of the 19th IFAC World Congress, (IFAC WC 2014), August 24–29,Cape Town, Africa, 2014.
Eleonora Saggini DoE in marine robotics 4th November 2015 29 / 31
Papers
M Bibuli, G Bruzzone, M Caccia, E Fumagalli, E Saggini, E Zereik, UnmannedSurface Vehicles for Automatic Bathymetry Mapping and Shores’ Maintenance,Proceedings of the MTS/IEEE OCEANS’14 Conference, Taipei, Taiwan, 2014.
M Caccia, E Saggini, M Bibuli, G Bruzzone, E Zereik, E Riccomagno, Towards GoodExperimental Methodologies for Unmanned Marine Vehicles, EUROCAST, Part II,LNCS 8112, Springer, Heidelberg, 2013, 365–372.
In preparation
E Saggini, M L Torrente, A new crossing criterion to assess path-followingperformance for Unmanned Marine Vehicles, Journal of Algebraic Statistics, (to besubmitted in November)
E Saggini, E Riccomagno, Adaptive experimental design for performance assessmentof marine robotic platforms, Computational Statistics & Data Analysis, Special Issueon Design Of Experiments, 2016
Eleonora Saggini DoE in marine robotics 4th November 2015 30 / 31
Eleonora Saggini DoE in marine robotics 4th November 2015 31 / 31