On Experimental Research in Sampling-based Motion Planning Roland Geraerts Workshop on Benchmarks in Robotics Research IROS 2006.

On Experimental Research in Sampling-based Motion Planning

Roland GeraertsWorkshop on Benchmarks in Robotics Research

IROS 2006

Probabilistic Roadmap Method

Construction (G =V,E )Loop

c a free sample

add c to the vertices V

Nc a set of nodes

for all c’ in Nc in increasing distance

if c’ and c are not connected in G then

if local path between c and c’ exists then

add the edge c’c to E

Forbidden spaceFree space

Sample

c

Colliding path

c

c’

c

Local path

c’c’c

c

Probabilistic Roadmap Method

Construction (G =V,E )Loop

c a free sample

add c to the vertices V

Nc a set of nodes

for all c’ in Nc in increasing distance

if c’ and c are not connected in G then

if local path between c and c’ exists then

add the edge c’c to E

Query connect sample s and g to roadmap

Dijkstra’s shortest path

Forbidden spaceFree space

Sample

Start / goalLocal path

Shortest path

Methods

• General setup– SAMPLE

• Implemented in C++ using VS.NET 2003• Easy API to add techniques• GUI: easily set up experiments• Repeatability: load/save an experiment• Easily comparing different techniques• Easily examining parameter of a technique• Automatically collect/process data of experiment

– Demo

Methods

• Test problems– Conclusions were often too general due to

limited set of problems– Also choose worst-case problems

Methods

• Interchangeability– Libraries taking take of common functionality

• Collision checking, visualizationCallisto: http://www.cs.uu.nl/dennis/callisto/callisto.html [Nieuwenhuisen]

• Graph utilitiesAtlas: http://www.cs.uu.nl/dennis/atlas/atlas.html [Nieuwenhuisen]

• Nearest neighborMPNN: http://msl.cs.uiuc.edu/~yershova/mpnn/mpnn.htm [Yershova, Lavalle]

• Deterministic sampling methodshttp://msl.cs.uiuc.edu/~yershova/so3sampling/so3sampling.htm [Yershova]

• Rotation in 3Dhttp://www.kuffner.org/james/software [Kuffner]

Methods

• Interchangeability– Source code of motion planning framework

• Motion planning kitMPK: http://ai.stanford.edu/~mitul/mpk [Latombe]

• Move3Dhttp://www.laas.fr/~nic/Move3D [Siméon]

• Motion strategy libraryMSL: http://msl.cs.uiuc.edu/msl [Lavalle]

– Unfortunately, code is often not up-to-date

Methods

• Interchangeability– Sources

• Geometry of environment/robot: VRML

• Problem descriptions: XML

– Advantages of using existing languages• Well documented• Parsers/type checkers are available for all platforms• Existing programs for creating/editing the files

Methods

• Interchangeability– Sources of geometry files and benchmarks

• http://www.give-lab.cs.uu.nl/movie/moviemodels [MOVIE]

• http://faculty.cs.tamu.edu/amato/dsmft/benchmarks [Amato]

• http://mpb.ce.unipr.it/ [Reggiani]

– Problems should be put online when article is published

Results

• Evaluation of solution– Compare new technique with existing ones

• Pitfall: parameter tuning only for the new technique

– Compare against optimal solution• Often only known for trivial cases• Approximate optimal solution by many runs

– User studies

Results

• Statistics– Large variances in running times

• Complicates statistical analysis• Makes analysis unreliable• Is undesirable from a user’s point of view

– Perform large number of runs– Provide more statistical info, e.g. box plots– Deterministic versus randomized techniques

• Deterministic techniques can respond sensitively to small changes in the problem setting

Conclusion

• Automate conducting experiments as much as possible

• Choose test problems carefully

• Source code, software components and problem data should be made available

• Use standard file formats (VRML, XML)

• Provide an extensive statistical analysis