Comparative Evaluation of Occupancy Grid Mapping Methods ... · To simplify the mapping problem, it is assumed that the occupancy of a given cell is not relevant to the estimation

Comparative Evaluation of Occupancy GridMapping Methods Using Sonar Sensors

Joao CarvalhoInstituto Superior Tecnico

Technical University of [email protected]

Abstract—Building occupancy grid maps with sonar sensors isa challenging task due to angular uncertainty, specular reflectionsand crosstalk. This paper presents a quantitative comparison oftwo probabilistic and one heuristic approaches to the roboticmapping using real sonar data – inverse and forward sensormodels and the CEMAL method. Moreover, the two probabilisticmethods are also tested pre-filtering the sonar data with theCEsp filter, which is part of the CEMAL approach. The resultsshow that, using the pre-filtering, all algorithms present a similarand significantly better performance, while without filtering theinverse method presents the highest error.

I. INTRODUCTION

One commonly used map representation in robotics is theoccupancy grid map (OccGrid map), which aims to geometri-cally represent the environment through a grid discretization ofthe space. To gather information about the robot environment,one frequently used sensor is the sonar. Sonars are cheap andallow the construction of maps even with a low number ofsensors. Despite these advantages, sonars suffer from angularuncertainty, specular reflections and crosstalk between eachother, causing erroneous and conflicting measurements [3].Angular uncertainty rises from the fact that the sonar waveis spread through the air, leading to poor angular resolution,Figure 1. Smooth surfaces at oblique incidence to the sonardo not produce detectable echoes, functioning as a mirror tothe wave. This is the specular reflection phenomenon, Figure2. A phenomenon that causes sonar measurements with valuesinferior to the real range value is the crosstalk. The crosstalkis an event that occurs when a given sonar sensor receives thepulse emitted by other sonar before it receives the echo of itsown pulse, Figure 3.

x1 x2 x3

Fig. 1. Example of conflicting occupancy information due to angularuncertainty.

Fig. 2. Specular reflection example.

Fig. 3. Crosstalk example.

OccGrid mapping was first introduced by Elfes and Moravecin 1985, making use of inverse sensor models (ISM) [5]. Inthis approach, cells are assumed conditionally independentgiven the robot poses and measurements. In 2001, Thrunproposed an alternative approach using forward sensor models(FSM) [6]. This method approaches the mapping problemin the high-dimensional space of all binary maps, trying tosolve erroneous and conflicting sonar measurements whichaffect the ISM results. Recently, Lee and Chung proposedthe Conflict-Evaluated Maximum Approximated Likelihood(CEMAL), which is a method that includes measurementfiltering with the CEsp filter[4].

This paper aims at quantitatively comparing these threeOccGrid mapping methods using real sonar data. Additionally,the ISM and FSM methods were also tested pre-filtering thesonar data with the CEsp filter. The FSM was subject to slightchanges to the original formulation.

II. OCCGRID MAPS WITH INVERSE SENSOR MODELS

In this approach, the mapping problem is treated inverselyto how sonar data is generated, being formulated as

p(M |z1:T , x1:T ) (1)

where M represents the complete map, z1:T represents thecomplete set of measurements and x1:T are the correspondingposes. This is the denominated inverse sensor model.

To simplify the mapping problem, it is assumed that theoccupancy of a given cell is not relevant to the estimation ofthe occupancy of its neighbour cells, i.e., cells are condition-ally independent given measurements and the robot trajectory,transforming the mapping problem into a binary estimationproblem,

p(mi|z1:T , x1:T ), (2)

where mi is an individual cell of the complete map. A secondassumption made is the static world assumption, consideringa measurement t conditionally independent from the previousmeasurements given the map knowledge. This is a commonassumption in mapping but given the decomposition into a bi-nary problem this becomes a much stronger and also incorrectassumption, since it considers conditional independence givenonly a map cell and not the complete map. Additionally, thepose in the instant t is independent from the poses in previousinstant times. So, for time t, one can write:

p(zt|z1:t−1, x1:t,mi) = p(zt|mi, xt). (3)

Given these assumptions and applying the Bayes rule to (2),we have:

p(mi|z1:t, x1:t) =p(mi|zt, xt)p(zt|xt)p(mi|z1:t−1, x1:t−1)

p(mi)p(zt|z1:t−1, x1:t). (4)

As usual, one will compute the log odds of this probabilityinstead of the probability itself:

lti = logp(mi|zt, xt)

1− p(mi|zt, xt)− log

p(mi)

1− p(mi)+ lt−1

i , (5)

where lti represents log p(mi|z1:t,x1:t)1−p(mi|z1:t,x1:t)

. The term lt−1i equals

log p(mi)1−p(mi)

when t = 1. The probability p(mi) is the prior ofoccupancy of the cell i of the map. A typical and simple ap-proach is to model the posterior not as a fixed functional formbut by a finite number of values which roughly approximatethe posterior [3]. For the cells at distances between 0 and theneighbourhood of the measurement the occupancy probabilityhas a low value, in the neighbourhood it has a high value and0.5 beyond.

Making use of (5) the log-odds occupancy representationcan be easily computed for each cell that falls into thecoverage cone of the sonar measurements. So, finally, thedesired occupancy probability of the cells can be recoveredthrough:

p(mi|z1:t, x1:t) = 1− 1

1 + elti

, (6)

resulting in a map of occupancy beliefs for each individualcell.

III. OCCGRID MAPS WITH FORWARD SENSOR MODELS

This approach deals with the mapping problem in itscomplete state space and assumes the world is static. It usesforward sensor models, being able to make use more complexsensor models. The forward mapping approach is modelled asa likelihood

p(z1:T |M,x1:T ). (7)

This is a generative model, being formulated as the phe-nomenon happens; given the world (represented by the mapM ) and a given set of poses, a particular set of sonar readingis generated. The goal is to maximize (7), iteratively adjustingM till no better model is found. This problem can then beformulated as a maximum likelihood estimation problem.

Rather than assuming that all measurements are causedby an obstacle, three possible cases of beam reflection areconsidered, maximum reading, random and non-random. Anon-random measurement is caused by an obstacle in the sonarbeam. A maximum value reading happens with the failure indetecting all the obstacles, when present, and returning themaximum range value, zmax. The random case models theremaining causes. Since in practice the true cause of the sonarreading is not known, a classifier has to be used to identify it.

For the measurement with index t, consider Kt to be thenumber of obstacles present in the sonar cone, dt,k the distancefrom the k’th obstacle in the cone and Dt the set of obstacledistances in ascending order. The model (7) is defined as thecombination of the models of each possible cause. Considerthe binary variables ct,∗, ct,k, ct,0, which are equal to 1 whenthe measurement is random, caused by obstacle k or equal tothe maximum range, respectively. For each t, only one can beequal to 1.

The random case is modeled as a uniform distribution in theentire sonar range, since the reading could have been causedin any part of the sonar cone. When the beam is reflectedby an obstacle, it is considered that it is affected by additivewhite gaussian noise. In the case where zt = zmax, since itis a discrete event, a Dirac delta function is considered. In asingle expression, the measurement distribution can be writtenas

p (zt|M,xt, ct) = p (zt|M,xt, ct,∗ = 1)ct,∗Kt∏k=0

p(zt|M,xt, ct,k = 1

)ct,k.

(8)

Since there is no prior knowledge of the measurement’scause we define the posterior probability

p (ct|M,xt) = p (ct,∗ = 1|M,xt)ct,∗

Kt∏k=0

p(ct,k = 1|M,xt

)ct,k, (9)

p(ct|M,xt) = (10){prand if ct,∗ = 1,

pmax if ct,0 = 1, Kt ≥ 1,

(1− prand − pmax)∏k−1

i=1

[(1− p

(i)

hit

)]p(k)

hitif ct,k = 1, k ≥ 1,

where prand is the prior probability of a measurement beingrandom, pmax is the prior probability of a measurement beingmaximum and p(i)hit is the prior of the obstacle i to reflect thesonar beam. The phit probability is function of the obstacle’swidth coverage in the sonar cone, varying linearly betweena minimum and a maximum value and being equal to themaximum value when the obstacle covers 100% of the conewidth. Therefore, an obstacle might be formed by one or moreoccupied cells, forming a cluster. Cells are clustered havingas criterium its distance to the sonar cone origin. A cluster isinitially formed by a single cell in which further cells are addedif the difference between its distance, dt,k, and the clustercenter of mass is smaller then a given threshold. When a celldoes not meet this criterium, a new cluster is created with it.

One can then define the log-likelihood of the complete set ofmeasurements with the correspondence variables as the latentvariables, compute the expected likelihood over those variablesand use the Expectation-Maximization (EM) algorithm tomaximize the resulting likelihood [1]. The logarithm is usedto avoid truncation problems. The expected log-likelihood tomaximize is given by:

E [log p (z1:T , c1:T |M,x1:T ) |z1:T , x1:T ,M ] (11)

= E

{T∑t

log p (zt|M,xt, ct) p (ct|M,xt) |z1:T , x1:T ,M

}.

In the computation of this likelihood (8) and (9) are used.Being ct a Bernoulli random variable, its expected value equalsits probability, p(ct|M, zt, xt).

On the original formulation, the event of a maximummeasurement is a particular case of the non-random case.As proposed in this paper, considering the maximum readingevent as a different event of the non-random case and definingp(zt|M,xt, ct,0 = 1) as a Delta dirac function makes thosereadings have no influence in the likelihood and in the processof maximization, contrary to what happens in the originalformulation. Making phit function of the coverage and theintroduction of clustering allows the representation of theangular uncertainty, which is a process not clear in [6].

On the maximization process with the EM, no terms arediscarded in (11), since any change in M might producesignificant value variations in those terms. To find the mapM that maximizes the likelihood, the occupancy of the cellsthat fall into the measurements cone is flipped and maintainedif its new value increases the likelihood value. The maximiza-tion process stops when no flipping increases the likelihood.Given the discretization made, this results in a very greedyalgorithm, in which the final result highly depends on the cellflipping order. Since it gave empirically good results, in thisimplementation we chose to first flip the cells closest to the

measurement and progressively moving away. The Dirac deltafunction in p (zt|M,xt, ct,k = 1) is implemented as a gaussiandistribution with a very low variance.

IV. CEMAL

Sonar sensor readings are characterized by two regions:occupied and empty. The occupied region is the region ofoccupied cells within a certain neighbourhood of the mea-surement. The empty region is the region of unoccupiedcells between the sonar cone origin and the measurementneighbourhood. For time t, the occupied and empty regionsare represented by O(t) and E(t), respectively.

Inconsistencies occur when multiple measurements overlap.An inconsistent region, I(t), is defined as a region withconflicting occupancy information, whereas a consistent re-gion, C(t), is defined as a region in where there is only onetype of occupancy information. Uncertain regions, U(t), areinconsistent regions where only a partial part of the occupancyregion of a measurement is overlapped with empty regions ofother measurements, Figure 4(a). A conflict region, F (t), isan inconsistent region where the complete occupied regionof a measurement is overlapped with empty regions of othermeasurements, Figure 4(b). [4]

The candidates to incorrect measurements are the mea-surements causing conflict regions, such as the measurementsFigure 4(b) presents. To recognize the incorrect measurementsbetween the candidates the Conflict Evaluation with soundpressure (CEsp) filter is used.

A. CEsp

The CEsp method is a filtering method based on thecomparison of the sound pressure of the waves received bythe conflicting measurements. The sound pressure of a wavereceived in a sonar can be represented as

SPR(r, θ) = c2SPT (2r, 0)10DT (θ)

20 10DR(θ)

20 (12)

=c1r10

DT (θ)+DR(θ)

20 ,

where r is the distance from the cone origin to the obstacle,θ is the angle of the obstacle relative to the measurementcone heading, DT and DR are the transmitting and receivingdirectivity, respectively, and c1 is an unknown constant whichis canceled when sound pressure levels are compared. Forfurther information on the deduction of (12), consult [4].

Using (12), one can identify the incorrect measurementswithin the candidates. Figure 4(c) illustrates the conflict cellson a OccGrid map. Consider the measurement that indicatesoccupancy in the conflicting area as being the positive mea-surement, P , and the ones that indicate empty space in thatarea as being the negative measurements, N . Consider the

t

t

(a)

t

t(b)

t

t

(c)

t

(d)

Fig. 4. Inconsistent and conflict cells: (a) uncertain cells, U(t); (b) conflict cells, F (t); (c) discretized conflict cells F (t); (d) if the marked cell is occupied,j1 and j2 are incorrect. Adapted from [4].

hypothesis that a cell in the conflict region is occupied. Com-paring the sound pressures of the conflicting measurements,one can conclude about the occupancy of that cell:• SPP ≥ SPN : If there is an obstacle in the conflict

cell, the negative reading might miss it, since its soundpressure is lower the sound pressure from the positivereading. Thereby, it is considered that the obstacle exists;

• SPP < SPN : If there is an obstacle in the conflict cell,the negative measurement cannot miss it, as its soundpressure is higher than the pressure of the positive one.Hence, it is considered that there is no obstacle present.

For instance, for the example of Figure 4(c), consider thatit is revealed, through the just described comparison method,that there is an obstacle as shown in Figure 4(d). Then, sincethe obstacle is in their empty region, it is considered thatmeasurements j1 and j2 are incorrect and must be discarded.

This filtering method is used to ensure that incorrectmeasurements are discarded and no conflict regions exist,remaining only consistent and uncertain regions.

B. Maximum Approximated Likelihood

In the MAL method, it is assumed that the incorrectmeasurements were discarded by the CEsp filtering. Therefore,and given that sonar sensors were designed to provide thedistance to the closest obstacle in their perceiving cone, it isconsidered that a reliable OccGrid map can be built through

argmaxM

p(z1:T |M,x1:T ) = argminM

∑t

(zt − d(Nt))2, (13)

where M is the map, z1:T and x1:T are the set of measure-ments and corresponding poses, zt is the measurement in timeinstante t and d(Nt) is the distance from the sonar cone originto its nearest obstacle Nt. It is made the assumption that ameasurement zt is conditionally independent from the previousmeasurements given the map M and the robot path.

It is considered that the error is minimized when|zt − d(Nt)| ≤ β, where β is the range uncertainty. Thereby,it is considered that the global solution to the likelihoodcan be found when Nt is placed in the occupied region ofthe measurement. Therefore, the mapping problem is reducedto a simple problem: cells in uncertain regions are set to

unoccupied and the remaining cells in occupied regions areset to occupied.

V. RESULTS

The robot used was the Pioneer P3-AT, equipped with anarray of eight SensComp 600 Series sonar sensors and aSick LMS200 laser rangefinder. Both the ground truth mapand the robot pose estimates were obtained using the laserrangefinder and the GMapping method [2]. The measurementswere taken with the robot moving approximately at 0.6m/sand measurements being taken with a 4Hz frequency on asingle lap to the environment. Two variations of the ISMapproach were implemented. The first is as described insection II but removing the maximum range measurements, inwhich specular reflections often result, and in the second it wasgiven less weight to larger measurements. So in (5), the termlog p(mi|zt,xt)

1−p(mi|zt,xt) comes multiplied by a variable, restrictedbetween 0 and 1, that is inversely proportional to the sonarmeasurement zt. The ISM and FSM methods were also testedwith the CEsp pre-filtering.

The ground truth map is presented on Figure 5(a), over-lapped with the path made by the robot while the it wasacquiring sonar data. Figure 5(b) presents the cones of thesonar measurements. The cones marked in red are the onesthe CEsp filtering discarded. The maps resulting from theimplemented algorithms are presented in remaining items ofFigure 5.

To compare the results, one computed for each algorithm theoverall errors (OE) and the true positive rates (TPR) and falsepositive rates (FPR). The OE is the percentage of incorrectlyclassified cells. The TPR is the ratio of the number of occupiedcells correctly classified cells over the occupied cells in theground truth, while FPR is the rate of the number of cellsincorrectly classified as unoccupied over the unoccupied cellsin the ground truth map. Since the comparison metrics areonly valid for binary maps, the ground truth and ISM mapsare binarized. The overall error for each dataset is presentedin Figure 6(a). Figure 6(b) shows a plot of the ROC space inwhich one is able to present the TPR and FPR.

From OE, one can conclude that, without filtering, the FSMpresents better map than the ISM. All the methods includingfiltering present a similar error, which is about half of the

(a) (b)

(c) (d)

(e) (f)

(g) (h)

Fig. 5. Results: (a) ground truth map and robot path; (b) sonar measurements– marked in red are the ones discarded by CEsp; (c) ISM map; (d) ISM withmeasurement decay map; (e) ISM with CEsp map; (f) FSM map; (g) FSMwith CEsp map; (h) CEMAL map.

errors without filtering. Through the ROC, one can see that themethods with filtering represent more of the obstacles presentin the ground truth, with the ISM plus CEsp presenting the bestTPR. Moreover, they do not present as many ghost obstaclesas the others.1

1Additional datasets and corresponding results can be accessed inwww.isr.ist.utl.pt/ jcarvalho.

0 1 2 3 4 5 6 70

0.01

0.02

0.03

0.04

0.05

0.06

0.07Overall Error

Algorithm

Overa

ll E

rror

ISM

ISM + Decay

ISM + CEsp

FSM

FSM + CEsp

CEMAL

(a)

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55ROC

TP

R

FPR

ISM

ISM + Decay

ISM + CEsp

FSM

FSM + CEsp

CEMAL

(b)

Fig. 6. Comparison metrics results: (a) overall error for each dataset; (b)ROC graphic.

VI. CONCLUSION

This paper presented a comparison between OccGrid map-ping using ISM, FSM and the CEMAL methods, with theCEsp filtering also being tested with the first approaches. Theresults showed that the CEsp filtering as a significant impact onthe final map produced by the methods. Future work consistsin presenting statistically significant results using multipledatasets and in studying better approaches using forwardsensor models, to achieve better results without filtering.

REFERENCES

[1] A.P. Dempster, N.M. Laird, and D.B. Rubin, Maximum likelihood fromincomplete data via the EM algorithm. Journal of the Royal StatisticalSociety. Series B (Methodological), pages 138, 1977.

[2] G.Grisetti, C.Stachniss, andW.Burgard, Improvedtechniquesfor grid map-ping with rao-blackwellized particle filters. Robotics, IEEE Transactionson, 23(1):3446, 2007.

[3] S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics. MIT Press,ISBN 0262201623, 2005.

[4] K. Lee, J.S. Lee, C. Kim, and W.K. Chung, Effective maximum likelihoodgrid map with conflict evaluation filter using sonar sensors. Roboticsand Automation, 2009. ICRA09. IEEE International Conference on, pages16231630. IEEE, 2009.

[5] H. Moravec and A. Elfes. High resolution maps from wide anglesonar. In Robotics and Automation. Proceedings. 1985 IEEE InternationalConference on, volume 2, pages 116–121. IEEE, 1985.

[6] S. Thrun. Learning occupancy grids with forward models. In IntelligentRobots and Systems, 2001. Proceedings. 2001 IEEE/RSJ InternationalConference on, volume 3, pages 1676–1681. IEEE, 2001.