Perception-Based Image Segmentation Using the Bounded Irregular Pyramid

Perception-Based Image Segmentation Using theBounded Irregular Pyramid

Rebeca Marfil, Antonio Bandera, and Francisco Sandoval

Grupo ISIS, Dpto. Tecnologıa ElectronicaETSI Telecomunicacion, Universidad de Malaga

Campus de Teatinos 29071-Malaga (Spain)

Abstract. This paper presents a bottom-up approach for fast segmen-tation of natural images. This approach has two main stages: firstly, itdetects the homogeneous regions of the input image using a colour-baseddistance and then, it merges these regions using a more complex distance.Basically, this distance complements a contrast measure defined betweenregions with internal region descriptors and with attributes of the sharedboundary. These two stages are performed over the same hierarchicalframework: the Bounded Irregular Pyramid (BIP). The performance ofthe proposed algorithm has been quantitatively evaluated with respectto ground-truth segmentation data.

1 Introduction

Image segmentation is typically defined as the low-level process of grouping pixelsinto clusters which present homogeneous photometric properties. However, if thegoal of the segmentation process is to divide the input image in a manner similarto human beings, then this definition is not valid. Natural images are generallycomposed of physically disjoint objects whose associated groups of image pixelsmay not be visually uniform. This makes extremely difficult to formulate whatshould be recovered as a region from an image or to separate complex objectsfrom a natural scene [4].

In order to reduce the complexity of segmenting real objects from their back-ground, the particular application could be taken into account. In these cases,the higher-level information is known a priori and it can be used to group theimage pixels into logical regions that resemble the real objects. To maintain thegenerality of use, several authors have proposed generic segmentation methodswhich are based neither on a priori knowledge of the image content nor on any ob-ject model [1,3]. These approaches typically combine a pre-segmentation stagewith a subsequent perceptual grouping stage. Basically, the pre-segmentationstage conducts the low-level definition of segmentation as a process of groupingpixels into homogeneous clusters and the perceptual grouping stage performs adomain-independent grouping which is mainly based on properties like the prox-imity, similarity, closure or continuity. Although the final obtained regions do notalways correspond to the natural image objects, they provides a mid-level seg-mentation which is more coherent with the human-based image decomposition.

F.A. Hamprecht, C. Schnorr, and B. Jahne (Eds.): DAGM 2007, LNCS 4713, pp. 244–253, 2007.c© Springer-Verlag Berlin Heidelberg 2007

Perception-Based Image Segmentation Using the Bounded Irregular Pyramid 245

Thus, it must be considered as a precursor to the detection of the real salientobjects in the image.

This paper presents a segmentation approach which is also divided into thesetwo successive stages. The pre-segmentation stage groups the image pixels into aset of regions whose spatial distribution is physically representative of the imagecontent. The size of this set of regions is commonly very much less than the originalnumber of pixels. Thus, these regions constitute an efficient image representationthat replaces the pixel-based image representation. Besides, these regions preservethe image geometric structure as each significant feature contain at least one re-gion. Our approach accomplishes this pre-segmentation stage by means of an ir-regular pyramid: the Bounded Irregular Pyramid (BIP). The BIP combines the2x2/4 regular structure with an irregular simple graph. The regular decimationis applied in the homogeneous parts of the image, while the heterogeneous partsare decimated using a classical irregular process [6,7]. The perceptual groupingstage groups the set of homogeneous regions into a smaller set of regions takinginto account not only the internal visual coherence of the obtained regions butalso the external relationships among them. For managing this process, a hierar-chical segmentation operator (HSO) can be applied [1]. The hierarchical structureis well-adapted to the parallel manipulation of regions, permitting to simultane-ously group several regions. Thus, the perceptual organization of the image canbe represented by a tree of regions, ordered by inclusion [1]. The roots of the treeis a set of regions which represent the entire scene and the leaves are the finest de-tails. To achieve this perceptual grouping, the proposed approach generates a setof new pyramid levels over the previously built BIP. However, while the nodes ofthe pre-segmentation pyramid are merged taking into account a colour criterion,the similarity among nodes of these new levels is defined using a more complex dis-tance which takes into account information about their common boundaries andinternal features like their colour or size.

The proposed method is related to the previous works of Arbelaez and Cohen[1,2] and Huart and Bertolino [3]. In all those papers, the perceptual group-ing is achieved by means of a hierarchical process. Besides, they employ a pre-segmentation stage prior to the perceptual grouping stage: Arbelaez and Cohenpropose to employ the extrema mosaic technique [2], and Huart and Bertolino usethe localized pyramid [3]. Our approach is very related to this last work. Thus,it uses different decimation techniques in the homogeneous and heterogeneousregions of the input image. The main difference with [3] is that both decima-tions are performed simultaneously using the BIP, being unnecessary to previ-ously detect the homogeneous areas of the input image. The pre-segmentationstage uses a colour distance to group image pixels into homogeneous regionsin a fast manner. The roots of these regions constitute the base level of theperceptual grouping stage. Thus, the perceptual grouping is integrated into thesame hierarchical structure. However, the distance employed to achieve this laststage is significantly more complex, complementing the colour contrast betweenregions with attributes extracted from the local boundary which they sharedand internal region features.

246 R. Marfil, A. Bandera, and F. Sandoval

The rest of the paper is organized as follows: Section 2 describes the pro-posed approach. It briefly explains the main aspects of the pre-segmentationstage, which is based on previous works of Marfil et al. [6,8]. This section alsodescribes the perceptual grouping process and the distance employed to groupthese homogeneous regions. These two stages are achieved using the BoundedIrregular Pyramid (BIP). The experimental results revealing the efficiency of theproposed method are presented in Section 3. The paper concludes along withdiscussions and future work in Section 4.

2 Perception-Based Segmentation Approach

2.1 Pre-segmentation Stage

Pyramids are hierarchical structures which have been widely used in segmen-tation tasks [7]. Instead of performing image segmentation based on a singlerepresentation of the input image, a pyramid segmentation algorithm describesthe contents of the image using multiple representations with decreasing res-olution. Pyramid segmentation algorithms exhibit interesting properties whencompared to segmentation algorithms based on a single representation. Thus,local operations can adapt the pyramidal hierarchy to the topology of the im-age, allowing the detection of global features of interest and representing themat low resolution levels [3].

The Bounded Irregular Pyramid (BIP) is a mixture of regular and irregularpyramids whose goal is to combine their advantages. A 2x2/4 regular structureis used in the homogeneous regions of the input image and a simple graph struc-ture in the non-homogeneous ones. The mixture of both structures generates anirregular configuration which is described as a graph hierarchy in which eachlevel Gl = (Nl, El) consists of a set of nodes, Nl, linked by a set of intra-leveledges El. Each graph Gl+1 is built from Gl by computing the nodes of Nl+1from the nodes of Nl and establishing the inter-level edges El,l+1. Therefore,each node ni of Gl+1 has associated a set of nodes of Gl, which is called thereduction window of ni. This includes all nodes linked to ni by an inter-leveledge. The node ni is called parent of the nodes in its reduction window, whichare called sons. Two nodes ni and nj of Nl are said to be adjacent or neighboursat level l, if their corresponding reduction windows wi and wj are neighbours atlevel l − 1. Two reduction windows wi ∈ Nl−1 and wj ∈ Nl−1 are neighbours ifthere are at least two nodes nr ∈ wi and ns ∈ wj which are connected by anintra-level edge er,s ∈ El−1. The set of nodes in Nl which are neighbours of anode ni ∈ Nl is called the neighbourhood of ni. An intra-level path is a sequenceof ordered nodes linked by intra-level edges. Two nodes ni ∈ Nl and nj ∈ Nl

are said to be connected if there exists an intra-level path that includes themboth. Equivalently, an inter-level path is a sequence of ordered nodes linked byinter-level edges. Two nodes ni ∈ Np and nj ∈ Nq are said to be connected ifthere exits an inter-level path that includes them both. The receptive field ri ofa node ni ∈ Nl is the set of nodes at level 0 which are connected to it by aninter-level path.

Perception-Based Image Segmentation Using the Bounded Irregular Pyramid 247

The successive levels of the hierarchy are built using a regular decimationprocess and a union-find strategy. Therefore, there are two types of nodes: nodesbelonging to the 2x2/4 structure, named regular nodes, and virtual nodes ornodes belonging to the irregular structure. In any case, two nodes ni ∈ Nl

and nj ∈ Nl which are neighbours at level l are linked by an intra-level edgeeij ∈ El. This mixture of processes inside the same hierarchical structure presentsinteresting properties. Thus, the irregular part of the BIP allows to solve themain problems of regular structures: their inability to preserve connectivity orto represent elongated objects [8]. On the other hand, the BIP is computationallyefficient because its height is constrained by its regular part [7].

The proposed approach uses a BIP structure to accomplish the segmentationtask. In this hierarchy, the first levels perform the pre-segmentation stage usinga colour-based distance to group pixels into homogeneous regions. In order tointroduce colour information within the BIP, all the nodes of the structure haveassociated 3 parameters: chromatic phasor S � H(n), luminosity V (n) and areaA(n), where S, H and V are the saturation, hue and value of the HSV colourspace. The chromatic phasor and the luminosity of a node n are equal to theaverage of the chromatic phasors and luminosity values of the nodes in its re-duction window. The area of a node is equal to the sum of the areas of the nodesin its reduction window, i.e. the cardinality of its receptive field.

The employed similarity measurement between two nodes is the HSV colourdistance. Thus, two nodes are similar or have a similar colour if the distancebetween their HSV values is less than a similarity threshold T . This thresholdis not fixed for all levels. Its mathematical expression is the following:

T (l) = Tmax ∗ α(l) (1)

being

α(l) ={

1 − lLreg

∗ (1 − α) if l ≤ Lreg

α if l > Lreg(2)

Lreg is the highest level of the regular part of the BIP. This threshold takesinto account that usually the receptive field of a vertex in a high level is biggerthan the receptive field of a vertex in a low level. Therefore, the linking of twovertices of a high level implies the merging of two larger regions at the base.This threshold makes more difficult this linking process at upper levels.

The graph G0 = (N0, E0) is a 8-connected graph where the nodes are thepixels of the original image. The chromatic phasors and the luminosity values ofthe nodes in G0 = (N0, E0) are equal to the chromatic phasors and luminosityvalues of their corresponding image pixels. Then, the process to build the graphGl+1 = (Nl+1, El+1) from Gl = (Nl, El) is the following:

1. Regular decimation process. In order to perform this decimation, each regularnode has associated two parameters: homogeneity Hom(i, j, l) and parentlink Parent(i, j, l). Hom(i, j, l) of a regular node is set to 1 if the four nodesimmediately underneath present a similar colour (according to the thresholdT (l)) and their homogeneity values are equal to 1. Otherwise, it is set to 0.

248 R. Marfil, A. Bandera, and F. Sandoval

If the node (i, j, l) is a node of the regular structure with Hom(i, j, l) = 1,then the parent link of the four cells immediately underneath (sons) is setto (i, j). It indicates the position of the parent of a regular node in its upperlevel. A regular node without parent has its parent link set to a NULL value.Parent links represent the inter-level edges of the regular part of the BIP.All the nodes of G0 = (N0, E0) are initialized with Hom(i, j, 0) = 1 andA(i, j, 0) = 1. Table 1 summarizes this stage in pseudocode, where d(·, ·) isthe HSV colour distance and ComputeColour() determines the colour of theinput node using the nodes in its reduction window.

2. Parent search and intra-level twining. Once the regular structure is gener-ated, there are some regular orphan nodes (regular nodes without parent).From each of these nodes (i, j, l), a search is made for the most similar regularnode with parent in its 8-neighbourhood ξ(i,j,l) (see Table 1). If this neigh-bour node is found, the node (i, j, l) is linked to the parent of this neighbournode. On the contrary, if for this node a parent is not found, then a search ismade for the most similar regular neighbour node without parent to link toit (see Table 1). If this node is found, then both nodes are linked, generatinga virtual node at level l + 1.

3. Virtual parent search and virtual node linking. Each virtual orphan node ni

searches for the most similar virtual node with parent in its neighbourhoodξni (see Table 1). If for ni a parent is found, then it is linked to it. On theother hand, if a parent is not found, the virtual orphan node ni looks forthe most similar virtual orphan node in its neighbourhood to generate a newvirtual node at level l + 1 (see Table 1).

4. Intra-level edge generation in Gl+1. The intra-level edges of Gl+1 are computedby taking into account the neighbourhood of their reduction windows in Gl.

The hierarchy stops to grow when it is no longer possible to link togetherany more nodes because they are not similar. In order to perform the pre-segmentation, the orphan nodes are used as roots. The receptive field of eachof these nodes is a region of the pre-segmented image. The described methodhas been tested and compared with other similar pyramid approaches for colourimage segmentation [7]. This comparative study concludes that the BIP runsfaster than other irregular approaches when benchmarking is performed in astandard sequential computer. Besides, the BIP obtains similar results than themain irregular structures. Fig. 1.b shows the pre-segmentation images associatedto the images in Fig. 1.a.

2.2 Perceptual Grouping Stage

After the local similarity pre-segmentation stage, grouping regions aims at sim-plifying the content of the obtained partition. As it was pointed out by [3], twoconstraints are respected for an efficient grouping process: first, although allgroupings are tested, only the best groupings are locally retained; and second,all the groupings must be spread on the image so that no part of the image isadvantaged. For managing this grouping, the BIP structure is used: the roots of

Perception-Based Image Segmentation Using the Bounded Irregular Pyramid 249

Table 1. Pseudo-code for the regular decimation, parent search and intra-level twin-ning and virtual parent search and virtual node linking steps of the pre-segmentationstage (see text for details)

//Regular decimation processfor i=0:SizeX[l], for j=0:SizeY[l],if (Hom(i,j,l) & Hom(i+1,j,l) & Hom(i,j+1,l) & Hom(i+1,j+1,l))if (d((i,j,l),(i+1,j,l))<T(l) & d((i,j,l),(i,j+1,l))<T(l) && d((i,j,l),(i+1,j+1,l))< T(l))Parent(i,j,l)=(i/2,j/2);Parent(i+1,j,l)=(i/2,j/2);Parent(i,j+1,l)=(i/2,j/2);Parent(i+1,j+1,l)=(i/2,j/2);Hom(i/2,j/2,l+1)=1; ComputeColour((i/2,j/2,l+1));

end if

//Parent search and intra-level twiningfor i=0:SizeX[l], for j=0:SizeY[l],if (Hom(i,j,l) & Parent(i,j,l)==NULL)dmin=T(l);sel=NULL;for each (m,n,l) ∈ ξ(i,j,l),if (Parent(m,n,l)!=NULL & d((i,j,l),(m,n,l))<dmin)dmin=d((i,j,l),(m,n,l));sel=(m,n,l);

if (sel!=NULL) Parent(i,j,l)=Parent(sel);elsedmin=T(l);for each (m,n,l) ∈ ξ(i,j,l),if (Parent(m,n,l)==NULL & d((i,j,l),(m,n,l))<dmin)dmin=d((i,j,l),(m,n,l));sel=(m,n,l);

if (sel!=NULL) CreateVirtualNode((i,j,l),sel);end else

end if

//Virtual parent search and virtual node linkingfor i=0:NumberOfVirtualNodes[l],if (Parent(ni)==NULL)dmin=T(l);sel=NULL;for each (nm) ∈ ξni,if (Parent(nm)!=NULL & d(ni,nm)<dmin)dmin=d(ni,nm);sel=(nm);

if (sel!=NULL) Parent(ni)=Parent(sel);elsedmin=T(l);for each (nm) ∈ ξni,if (Parent(nm)==NULL & d(ni,nm)<dmin)dmin=d(ni,nm);sel=(nm);

if (sel!=NULL) CreateVirtualNode(ni,sel); virtualnodes=true;end else

end if

250 R. Marfil, A. Bandera, and F. Sandoval

Fig. 1. a) Original images; b) pre-segmentation images; and c) obtained regions afterthe perceptual grouping

the pre-segmented regions are considered as virtual nodes which constitute thefirst level of the perceptual grouping multiresolution output. Successive levelscan be built using the virtual parent search and virtual node linking scheme pre-viously described in Section 2.1, but, in order to achieve the perceptual groupingprocess, a specific distance must be defined.

This distance uses the colour information stored in the nodes of the pyramidfor measuring the colour contrast between image regions. In order to speed upthe process, a global contrast measure is used instead of a local one. It avoids towork at pixel resolution. The contrast measure is complemented with internalregions properties and with attributes of the boundary shared by both regions.To perform correctly, the nodes of the BIP which are associated to the perceptualgrouping multiresolution output store statistics about the HSV values of theroots generated by the pre-segmentation stage which are linked to them. Then,the distance between two nodes ni and nj is defined as

Υ (ni, nj) = d(ni, nj) · min(bi, bj)bij

· min{Ai, Aj} (3)

where bij is the length of the common boundary between ni and nj , and {bi, Ai}and {bj, Aj} the total length of the boundaries and area of the nodes ni and nj ,respectively.

In order to build a new hierarchy level Gl+1, the virtual parent search and vir-tual node linking process described in Section 2.1 is applied. However, a different

Perception-Based Image Segmentation Using the Bounded Irregular Pyramid 251

threshold value Tperc is employed. The grouping process is iterated until the num-ber of vertices remains constant between two successive levels. Fig. 1.c shows theset of regions associated to the images in Fig. 1.a. It can be noted that the ob-tained regions do not always correspond to the set of natural objects presented inthe image, but they provide an image segmentation which is more coherent withthe human-based image decomposition.

3 Experimental Results

In order to evaluate the performance of the proposed approach, the BerkeleySegmentation Dataset and Benchmark (BSDB) has been employed1[9]. In thisdataset, the ground-truth data is provided by a large database of natural im-ages, manually segmented by human subjects. The methodology for evaluatingthe performance of segmentation techniques is based on the comparison of ma-chine detected boundaries with respect to human-marked boundaries using thePrecision-Recall framework [10]. This technique considers two quality measures:precision and recall. The precision (P ) is defined as the fraction of boundarydetections that are true positives rather than false positives. Thus, it quantifiesthe amount of noise in the output of the boundaries detector approach. On theother hand, the recall (R) is defined by the fraction of true positives that aredetected rather than missed. Then, it quantifies the amount of ground-truth de-tected. Measuring these descriptors over a set of images for different thresholdsof the approach provides a parametric Precision-Recall curve. The F -measurecombines these two quality measures into a single one. It is defined as theirharmonic mean:

F (P, R) =2PR

P + R(4)

Then, the maximal F -measure on the curve is used as a summary statistic forthe quality of the detector on the set of images. The current public version of thedataset is divided in a training set of 200 images and a test set of 100 images. Inorder to ensure the integrity of the evaluation, only the images and segmenta-tions from the training set can be accessed during the optimization phase. Fig. 1shows the partitions on the higher level of the hierarchy for five different im-ages. The optimal training parameters on the test set have been chosen. It canbe noted that the proposed approach is able to group perceptually importantregions in spite of the large intensity variability presented on several areas of theinput images. The pre-segmentation stage provides an oversegmentation of theimage which overcomes the problem of noisy pixels, although bigger details arepreserved in the final segmentation results (e.g., the legs of the bird in Fig. 1.c).

The F -measure associated to the individual results ranged from bad to sig-nificantly good values. Thus, the F -measure value of all images in Fig. 1 is over0.75. On the contrary, Fig. 2 shows several images which have associated a lowF -measure value. The main problems of the proposed approach are due to itsinability to deal with textured regions. Thus, the background, zebras or tigers1 http://www.cs.berkeley.edu/projects/vision/grouping/segbench/

252 R. Marfil, A. Bandera, and F. Sandoval

Fig. 2. a) Original images; and b) obtained regions after the perceptual grouping

in Fig. 2 are divided into a set of different regions. In any case, the maximalF -measure obtained from the whole test set is 0.65. Although this value is lessthan the F -measure obtained by Arbelaez [1], it is similar to the value pro-vided by other methods like the gradient paradigm implemented by Martin etal [10]. This approach optimizes, with respect to the BSDB, the combination oflocal discontinuities in feature channels like brightness, color and texture. Themain advantage of the proposed method is that it provides these results at arelative low computational cost. Thus, the processing times associated to thepre-segmentation stage are typically less than 200 ms, meanwhile the percep-tual grouping stage takes less than 150 ms to process any image on the test set.Therefore, the processing time of the segmentation approach is less than 350 msfor any image on the test set.

Finally, the proposed method requires choosing values for a set of parameters.These parameters are:

– The colour threshold, Tmax, which determines the maximum distance be-tween two colours that are considered similar at the pre-segmentationstage.

– The parameter α determines the colour threshold employed in the highestlevels of the hierarchy in the pre-segmentation stage.

– The threshold value Tperc, which determines the maximum distance betweentwo nodes that are considered similar at the perception-based stage.

In order to choose these parameters, several combinations were selected and thebest values were chosen. In our tests, the best choices for the thresholds wereTmax=50, α=0.8 and Tperc=150.

4 Conclusions and Future Work

This paper presents a new perception-based segmentation approach which istotally defined in the framework of the Bounded Irregular Pyramid. Thus, the

Perception-Based Image Segmentation Using the Bounded Irregular Pyramid 253

pre-segmentation and perceptual grouping stages are combined into the samestructure, without being necessary to define any threshold to bound these stages.The pre-segmentation is achieved using a colour-based distance and it worksfaster than similar hierarchical approaches [7]. The roots of the regions definedby the pre-segmentation stage are the first level of the hierarchy associated tothe perceptual grouping stage. This second stage employs a distance which isalso based on the colour difference between regions, but it includes informationof the area and boundary of each region. The processing time of this secondstage is also reduced because the obtained pre-segmentation regions constitutean efficient image representation.

Future work will be focused on testing different perception-based groupingparameters [5], studying its repercussion in the efficiency of the method. Besides,it is necessary that the pre-segmentation stage takes into account a texturemeasure to characterize the image pixels.

Acknowledgments

The authors have been partially supported by the Spanish Ministerio de Edu-cacion y Ciencia under project TIN2005-01359.

References

1. Arbelaez, P.: Boundary extraction in natural images using ultrametric contourmaps. In: Proc. 5th IEEE Workshop Perceptual Org. in Computer Vision, pp.182–189. IEEE Computer Society Press, Los Alamitos (2006)

2. Arbelaez, P., Cohen, L.: A metric approach to vector-valued image segmentation.Int. Journal of Computer Vision 69, 119–126 (2006)

3. Huart, J., Bertolino, P.: Similarity-based and perception-based image segmenta-tion. Proc. IEEE Int. Conf. on Image Processing 3(3), 1148–1151 (2005)

4. Lau, H., Levine, M.: Finding a small number of regions in an image using low-levelfeatures. Pattern Recognition 35, 2323–2339 (2002)

5. Luo, J., Guo, C.: Perceptual grouping of segmented regions in color images. PatternRecognition, 2781–2792 (2003)

6. Marfil, R., Rodrıguez, J.A., Bandera, A., Sandoval, F.: Bounded irregular pyramid:a new structure for colour image segmentation. Pattern Recognition 37(3), 623–626(2004)

7. Marfil, R., Molina-Tanco, L., Bandera, A., Rodrıguez, J.A., Sandoval, F.: Pyramidsegmentation algorithms revisited. Pattern Recognition 39(8), 1430–1451 (2006)

8. Marfil, R., Molina-Tanco, L., Rodrıguez, S.F.: Real-time object tracking usingbounded irregular pyramids. Pattern Recognition Letters (2007)

9. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented naturalimages and its application. In: Proc. Int. Conf. Computer Vision, vol. 2, pp. 416–423 (2001)

10. Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries us-ing local brightness, color and texture cues. IEEE Trans. on Pattern Anal. MachineIntell. 26(5), 530–549 (2004)