Scale-Space from a Level Lines Tree

Scale-Space from a Level Lines Tree

Pascal Monasse1 and Frederic Guichard2

1 CMLA, ENS Cachan61, av du President Wilson

94235 Cachan Cedex, [email protected]

2 Inrets/Dart2, av du General Malleret-Joinville

F-94114 Arcueil Cedex, [email protected]

http://www.ceremade.dauphine.fr/∼fguichar

Abstract. We decompose images into “shapes”, based on connectedcomponents of level sets, which can be put in a tree structure. This treecontains the purely geometric information present in the image, sepa-rated from the contrast information. This structure allows to suppresseasily some shapes without affecting the others, which yields a peculiarkind of scale-space, where the information present at each scale is alreadypresent in the original image.

1 Introduction

Depending on the problem at hand, different representations of images must beused. For deblurring, restoration and denoising purposes, representations basedon Fourier transform are well adapted because they rely on the generation pro-cess of the image (Shannon theory for the sampling step) and frequency modelsof degradations, for example concerning additive noise. Achieving a localizationof the frequencies, wavelets decompositions [1, 2] are known to be very efficientfor compression of images. These representations are said to be additive in thesense that they decompose the image on a given a priori basis of elementary im-ages and it is represented as the weighted sum of the basis images, the weightsbeing the coefficients of the decomposition. From the image analysis point ofview, these representations are not necessarily as well adapted because waveletsare not translation invariant, the Fourier transform is not local and both yieldquantized scales of observation.

Scale-space and edge detection theories represent the images by “significantedges”, the image being smoothed (linearly or not [3, 4]) and then convolved withan edge detector filter. This was first proposed by Marr [5] and then generalizedby [6], whereas many developments where proposed for edge detection [7].

Extraction of “edges” was shown to be generally the output of a variationalformulation [8, 9]. The image is approximated by a function from a class forwhich the definition of edge becomes clear. The balance between the precision of

the approximation and its complexity (which can be measured for example bythe length of the edges) yields a multi-scale representation of the image. Despiteits generality, this approach suffers from the absence of a good and universalmodel.

Scale-space representations based on edges are however incomplete (they donot allow to reconstruct the image) and the images at different scales are redun-dant [10, 11, 2].

Furthermore most of them do not take into account the fact that contrastmay strongly change without affecting much our perception of images, a problemunderlined and considered as central by the mathematical morphology school[12,13]. It proposes a parameter free, complete and contrast insensitive representa-tion of an image by its level sets. A recent variant [14] proposed to take as basicelements the boundaries of the level sets (called level lines), a representationnamed the “topographic map”.

Our work [15] decomposes the image into connected components of level linesstructured in a tree representing their geometrical inclusion.

This tree allows to compute easily the effect of a multi-scale operator intro-duced in section 3 which is special because it proceeds by eliminating some levellines while keeping the others without smoothing them. The advantage is thatimportant structures of the image are not damaged throughout the scale-spacederived from this operator.

The pyramidal decomposition of the image given by the tree can also beseen as a region growing decomposition (see [9] and references therein), wheretwo regions corresponding to interiors of nested level lines are merged when thesmaller enclosed region is too small. But here, no edge is moved and no spuriousedge is created. Indeed, the operator proceeds only by removing level lines, andthe contrast between two adjacent regions cannot increase, and no new gray levelis introduced.

The paper is organized as follow: Section 2 is devoted to the decompositionof images by connected components of their level sets into an inclusion tree-likestructure. Section 3 describes the natural multi-scale operator to simplify thistree, which yields a “scale-space” representation of the image. At last, in section4 some experiments are shown.

2 Level Sets and Connected Components

2.1 Contrast Insensitive Representations

Here an image u is defined as a function from a rectangle Ω = [O, W ] × [0, H ]to R being constant on each “pixel” (j, j + 1) × (i, i + 1). The value attributedto each edgel i × (j, j + 1) and (i, i + 1) × j is the max of the values atthe two adjacent pixels and at each point i × j it is the max value of u atthe 4 adjacent pixels. It is convenient to extend the image on the plane R

2 bysetting u = u0 outside Ω where u0 is an arbitrary fixed real value. This givesa continuously defined representation of a discrete array of pixels. Notice thatwith these conventions, u is upper semi-continuous.

Given an image u, upper (noted Xλ) and lower (Xµ) level sets are defined as

Xλ = x ∈ R2, u(x) ≥ λ Xµ = x ∈ R

2, u(x) < µ. (1)

u can be rebuilt from the data of any of the families of upper and lower levelsets [12, 16, 17]:

u(x) = sup λ / x ∈ Xλ = inf µ / x ∈ Xµ. (2)

The interest of these representations is their insensitivity to contrast change,that is to say g(u) and u have the same families of level sets whenever g is a realstrictly increasing function, representing a global contrast change.

A fundamental property of the level sets is their monotonicity:

∀λ ≤ µ, Xλ ⊃ Xµ, Xλ ⊂ Xµ. (3)

As in [14], to alleviate the global aspect of these basic elements, only con-nected components (cc)1 will be used (which are invariant to a local contrastchange):

Xλ =

i=Nλ⋃

i=1

cci(Xλ) Xµ =

i=Nµ⋃

i=1

cci(Xµ)

Relation (3) translates to cc’s into:

∀λ < λ′, i ∈ [1, Nλ′ ], ∃!j ∈ [1, Nλ] s.t. cci(Xλ′) ⊂ ccj(Xλ) (4)

Indeed, cci(Xλ′) ⊂ Xλ′ ⊂ Xλ and since it is connected, it is included in some ccj

of Xλ, with j unique. Equation (4) yields a tree structure for the cc’s of upperlevel sets (the same can be said of the cc’s of lower level sets).

Actually, suppose the image takes its values in the discrete set 0, . . . , U(typically, U = 255) and consider the graph where the nodes represent all cc’sof all level sets X0, . . . ,XU . Let us write N i

λ the node corresponding to cci(Xλ).Since a cc of upper level set may be extracted from several upper level sets(when it is sufficiently contrasted w.r.t. its neighborhood), suppose to avoidredundancy that only the one with greatest λ is kept. Then put a link betweenN i

λ and N jλ−1 whenever cci(Xλ) ⊂ ccj(Xλ−1). This graph is in fact a tree Tu, of

root N10 corresponding to cc1(X0) = R

2. Equation (4) ensures that the graph isconnected and without circuit, which are the two properties defining a tree. Foreach pixel P , ⋂

λ,i s.t. P∈cci(Xλ)

cci(Xλ)

is not empty (since P ∈ cc10) and is an intersection of non disjoint cc’s of upper

level sets, so that by (4), it is itself a cc of upper level set, call its associated

1 Notice that with the conventions above, connectedness corresponds to 8-connected-ness for upper level sets and 4-connectedness for lower level sets in the discretelydefined image.

node Nu(P ) and let Gray(Nu(P )) the corresponding gray level λ from which itis extracted. Then we get

u(P ) = maxλ : P ∈ Xλ

= maxλ : ∃i ∈ [1, Nλ] s.t. P ∈ cci(Xλ)

= Gray(Nu(P )) (5)

In other words, reconstructing the image from the tree is made by attributingto each pixel the gray level value of the smallest cc of upper level set that containsit. Thus the data of u is equivalent to the data of Tu and of Nu(P ) for all pixelsP . Notice that nothing obliges us to store the values Gray(N) when N is a nodeof Tu. If we do not want to store them, but still have a reconstruction formulaas (5), we attribute to each node N a gray level which is strictly decreasing whenwe follow up the (unique) path from N to the root in Tu (e.g. the depth of thenode in Tu). Then we can easily verify that the image reconstructed from it isu modulo a local contrast change. Reciprocally, if v is u modulo a local contrastchange, the trees of u and v are the same.

A

2

1

2 0

B

C

D

A BC

D

F

G

E

E F G

Fig. 1. The trees Tu (left) and Tl (right) of an image.

All the above results can be stated with the appropriate changes for lowerlevel sets to construct another tree Tl, and for each pixel P the node associatedto the smallest containing cc of upper level set, Nl(P ). An example of such adecomposition is given in fig. 1. This is what is done by [18].

2.2 The Inclusion Tree

Each one of the trees Tu and Tl satisfy our requirements stated in the introduc-tion, nevertheless they make an a priori choice of the “objects” in the image:Tu is adapted to clear objects on a darker background. We would like to dealsimultaneously with clear objects and with dark objects. It is not satisfactoryto keep both trees, because they are redundant since each one individually issufficient to represent the image. Thus, we have to eliminate some cc from thetrees. Two principles will guide us:

1. The “interesting objects” in u stretch over a finite portion of the plane.2. The “interesting objects” have no holes.

Our “interesting objects” will be cc’s of upper or lower level sets. The firstprinciple tells us to eliminate non bounded cc of level sets. The second principleallows us to build a new tree from the remaining nodes.

J J

J

2

0

CIntJ1 1

IntJ 2

IntJ 0

Fig. 2. C is a cc of a level set. Three level lines compose its border. The exterior levelline is J2. Int J2 = C ∪ Int J0 ∪ Int J1 is shown at the right.

Each bounded cc of level set C has a topological border that is composed ofone or several sets of connected edgels, called level lines. A level line separatesthe plane in two disjoint connected parts2, its interior (which is the boundedpart) and its exterior (the unbounded one) [27]. C is comprised in the interiorof one of the level lines composing its frontier, called its exterior border, and inthe exterior of all others, called its interior borders (see fig. 2). The interiors ofits interior borders constitute its holes. The principle 2 leads us to consider notC but C union its holes, which we call a “shape” S(C). The border of S(C) isnow only the exterior border of C. Therefore, we are led to consider the interiorsof level lines.

A property similar to (4) can be proved for shapes, namely that if two shapesare not disjoint, one of them is included in the other. This relies on the fact thatlevel lines do not cross: a level line cannot meet altogether the interior and

2 The connectedness considered here does not correspond to the topological connected-ness in the continuous plane R

2, but to the discrete notions of connectedness (4- and8-connectedness). More precisely, if C comes from an upper level set, the part of theplane containing C is taken in 8-connectedness and the other part in 4-connectedness,whereas it is the contrary if C is extracted from a lower level set.

2 0

1

A

G

2

D

F

A

D

FG

Fig. 3. The inclusion tree T corresponding to a simple image. Notice that D is a holein F . Compare with the upper and lower trees Tu and Tl given in fig. 1

the exterior of another level line. The proof of this is not trivial, and involveshypotheses on the image (semi-continuity is a sufficient condition), see [21].

The following operations are done for constructing the inclusion tree: Asso-ciate a node to each shape. Consider the entire plane (which is not a shape,because not bounded), as the root node. Put a link between two nodes wheneverone of the shapes is included in the other and no third shape can be insertedbetween both. The resulting graph is a tree T , constructed from bounded cc’sof both upper and lower level sets [15]. The “interesting objects” of our twoprinciples are now represented in one single tree (see fig. 3). For each pixel P , weassociate also the node N(P ) in T associated to the smallest shape containingP .

A reconstruction formula similar to (5) holds:

u(P ) = Gray (N(P )) .

2.3 Summary

We consider images made of closed upper level sets whose topological boundariesare a finite number of “level lines”. Such class of images contains discrete images(pixel-wise defined), or functions having a minimal regularity.

– We call shape the interior of a level line.

– Level lines are closed curves that do not cross.

– Thus two shapes are either disjoint, so that they are either contained in athird shape, or nested, in which case one is a descendant of the other. Thisyields a tree structure for the set of shapes, where the relation child-parentmeans the topological inclusion.

3 Scale-Space Representation

3.1 A Multi-Scale Operator

For a set B, we denote by |B| its area, i.e. Lebesgue measure, or any othermeasure which is increasing with respect to the inclusion of the sets (if B ⊂ Cthen |B| ≤ |C|). For a connected set B, we call its filled interior φ(B) the unionof B and its holes and its filled area the area of its filled interior. In other words,φ(B) is the smallest simply connected set containing B. Let Bt be the family ofclosed connected sets B whose filled interior contains the origin O, |φ(B)| ≥ tand such that if O is in a hole H of B, then |φ(H)| < t.

Let us introduce our multi-scale operator:

Ttu(x) = supB∈Bt

infy∈x+B

u(y). (6)

The operator Tt applied to the image u is equivalent to removing all theshapes of area less than t from the inclusion tree of u and constructing back theimage. This yields another formulation of the operator T :3

T ′tu(x) = inf

B∈Bt

supy∈x+B

u(y). (7)

This operator is at the same time a morphological opening and closing [13]!It is close to a filter proposed independently by several authors [22, 18, 23, 24]but in their case the applied filter was equivalent to remove only nodes from thetree of cc’s of superior level sets or from the tree of inferior level sets, so thatthey get two different operators which do not commute (the opening and theclosing version). The operator presented here is the grain filter studied in [19].

3.2 Properties

Let us consider the properties of this multi-scale operator:4

[Causality] The scale-space is causal, meaning that each scale can be de-duced from any anterior scale by a transition operator.

∀s, t, s ≤ t, ∃Tt,s so that Tt = Tt,s Ts (8)

A remarkable property is that the transition operator is the operator itself:Tt,s = Tt.

[Monotonicity] This scale-space is monotonous:

u ≤ v ⇒ ∀t, Ttu ≤ Ttv (9)

3T = T

′ if one switches the connectedness for upper and lower level sets. We conjecturethat they are also equal when acting on continuous functions.

4 Some of the properties suppose that the image is at least continuous, which is impos-sible with the continuously defined versions of discretely defined images we considered(except for trivial cases). Nevertheless whereas the notion of the inclusion tree is notclear in such a case, the operator Tt can be defined as in equation (6).

[Contrast covariance] If g is a contrast change (a strictly increasing realvalued function), then

∀t g Tt = Tt g (10)

[Negative covariance] Some other interesting feature of the operator isits negative covariance, that is that it commutes with taking a negative of theimage:5

∀t Tt(−u) = −Ttu (11)

Notice that this is not the case with the operator defined in [18], where regionalmaxima and regional minima do not play symmetrical roles.

[Local extrema conservation] A local regional extremum in the image uremains either a local regional extremum at scale t, or is included in a biggerlocal regional extremum or disappears. In other words, regional extrema cangrow, but they are never split during the scale-space and the operator proceedsby growing local regional extrema. Moreover, at scale t each regional extremum ofTtu contains a regional extremum of u. This implies that the number of regionalextrema is a decreasing function of the scale. Notice that this property is nottrue with the linear scale-space (convolution by a gaussian) in two dimensions.

[Idempotent] The operator has the property to be idempotent.

∀t, Tt Tt = Tt (12)

[No asymptotic evolution] If u is C2, there is no asymptotic evolution ofthe image. We have the two behaviors:

∀x,∇u(x) 6= 0 ⇒ ∃t so that ∀h ≤ t, (Thu − u)(x) = 0 (13)

∀x, if ∃r > 0 so that ∀y ∈ B(x, r),∇u(y) = 0 then

∃t so that ∀h ≤ t, (Thu − u)(x) = 0(14)

[Conservation of T-junctions] Since the level lines at Ttu are level linesof u, the T-junctions involving sufficient areas in u remain the same withoutalteration. Notice that this is not the case with all other usual scale-spaces: itis clearly false for the linear scale-space, for the median filter (mean curvaturemotion), but also for the affine invariant morphological scale-space, as shownin [25].

[Conservation of some regularity] We conjecture that if u is continuous,so is Ttu for all t (a demonstration for the case of area opening and closing isshown in [26], we think our operator would behave similarly). If u is moreoverLipschitz, so is Ttu with an inferior or equal Lipschitz norm. Nevertheless, we

5 However, −u is lower semi-continuous, so that appropriate changes of connectednessmust be applied: lower (resp. upper) level sets of −u must be considered in 8- (resp.4-) connectedness.

cannot say anything more about regularization: it is not true that if u is C1 sowould be Ttu. This scale-space is peculiar because it does not allow to estimatemore reliably the results of differential operators!

[Affine covariance] The operator commutes with all affine transforms ofdeterminant 1:

∀t, ∀A ∈ GA(R2), Tt(u A) = (Tt/| detA|u) A (15)

Notice that equations (8), (9), (10) and (15) are properties that our scale-space share only with the affine morphological scale-space. Nevertheless, thelatter has an infinitesimal evolution law, whereas the former has not.

Remark: the geometrical covariance of the operator is linked to the geomet-rical invariance of the measure, here the area under any affine transformationof determinant 1. With a different measure invariant under another group oftransformations preserving the connectedness (so probably continuous transfor-mations would be welcome) and non decreasing with respect to inclusion, ouroperator would commute with these transformations.

4 Experiments

Fig. 4 illustrates the fact that this scale-space is different from the one deducedby iterating area opening and closing (see [18]) with increasing area.

Different scales of the scale-space based on the inclusion tree are shown infig. 5. Another example showing also the level lines is shown in fig. 6. Notice howthe important structures of the image (in particular T-junctions) are preserved.

The inclusion tree can also be used to remove impulsional noise: supposingthat speckle noise creates only small shapes, we represent the image at a suffi-ciently large scale (see fig. 7). This suppresses most of noise, without attemptingto restore the image, so a subsequent treatment should follow [19].

Other uses of the inclusion tree are proposed in [15].

Fig. 4. Left: an image. Up: Successive removals of the cc’s of upper, then lower levelsets with increasing area threshold. The black ring disappears before the white circle.Down: The image across the scales of the inclusion tree: the circles disappear accordingto their interior size.

Fig. 5. Up-left: original image 650×429, Up-right: image at scale 50 (all shapes of arealess than 50 pixels are removed), Down: image at scale 500, and 5000.

5 Summary and Conclusions

The inclusion tree is a complete and non-redundant representation of image,insensitive to local changes of contrast. The basic elements are the interiors ofconnected components of level lines, called “shapes”. The structure of tree rep-resents the geometrical inclusion, allowing to easily manipulate it, like remov-ing some shapes, which is the fundamental operation. This yields a scale-spacerepresentation of the image which, on the contrary to most other scale-spacerepresentations, does not smooth the image, but rather selects the informationto keep at each scale. As a consequence, its application field will be differentfrom the classical scale-space.

These shapes, appearing as natural geometrical contrast insensitive informa-tion, can also be used for various image analysis tasks, like image simplification,image comparison and registration [15, 20].

Acknowledgments

The authors thank Jean-Michel Morel and Vicent Caselles for fruitful discussionsabout the subject. Part of this work has been done using the MegaWave2 imageprocessing environment (http://www.ceremade.dauphine.fr), Cognitech, Inc.image processing facilities (http://www.cognitech.com), and Inrets facilities.

References

1. Meyer, Y.: Wavelets: Algorithms and Applications, SIAM, Philadelphia (1993)2. Mallat, S.: A Wavelet Tour of Signal Processing, Academic Press (1998)

Fig. 6. Up-left: original image 508×500, Up-right: image at scale 150, Down: the levellines of the original image (left) and of the image at scale 150 (right). All the level linesdisplayed in right are present and identical to level lines of the original image (left).

3. Alvarez, L., Guichard, F., Lions, P.L., Morel, J.M.: Axioms and Fundamental Equa-tions of Image Processing, Arch. Rational Mech. and Anal., 16, 9 (1993) 200–257

4. Ed. Romeny, B.M. ter Haar: Geometry-Driven Diffusion in Computer Vision,Kluwer Academic Publishers (1994)

5. Marr, D.: Vision, Freeman and Co. (1982)6. Hummel, R.A.: Representations Based on Zero-Crossing in Scale-Space, Proc. of

CVPR. IEEE (1986) 204–2097. Canny, J.: A Computational Approach to Edge Detection, IEEE Trans. on PAMI,

8, 6 (1986) 679–6988. Nitzberg, M., Mumford, D.: The 2.1 Sketch, Proc. of ICCV, Osaka, Japan (1990)9. Morel, J.M., Solimini, S.: Variational Methods in Image Processing, Birkhauser

(1994)10. Koenderink, J.J.: The Structure of Images, Biological Cybernetics, 50 (1984) 363–

37011. Witkin, A.P.: Scale-Space Filtering, Proc. of IJCAI, Karlsruhe (1983) 1019–102212. Matheron, G.: Random Sets and Integral Geometry, John Wiley, N.Y. (1975)13. Serra, J.: Image Analysis and Mathematical Morphology, Academic Press (1982)14. Caselles, V., Coll, B., Morel, J.M.: Topographic Maps, preprint CMLA (1997)15. Monasse, P., Guichard, F.: Fast Computation of a Contrast-Invariant Im-

age Representation, Preprint CMLA 9815, available from http://www.cmla.ens-cachan.fr/index.html (1998)

Fig. 7. Left: an image (size 240 × 255) where 50% of the pixels are changed to anarbitrary value. Right: this image at scale 30 pixels.

16. Guichard, F., Morel, J.M.: Partial Differential Equations and Image Iterative Fil-tering, Tutorial ICIP, Washington D.C. (1995)

17. Guichard, F., Morel, J.M.: Partial Differential Equations and Image Iterative Fil-tering, State of the Art in Numerical Analysis (1996)

18. Vincent, L.: Grayscale Area Openings and Closings, Their Efficient Implementationand Applications, Proc. of 1st Workshop on Math. Morphology and its Appl. toSignal Proc., J. Serra and Ph. Salembrier, Eds. (1993) 22–27

19. Masnou, S., Morel, J.M.: Image Restoration Involving Connectedness, Proc. ofthe 6th Int. Workshop on Digital I.P. and Comp. Graphics, SPIE 3346, Vienna,Austria (1998)

20. Monasse, P.: Contrast Invariant Image Registration, Proc. of ICASSP, Vol. 6,(1999) 3221–3224

21. Monasse, P.: An Inclusion Tree Describing the Topological Structure of an Image,in preparation

22. Cheng, F., Venetsanopoulos, A.N.: An Adaptive Morphological Filter for ImageProcessing, IEEE Trans. on PAMI, Vol. 1, 4 (1992) 533–539

23. Andrew Bangham, J., Ling, P.D., Harvey, R.: Scale-Space from Nonlinear Filters,IEEE Trans. on PAMI, Vol. 18, 5 (1996) 520–528

24. Andrew Bangham, J., Ling, P.D., Harvey, R., Aldridge, R.V.: Morphological Scale-Space Preserving Transforms in Many Dimensions, Journal of Electronic Imaging,Vol. 5, 3 (1996) 283–299

25. Caselles, V., Coll, B., Morel, J.M.: Is Scale-Space Possible?, Proc. of the 1st Work-shop on Scale-Space Theories in Computer Vision, Utrecht, the Netherlands (1997)

26. Guichard, F., Morel, J.M.: Image Iterative Filtering, Lecture Notes of InstitutHenri Poincare (1998)

27. Kong, T.Y., Rosenfeld, A.: Digital Topology: Introduction and Survey, CVGIP,Vol. 48, 3 (1989) 357–393