The Gaussian Scale-Space Paradigm and the Multiscale Local Jetmate.tue.nl/mate/pdfs/5162.pdf · 1)-dimensional scale-space neighbourhood. 2. Theory In order to appreciate the notion

International Journal of Computer Vision, 18, 61-75 (1996) (9 1996 Kluwer Academic Publishers, Boston. Manufactured in The Netherlands.

The Gaussian Scale-Space Paradigm and the Multiscale Local Jet

LUC FLORACK DETIINESC-Aveiro, Universidade de Aveiro, 3800 Aveiro, Portugual, and Computer Vision Research Group,

Utrecht University Hospital, Heidelberglaan 100, NL-3584 CX Utrecht, The Netherlands

luc @ cv. ruu.nl BART TER HAAR ROMENY

Computer Vision Research Group, Utrecht University Hospital, Heidelberglaan 100, NL-3584 CX Utrecht, The Netherlands

bart @ cv. ruu. nl MAX VIERGEVER

Computer Vision Research Group, Utrecht University Hospital, Heidelberglaan 100, NL-3584 CX Utrecht, The Netherlands

max@ cv. ruu.nl JAN KOENDERINK

Dept. of Medical and Physiological Physics, Utrecht University, Princetonplein 5, NL-3584 CC Utrecht, The Netherlands

Received September 1, 1994; Accepted January 11, 1995

Abstract. A representation of local image structure is proposed which takes into account both the image’s spatial structure at a given location, as well as its “deep structure”, that is, its local behaviour as a function of scale or resolution (scale-space). This is of interest for several low-level image tasks. The proposed basis of scale-space, for example, enables a precise local study of interactions of neighbouring image intensities in the course of the blurring process. It also provides an extrapolation scheme for local image data, obtained at a given spatial location and resolution, to a finite scale-space neighbourhood. This is especially useful for the determination of sampling rates and for interpolation algorithms in a multilocal context. Another, particularly straightforward application is image enhancement or deblurring, which is an instance of data extrapolation in the high-resolution direction.

A potentially interesting feature of the proposed local image parametrisation is that it captures a trade-off between spatial and scale extrapolations from a given interior point that do not exceed a given tolerance. This trade-off suggests the possibility of a fairly coarse scale sampling at the expense of a dense spatial sampling (large relative spatial overlap of scale-space kernels).

The central concept developed in this paper is an equivalence class called the multiscale Zocal jet, which is a hierarchical, local characterisation of the image in a full scale-space neighbourhood. For this local jet, a basis of fundamental polynomials is constructed that captures the scale-space paradigm at the local level up to any given order.

1. Introduction ture of image structure (Witkin, 1983), (Koenderink, 1984b), (Koenderink and Van Doorn, 1987, 1990),

A crucial aspect to be taken into account in the de- (Babaud et al., 1986) (Lindeberg, 1990ab), (Florack scription of local image structure is the resolution, et al., 1992, 1993, 1994) In particular, such a free or equivalently, the inner scale of spatial structures. scale parameter provides a sensible meaning to the In scale-space theory, a free scale parameter is intro- notion of locality, independent of the image’s un- duced to account for the intrinsically multiscale na- derlying sampling characteristics (pixels, voxels), by

62 Florack, et al.

“blowing up” an infinitesimal neighbourhood to a finite extent.

The unconfounding of sampling grid and local image descriptors is important, because one usually does not care about the grid; images are sampled and ren- dered on various devices (including the human retina) based on fundamentally different grids. One can even display a greylevel image on a black-and-white de- vice without affecting its relevant content, provided the details of interest are sufficiently large relative to the grid constant (cf. the halftone images in your ev- eryday newspaper). Since the grid entails inevitable sampling and display artifacts, one would like to minimise its disturbing effect on any task the image is intended to support. In practice this means that the sampling grid constant must be small relative to the smallest inner scale of interest.

In view of the above arguments, it is natural to abandon local image descriptors that rely explicitly on the grid. Typical examples of these are the 5- point Laplacean widely applied in 2-dimensional image processing for the extraction of zero-crossings, the 2-point difference quotient intended to approxi- mate a first order image derivative, etc. Operations like these should only show up as small-scale limiting cases (i.e. at scales close to the grid constant).

According to the theory of regular tempered distributions (Schwartz, 1950), a derivative of a (not necessarily smooth) function LO can be obtained in a well-posed way by correlating it with (i.e. taking the scalar product with) the conjugate derivative of a smooth test function, 3’ E Cm(lR) say:

Here, ail...i, denotes the n-th order derivative w.r.t. the spatial variables x z k ( k = 1,. . . , n), in which each index il, has a value from the range 1,. . . , d, labelling the d axes of some arbitrary Cartesian coordinate frame centred at 2 = 0’. The input function is denoted by Lo. The factor (-l)n expresses the anti- symmetric nature of differentiation in a real function space; it is just the sign factor that shows up in an n-fold partial integration of (1).

The scale-space paradigm is in fact an operational instance of Schwartz theory, in which the test functions y are constructed to account for the physical notion of scale. To this end, the test functions are taken to be translated and isotropically scaled ver-

sions of a basic, normalised Gaussian kernel. Hence each Gaussian is characterised by its centre location 5 and width 0, the spatial location and scale of interest, respectively.

Since we will be dealing with local aspects of image structure, it is convenient to restrict attention to a single base point 2 = 0’. Clearly this is inessential; the generalisation of (1) to any other spatial location is obtained by a mere shift, thus yielding the more familiar convolution expression:

in which ya denotes the normalised Gaussian of width 0, centred at 2 = 0’. The restriction to a Carte- sian frame is likewise immaterial; we may assume (1) and (2) to hold in any (not necessarily recti- linear) coordinate system, such as a polar coordinate system, simply by interpreting the derivatives as covariant derivatives. This will be done through- out the paper. For the sake of notational simplicity we will use the Einstein summation convention for each pair of identical upper and lower indices. In- dices may be raised and lowered by means of the metric tensor. See also (Kay, 1988), (Lawden, 1962) for an easy introduction to classical tensor calculus, and (Spivak, 1970), chapter 4, vol. I for a more elab- orate exposition.

The correlation scheme based on Gaussian derivative filters is pretty straightforward. One can use it to obtain a discrete set of local measurements (the Li1..,2,, for n = 0, . . . , N , say), and one can repeat this at any discrete number of locations and scales. Although (2) is a locally smooth representation from a pure mathematical point of view, it is important to appreciate that smoothness of this representation does not have any operational significance. That is to say, one cannot store a smooth representation in terms of mere numbers on a physical medium. What can be represented, however, is a routine that enables one to retrieve any sample from some presumed smooth representation. In other words, smoothness is not an attribute of the measurements, despite the smoothness of the underlying kernels by which they may have been obtained. Even worse, the very scale-space paradigm cannot sensibly be attributed to the (virtu- ally arbitrary!) local measurements; any information reminiscent of their prior kernels has been integrated out. From these arguments it is clear that one needs

The Caussian Scale-Space Paradigm and the Multiscale Local Jet 63

a dual representation (in the form of a local routine) that captures the scale-space paradigm in addition to the measurements. It is the purpose of this paper to make this explicit.

To this end, we propose a smooth, finite parametrisation of local image structure consistent with the scale-space paradigm. It takes into account both the image’s spatial structure at a given location, as well as its “deep structure”, that is, its local behaviour as a function of scale. This is of interest for fundamental as well as practical low-level image problems. For example, the parametrisation enables a precise study of local interactions of neighbouring image intensities when changing inner scale (Damon, 1990), (Johansen et al., 1986), (Lindeberg, 1992ab). It also provides a scheme for a smooth extrapolation of local image data, obtained at a given spatial location and resolution, into a finite scale-space neighbourhood (not confined to the grid). The extent of the extrapolation region can be related to an accuracy mea- sure. This is especially useful in the determination of sampling strategies (spatial overlap of kernels, scale discretisation). Finally, one may use the parametrisation for image enhancement or deblurring gen- eralising the well-known scheme based on subtrac- tion of the Laplacean of the image (which turns out to be the lowest order case) (Hummel et al., 1987), (Kimia and Zucker, 1993), (Wang et al., 1983).

The central concept in this paper is the multiscale local jet, which will be explained in the next section. It is basically a hierarchical, local characterisation of the image (or rather, of an equivalence class of metamerical images (Koenderink, 1992)) in afull, (d + 1)-dimensional scale-space neighbourhood.

2. Theory

In order to appreciate the notion of a multiscale local jet, consider the scale-space L : Rd x lR+ -+ lR : ($0 ) H L(< a). It is constrained by the diffusion equation

(A - g) L = o (3)

The evolution paratneter t relates to spatial scale according to 2s = 0‘ if t = s - SO. In other words, t = 0 corresponds to a physical inner scale (T = go,

which will be assumed to lie somewhere inbetween

physically sensible limits, say max(E/cO, ao/@) < 1, where E denotes the grid constant (pixel or voxel sep- aration) and e the size of the image domain or region of interest.

The multiscale local jet of order N can now be defined as the equivalence class of images that have the same local structure up to N-th order (inclusive), in space as well as in scale. This is nontrivial, because scale and space are confounded by (3). Consequently, the order of the jet relates to both space and scale in some dependent way, and has to be given a precise meaning.

A simple way of parametrising a local jet is by considering a truncated local Taylor expansion that forms the common lowest order part of all local jet members. The coefficients in that expansion then parametrise the local jet. This definition seems to require a choice of a local coordinate system. It must be realised, however, that the concept of a local jet is really coordinate independent. In order to make this coordinate independence manifest, the coefficients in the Taylor expansion are to be regarded as covariant image derivatives, that is, image derivatives that behave as covariant tensors.

In order to make the above definition more precise, consider a formal expansion near the origin (.’,t) = (d,O) of a coordinate system centred at a given interior point (recall that t corresponds to a physical scale a, so t = 0 corresponds to some interior point a = 00). If (E, 6t) is a “small” excursion away from the origin, “small” meaning 6t = O(s0) and IIS511 = O(ao), then the formal expansion reads

L,(S?, S t ) = w o 3 . .

in which a subscript of L refers to a covariant spatial derivative and a parenthesised superscript to a scale derivative, evaluated at the point of interest. It is understood that all coefficients are to be evaluated at the origin (?,t) = (6,0), unless indicated otherwise. It will turn out that these coefficients can be obtained by convolving the input image with certain spatial derivatives of the Gaussian kernel corresponding to the level of scale at the point of interest.

The aim is to truncate the expansion to obtain a hierarchy of polynomials of increasing order which identically satisfy the diffusion constraint (3) for

64 Florack, et al.

proper scale-space behaviour. These ordered polynomials may serve to represent the scale-space of L near any given point with increasing accuracy. They may be used to represent the image's local jet of a given order, that is, the equivalence class of all images with a certain order of contact at the point of interest. The notion of contact now refers to both its spatial as well as its scaling behaviour, and because these are linked through diffusion it is clear that the coefficients in (4) are interrelated as well. Indeed, all t-derivatives are determined by spatial derivatives only, for we have, for all j > 0:

or, upon resolving the recursion:

Here and henceforth, g i j denotes the contravariant metric tensor, that is, the inverse of the covariant metric tensor g i j . In a Cartesian coordinate system they both boil down to the invariant Kronecker sym- bol S i j (that is, 1 if 1: = j , 0 otherwise), and one may forget about the distinction between covariant and contravariant indices. In any case, the length of the infinitesimal arc connecting xi to xi + dxi is, by definition, given by the covariant metric tensor:

d12 = gijdx"dxi . (7)

The Laplacean aL/dt = A L (at the point of interest) is the lowest order and most familiar instance of (6). Equations ( 5 ) and (6) follow from substituting the formal expansion (4) into the diffusion equation (3), using

n n

A{x i l . . . xi.'} = g k l 1 1 (8) s=l,s#r r=l

. . zil. . .z2p-~d~1.x~7+i k . . . z i , - i ~ % z Z s + i 1 . . .XZ" ,

or, equivalently, by exploiting the linearity of (3). So the full solution to the diffusion equation in a

neighbourhood of the origin is given by the expansion

1 1 n! j !

L,(SZ,St) = y,y;-- n=O j=o

(9)

which can also be written as

L,(62, 6 t ) = exp { 6tA + 6 3 . ?} L . (10)

This condensed formulation nicely reveals the role of the Laplacean as the infinitesimal generator of rescal- ings, quite similar to the role of the gradient as the generator of translations. The expansion (10) is valid on any finite (2, t)-neighbourhood of the origin.

Our objective is to determine how to truncate the expansion such as to retain full compatibility with the diffusion equation. In other words, we want to consider a finite, N-th order approximation L ~ ( 6 2 , 6t) of L, ( 6 2 , 6 t ) with exact scale-space properties.

PROPOSITION 1 (SCALE-SPACE POLYNOMIALS) The N-th order polynomial solution to the diffusion equation is given by

Here, [ X I denotes the entier of x, that is the largest integer n for which n 5 x.

The finite expansion in Proposition 1 represents the image's multiscale local N-jet. Before turning to the proof, note that the N-th order polynomial LN is

(n) a sum of n-th order, homogeneous polynomials L , each of which contains homogeneous monomials of degree n. For consistency, one has to attribute an order or degree of homogeneity to t which is twice that of xi (dimensional analysis):

LEMMA 1 (HOMOGENEOUS POLYNOMIALS) The N-th order polynomial LN dejned in Proposition I can be written as

n=O

(n) with homogeneous polynomials L = Ln - Ln-l (L-1 = 0 ) given by def

The Gaussian Scale-Space Paradigm and the Multiscale Local Jet 65

Lemma 1 is merely an order-by-order rearrangement of terms in Proposition 1. Now it is easily

verified that the homogeneous polynomials L , and hence the L N , indeed satisfy the diffusion equation:

(n )

Proof of Proposition 1: From (8) it follows, after some dummy index manipulations, that

1 - 1 lnlal-1

( n ) n L (62,6t) = (. - 2 j - 2)! j ! j =O

g k l h . , . y k , + l b + l L 2 1 . . . z , , - 2 , - z k 1 1 1 . . l C , ) k 1 J + 1

Sx‘1. . .6z‘V - 2 J - 2 6 t 3 .

(n) One readily verifies that this is just & L ( S Z S t ) .

The L have a definite parity, viz. (-l)n. Since the coefficients in Lemma 1 may have arbitrary values, they can be put in isolation so as to yield an n-th order, symmetric contratensor, which represents a fundamental, n-th order, homogeneous polynomial solution to the diffusion equation:

(n)

COROLLARY 1 ( P o L Y N o n l I A L BASIS) The n-th order homogeneous polynomial solution to the diffusion equation can be written as

in terms of the basis polynomials

in which parentheses surrounding upper indices denote index symmetrisation.

The homogeneous polynomials Pal ...z7i are independent of the measurement data. They can apparently be contracted onto arbitrary n-th order cotensors Lt,,,,z,, so as to yield polynomials that mani- festly satisfy the diffusion equation, thus providing a smooth interpretation of the point measurements on a

full local neighbourhood in space and scale. In other words, regardless of how the measurements are obtained, the contraction onto the corresponding fundamental contratensor may well be regarded as the exact local scale-space of some underlying d-dimensional input signal: at the local level, nothing indeed re- minds us of a prior Gaussian filtering stage. Re- versely, given the initial d-dimensional data, the n-th order cotensor Ltl...zn represents its n-th order covariant spatial derivative at a given interior point; its multiscale local jet is then represented by Lemma I . Thus the fundamental tensor polynomials in Corol- lary 1 are the dual objects which, by contraction onto the corresponding measurements, enforce a smooth internal representation by the scale-space paradigm. Note that, because of symmetrisation, the number of essential components in Corollary 1 equals

The degrees of freedom that have been discarded by truncation lead us to the issue of metamerical images (Koenderink, 1992). Recall that our starting point was the formal expansion of a locally defined scale- space (9), and our aim was to reconcile this with a set of local measurements up to some finite order. This led to the operationally well-defined expression in Proposition 1. Since both expressions satisfy the diffusion equation, so does their difference R~(62,S t ) = L,(63,6t) - L ~ ( 6 2 , S t ) (we will call it the ghost image), by virtue of linearity. This ghost image can be represented in terms of the polynomials Pzl...zrb (65, S t ) , for n 2 N -t 1. Since the cotensors in this polynomial representation do not correspond to actual measurements, they must be considered as mere formal parameters. They constitute a local, order by order characterisation of an infinite-dimensional class of scalar field configu- rations that induce identical local measurements of orders 0, . . . , N . For this reason the hypothetical images, obtained by arbitrary extensions of the physical measurement data by some cross-section of the formal parameter space, are called metamerical images.

What about the rate of convergence of the expansion? This is obviously a crucial question if a truncation of the expansion is required to be “sufficiently” accurate within a certain finite neighbourhood of the central point of expansion. Let us therefore take a closer look at the nature of the ghost image.

. .

66 Florack, et al.

Definition. Let R be some neighbourhood containing (2, t ) = (6,O) E IRd x IR, such that (62, S t ) E R implies that the path C = {(2,t)=(XS2, X’St) 1 X E I = [0,1]} is contained in R. Then the homotopy A : R x I + IR for the image L : R -+ IR is defined by

A(Slc’, S t ; A) = L(XS?, A’&). (12)

The homotopy A describes the smooth transition of greylevels one encounters when walking away from the origin (6,O) along a specific path C c R towards a given neighbouring point (62, S t ) . At the endpoints we have L(6,O) = R(S2,St;O) and L(S2,St) = A(S2, S t ; 1). The path itself is obtained by a scaling of the excursion parameters (S? ,S t ) . Note that R is star-shaped in the (Z, a)-domain, not in the (2, t)- domain.

In order to appreciate the greylevel deviation one encounters during the traversal of the path, consider the Taylor expansion of A(&?, St; A) near X = 0, and evaluate the result at X = 1 (which is possible by virtue of the star-shaped domain). Recall from analysis that the Taylor expansion can be derived by re- peated partial integration of the identity

A(SZ, S t ; 1) = A(S2, S t ; 0) t

which immediately gives us the exact integral form for the remainder after N-fold repetition:

COROLLARY 2 (PHYSICAL vs. GHOST IMAGE) The full solution to the difusion equation, L,(S2, S t ) , can be separated into a finite dimensional “physical image”, LN (62, S t ) , and an infinite dimensional ghost image, RN (SZ, St), as follows:

Lm(SZ, S t ) = L ~ ( 6 2 , S t ) + R ~ ( 6 2 , S t ) , with

1 d n h n! dXn

N

L N ( S 2 , S t ) = --(62,6t; 0 ) , n=O

RN(S?, S t ) =

and with A given by (1 2).

choice of homotopy: Now we see the motivation behind our particular

(n) OBSERVATION 1 Let L ( S 2 , S t ) be dejined as in Lemma 1. Then

1 dnA (n)

n! dXn (62, S t ; 0) = L (S?, S t ) , ___

with the identiJcation given by the scale-space dejin- ing difusion equation

= A . d at -

From Corollary 2 and Observation 1 it follows that the remainder R ~ ( 6 2 , S t ) in the expansion of the homotopy is indeed the local ghost image L,(S?, S t ) - L ~ ( S 2 , 6 t ) , that is, all “lacking evidence” for an N- th order local measurement.

In some sense one would like to minimise the effect of the ghost image, since it induces an uncertainty in the measurements even in the hypothetical case that the local measurements are obtained with infinite pre- cision. It is therefore of fundamental interest to find an upper bound for the ghost image as a function of order N and excursion parameters ( S 2 , S t ) . In fact, we immediately obtain the following result:

LEMMA 2 (GHOST IMAGE) Let R~(62 ,S t ) be defined as in Corollary 2. Then there exists a point A E I on the path defined by the homotopy (12), such that

(62, S t ; A) . 1 dN+lA ( N + l)! dXN+1 RN (62, S t ) =

This is a standard way of rewriting the integral expression for the remainder RN (62, S t ) in Corol- lary 2, known as the formula of Lagrange, which follows straightforwardly by virtue of continuity of the derivative in the integrand as a function of X (A E I is the parameter value at which the ( N + 1)-th order derivative of A(S2, S t ; A) attains its maximum value on the path C). It is clear that the ghost image must vanish at the origin, and that, by convergence of the Taylor series, it is a monotonically decreasing function of order N. However, Lemma 2, as it stands, is not a very useful way of writing the ghost image,


since it does not make the (62,St)-dependence explicit. Since A(&?, S t ; A) is just L(AS2> X 2 S t ) , we may substitute A-derivatives for 22- and t-derivatives using the chain rule together with Leibnitz's product rule. Since this is a rather tedious and cumber- some exercise, we will only present the result here and leave the details to the appendix:

RESULT 1 (GHOST IMAGE) Cj: Lemnza 2:

Here, it is understood that each of the k t-derivatives stands for a Laplacean as usual.

By taking terms with index k = N - q + 1 apart, we see that

which entails Observation 1 as a special case (viz. for which = 0).

Result 1 shows that the value of the ghost image at the point (62, S t ) can be written as ajnite polynomial of excursion parameters, rather than an infinite series, the coefficients of which are image derivatives of orders N + 1, . . . , 2 ( N + 1) evaluated at some point ( b 2 , i 2 6 t ) on the path C that connects (S2,St) to the origin. This may come as a surprise, since the ghost image clearly has an infinite number of degrees of freedom. To resolve this apparent paradox, recall that the formal parameter E I corresponds to the point on the path C at which the ( N + 1)-th order derivative of A(&?, S t ; A) attains its maximum value. This path, and hence the value of the formal parameter A, depends on the endpoint (S2 ,b t ) . Thus the function i(62, S t ) is the rug that covers the missing degrees of freedom.

3. Examples

3.1. Some Lowest Order Cases

Up to fourth order, the homogeneous polynomial solutions are given by

Addition of these, with the coefficients given by the corresponding fixed-scale spatial derivatives of a given image, will yield a representative member of the image's multiscale local 4-jet.

The symmetric tensors corresponding to the above scalar polynomials are given by

P ( 2 , t ) = 1 , P"2,t) = x z ,

P2j (2, t ) = -xzx3 + 9%' t , 1 2

6 P"k(2, t ) = L t x J x k + g(ZJ x k ) t !

(i3 k 1 ) t Jr-g x x 2 (ij k l ) t 2 + p 9 .

These fundamental polynomials, together with all higher order ones, constitute a (non-orthogonal) basis for the local decomposition of any scale-space.

3.2. Scale-Space Germs

By means of suitable contractions, all kinds of so- called germs can be constructed from the fundamental tensors of Corollary 1. Germs constrained

68 Florack, et al.

by diffusion play an important role in local Morse theory for the diffusion equation (Damon, 1990), (Koenderink, 1986), (Lindeberg, 1992b). Examples are the “H-versa1 germs” f ( Z ) and g ( 2 , t ) given by

d d 1 2

f(2) = - uiz! with ai = 0 , (14) i=l a=l

and

- k l

To see how these relate to the second order fundamental polynomial Pij(Z, t ) , take an arbitrary, symmetric cotensor Lij of rank 2 , and decompose it into its traceless part and the remainder, say

with

Note that this decomposition is covariant. Then the germ f(2) (14) is obtained by contraction of Pij(Z, t ) and 2,:

f(Z) = ZijPij(?,t), (19)

(equivalently, one can decompose Pij (Z, t ) into a traceless tensor and one proportional to g i j and then contract Lij onto the former). The germ g ( Z , t ) in (15) emerges, up to a scaling factor, by contracting Lij onto Rij:

X g ( Z , t ) = RijPij(Z,t) , (20)

in which the scalar X is given by

This decomposition makes the complementarity of the germs f(2) and g ( Z , t ) obvious; the local be-

haviour of second order image structure can always be explained in terms of these two germs.

Observe that the t-dependence drops out of (I!)), as a consequence of g z j Z z j = 0. To see that (19) is really nothing but (14), take any local Cartesian frame and diagonalise L,j by a suitable rotation of the frame (this construction of coordinates is usually called a gauge condition), say L,j c-) diag(X1, . . . , A d ) , upon which the scalar (1 9) reduces to

* i=l

with the ai given by

-- j=1

These numbers are indeed subject to the constramt in (14), but are otherwise arbitrary, thus capturing d - 1 structural degrees of freedom of the Hessian. It can be shown that the eigenvalues Xk: (k = 1,. . . , d) can be expressed in terms of traces over powers of the Hessian Lij up to order d, inclusive. Note in PiX- ticular that the residual degree of freedom that has dropped out of (23) is just (21):

(24)

The true scale-dependent part is apparently confined to the “blob-like” germ g ( 2 , t ) ; indeed, the rate of change of inner scale is completely determined by a single degree of freedom of the Hessian L,, viz. the Laplacean of the image. The germ g ( 2 , t ) is rhe canonical form describing the scale evolution at the umbilical points of the image’s second order structure (that is at the points where the quadratic sur-

(2) face L (Z,t = 0) is equally curved in all spatial directions), whereas the t-independent germ f (3) describes the stationary behaviour of the image’s second order structure at the well-known Laplacean zero- crossings (in d = 2 , these correspond to the anti- umbilical points). The latter germ can be decom- posed into a linear combination of d - 1 independent canonical forms that describe the stationary behaviour at the various prototypical “hypersaddles”.


Fig. 1. From left to right: a second order light blob, its complementary saddle, a third order blob modulated by a linear ramp, and its complementary “monkey saddle”. Second and third order local image structure can always be explained in terms of linear combinations of such pairs.

Figure 1 shows a typical blob and saddle in 2D, corresponding to the decomposition of Lij into Rij = Lij - Zij and Zij, respectively.

As a less trivial example, consider the complementary cotensors

Again we have (cf. (16))

Contraction of Zijk onto pi jk (Z , t ) yields the monomial

or

which is t-independent and is indeed a stationary solution of the diffusion equation. Its complement is given by the t-dependent monomial (a “blob” modulated by a linear ramp)

Note that $(Z,t) contains only a single degree of freedom, say wi = &gjkLijk; all other degrees of freedom contribute to the scale-independent $(?).

For the sake of simplicity, consider the 2- dimensional case. From differential geometry it is well-known that a cubic binary form f ( x , y ) = aoy3 + alxy2 + a2x2y + a3z3 represents a Monge patch parametrisation of a monkey saddle, if and only if the corresponding cubic obtained by setting (z,y) = (1,<) in the binary form has three distinct roots (Lipschutz, 1969). That this is indeed the case in (28) for arbitrary coefficients Lijk (degenera- cies apart) follows from algebra (Salden et al., 1994): the cubic j (<) = aoC3 + a1C2 + a2< + a3 has three distinct roots if and only if its discriminant’

is positive. The reader may verify that the discriminant corresponding to (28) in the 2-dimensional case is given in covariant form by

D = U ~ U E + 1 8 ~ 0 ~ 1 ~ 2 ~ 3 - 4~0~: - 4 ~ ! ~ 3 - 2 7 ~ : ~ ;

2 (g i lg j rngkn (4LijkLlmn - 3LijlLkmn)) D =

3072 ,

(32)

which is indeed nonnegative definite. In fact, D de- generates if and only if zijk, and hence $(.’), van- ishes identically, i.e. if and only if there exists a cov-

in which case $(.‘,t) =wixi(igjkzjxk + ( d f 2 ) t ) . This shows that, generically, (28) indeed parametrises a monkey saddle. Like with the second order saddle, a monkey saddle can neither be called a light nor a dark blob, and is indifferent with respect to blurring, while the “third order blob” (30) is a genuine

&Or W i such that Lijk = wigjk + wjgik f wkgi j ,

70 Florack, et al.

blob modulated by a linear ramp, which is sensitive to blurring. However, owing to this linear ramp mod- ulation a third order blob cannot be classified as light or dark either: changing the sign of the underlying blob essentially amounts to reversing the gradient direction of the modulating ramp, w, H -wa; the result is a n-rotated copy of the original picture. Third order local image structure can always be explained as a sum of a monkey saddle and a blob-on-a-ramp. Fig- ure 1 shows a typical monkey saddle and “third order blob” in 2D, corresponding to the decomposition of L a j k into Z,gk and R P J k = Lagk - Z a j k , respectively.

One may appreciate the advantage of representing germs in a manifest covariant form, such as (19), (20), (28), and (30) instead of the conventional, usually coordinate dependent representation, such as (14). One can always gauge down to canonical forms, such as (14), but in doing so things may get obscured by the choice of coordinates (in particular, the form of (14) depends on the coordinate system, so it is not at all clear whether it describes anything meaning- ful!). Using the covariant formalism, all seemingly different coordinate representations of one and the same germ are captured by a single, manifest invariant expression.

3.3. Debbrring Gaussian Blur

A special instance of (10) is deblurring, which is obtained by taking SZ = 0’ and S t = -&az for 0 < E 5 T . According to Proposition 1 we get, setting L ~ ( E ) = ~ ~ ( 6 2 = 6, fit = -&a2) for J = [ ~ / 2 ] :

(33)

with again L ( J ) = yklll . . . y k J I J ~ k l l l . . . k , ~ , evaluated at the origin. The tilde denotes spatial differentiation w.r.t. 5?, defined by the natural scaling xa = a?. The first order part of this scheme is a well-known deblurring technique, widely applied even long before scale-space theory had been established. In a scale-space, this scheme becomes operationally fea- sible up to some order N (but note that there is no a priori limitation for N). For a survey and a list of references, see e.g. (Wang et al., 1983). Some more

references, as well as a different deblurring strategy can be found in (Kimia and Zucker, 1993).

In order to say something about the truncation error (or equivalently, about the ghost image) at the point of interest (62 = O , S t = --&a2), we may use Result 1. However, since the spatial location is kept fixed, it is more convenient to take one step back and consider Lemma 2 with h ( S Z 7 S t ; A) replaced by G(6t;p) = L(6Z = 8,pSt) (p E I ) . This yields the conventional formula of Lagrange:

4

for some ji E I . Put differently, writing .@(E) instead of RJ(-&a2):

with Mj defined as the maximum value of the j-th scale derivative of the image at Z = 0 on the deblur interval S f E [ -E , 01 (a26t = S t ) :

ajo p E ~ a p Mj = niax-(St;p) = L(j)(O‘, -fi&a2). (36)

Note that deblurring is not the inverse of blurring (which is a semigroup operation that has no inverse), but rather an “image enhancement” operation; it does not create high resolution details that do not already exist at the current level of scale, but sharpens exist- ing details (including “noise” of course). From (33) this is evident from the fact that one can only manip- ulate a single (global) parameter E and, to a limited extent, the order N . One cannot expect to reconstruct the image at a higher resolution from this, since this requires an infinite number of local degrees of fr1:e- dom (by convergence of the series expansion (9), this would have been possible if the limit N --+ cm were physically realisable).

Examples of deblurring using (33) are presented in Figures 2 and 3. Note that despite the signifi- cant amount of noise used in the examples, it is czp- parently the case that the use of high order derivatives (up to order 12 in this case) may be bene- Jicial rather than prohibitive. An example of deblurring up to even higher orders is described in (Ter Haar Romeny et al., 1994). Note also that the global deblur parameter S t = -&a2 scales proportion- ally to a‘; in this example we have taken E slightly

The Gaussian Scale-Space Paradigm and the Multiscale Local Jet 7 1

below one half2 ( E = 0.469 at scale (T = 7.39, hence S t = -25.6). Although E is essentially a free parameter, it is clear that, given a certain order N , one should take it neither too large ( E >> 1/2), nor too small ( E << 1/2). In the first case one may expect the extrapolation to become unreliable, in the latter case the &-corrections become negligible (albeit very accurate) compared to zeroth order. The value E = 1/2 corresponds to the case s + S t = 0, i.e. to a (hypothetical) scale excursion to “infinite resolution” (recall that s = 02/2).

4. Conclusion and Discussion

We have introduced the concept of a multiscale local jet by means of a finite parametrisation of local image structure in the spatial as well as the scale domain. It is characterised by an order of contact N , and captures an equivalence class of local input signals that induce identical measurements (“metamerical images”) when correlated against Gaussian derivatives (scaled to a given scale and centred at a given location) of orders 0, . . . N .

The multiscale local jet can be represented by a finite hierarchy of fundamental polynomials with exact scale-space properties. These polynomials identically satisfy the diffusion equation underlying scale-space. By virtue of this property, and because of their explicit form, they are useful in the study of the local structure of scale-space: one can use them to make local approximations of scale-space within the “scale- space manifold” C, i.e. within a neighbourhood of a given set of dependent and independent variables (27 t ; L ) E C defined by the diffusion equation. In particular, they are convenient for the study of the generic behaviour of local scale-space features, and for the classification of bifurcations. Moreover, the fundamental polynomials constitute a smooth (non- orthogonal) basis for a local scale-space expansion. This basis embodies the local scale-space paradigm, and is independent of the input signal. Measurements extracted from the input signal by means of correlation with Gaussian derivative profiles correspond to the coefficients in a local expansion with respect to this basis. In this sense the fundamental polynomials are the duals of the local measurements: only as a pair do these convey a smooth, local representation of scale-space.

By means of an example it was shown that the lowest order local scale-space representation that is nontrivial with respect to space as well as scale requires a complete set of local measurements up to second order, inclusive (6 degrees of freedom for a two- dimensional image, 1 of which can locally be gauged away by rotation). The second lowest order refine- ment requires all local measurements up to fourth order (15 - 1 essential components in two dimensions), etc. Note that, with contemporary imaging techniques, these measurements usually have to be extracted a posteriori, that is, after the image acquisition and reconstruction stage. In biological visual systems, receptive fields of various sizes and weight profiles are probed to produce these measurements directly; individual rod and cone outputs are disre- garded already at a sensory stage.

Local image extrapolation in space and scale (with subpixel accuracy) is a straightforward application, comprising Gaussian “deblurring” as a special case. Deblurring according to such an extrapolation scheme is a remarkably simple, linear operation, which requires spatial derivatives of even orders 2N of the form AJL , J = 0 , . . . , N . It was shown empiri- cally that even for noisy images there is no a priori limitation to low orders, provided the details of interest have a sufficiently large intrinsic scale relative to those that are irrelevant (“noise”, grid). In the examples shown in this paper, the use of spatial derivatives of orders as high as 1 2 turned out to improve perfor- mance, even in images that are heavily corrupted by small-scale Gaussian additive noise (0 dB signal to noise ratio). Note that an extrapolation scheme (such as deblun-ing) can be combined with a resampling scheme, so as to maintain a fixed relative spatial den- sity of kernels at each scale (scale invariance). This requires the general extrapolation formula to be used (e.g., for appropriate spatial upsampling in the case of deblurring).

It was shown that there exists a local trade-off between spatial and scale extrapolation, which may lie at the basis of a scale-space sampling strategy. In particular, this trade-off seems to admit a coarse scale sampling at the expense of a fairly large spatial overlap of Gaussian filters. It may also provide a theoret- ical basis for physiological indications of overlapping receptive fields in mammalian visual cortex. A precise sampling strategy, however, can only be reached once several related problems have been dealt with.

72 Florack, et al.

ICV 3DCV 3DCV 3D

ICV 3DCV 3DCV 3D 3DCV 3DCV 3C

Fig. 2. Left image: A synthetic, binary test image (intensity difference 255 units) of 512 x 512 pixels. Second image: same as first one, but blurred to scale (T = 7.39 pixels. Third image: same as first one, but now with additive, pixel-uncorrelated, Gaussian noise with a standard deviation of 255 intensity units. Right image: same as third one, but blurred to scale (T = 7.39 pixels.

Fig. 3. Debluning as calculated for the low-resolution images of Figure 2 (second and fourth image respectively) of orders 1 to 6 (inclusive), using E = 0.469. The derivatives involved in the scale expansion have been evaluated at high resolution, scale U = 1.00 pixels, on the low-resolution input images (n = 7.39 pixels). Displayed from left to right, top to bottom, the first six images show the results of deblurring on the noise-free image: L. '=~(E) , , . . , L J = 6 ( ~ ) , while the last six images show the corresponding results for the noise-perturbed image.

F~~ example, we did not address the effect of internal noise inherent to the ~~~~~i~~ filtering process (random fluctuations, quantisation errors, etc.), nor did we consider multilocal issues, matters of physical limits and economy of hardware. We only considered the external errors that follow from a truncation of

the local scale-space expansion. Note that the internal errors (owing to quantisation and other sources of noise) of the filters used to obtain the measurements do not manifest themselves locally, that is, without the ability to compare neighbouring measurements. Because of their realisation through integration, the


measurements by themselves do not contain any information about the filters (which is just why the local scale-space paradigm had to be embodied in the form of the fundamental polynomials in the first place).

While the external errors decrease, the internal errors increase as a function of order (not treated in this paper, see e.g. (Blom, 1993)). The apparent trade- off remains to be investigated. Finding this trade-off would solve the notorious problem of determining the order of differentiation that optimises the representation of local image structure. The local ghost images defined by the optimal order then correspond to fundamentally inaccessible local degrees of freedom, quite similar to the ghost images one encounters in image reconstruction, corresponding to (near) zero eigenvalues of the acquisition matrix (Barrett, 1992).

By construction, it was shown that a contraction of measurements and corresponding fundamental polynomials always yields a smooth, local, internal representation, regardless of the filter characteristics (and hence, regardless of internal errors). It remains an intriguing problem how one should ‘‘glue” together all local expansions obtained at various positions and scales. It is also not clear whether one can actually endow the scale-space domain with a natural metric, i.e. one compatible with a connection. In any case it is clear that filter characteristics do become crucial when neighbouring measurements are considered si- multaneously. The scale-space paradigm prescribes that the ideal filter profiles should correspond to linear derivatives of a scaled Caussian. The closer a realisation of correlating filters matches the ideal design, the better a bilocal connection can be expected to fit. This connection problem is clearly related to the issue of scale-space sampling. It is also of physiological relevance to the problem of local sign determination, i.e. the determination of the spatial context of a set of unlabelled local measurements (this problem should not be confused with the somatotopic nature of neural mappings). Metaphorically speaking, each local measurement can be regarded as a piece of a jig-saw puzzle, and only by virtue of an internal representation of the scale-space paradigm, together with a signifi- cant filter overlap factor, one may hope to be able to reconstruct such a puzzle (Koenderink, 19844, (Koenderink, 1990), (Lotze, 1884).

Appendix

A.l. Proof of Result 1

Result 1 follows from

LEMMA 3 Definitions as for Result 1.

1 d”A n! dAn

(62, S t ; A) = --

1 1 k q=1 k=n-q

Before we turn to the proof of this, consider the following scale-space index conventions.

Definition. For each spatial index i = 1, . . . , d we define the scale-space index QI = 0, . . . , d, such that the values a = 0 and a = 1, . . . , d refer to the scale ( t ) and spatial (xi) dimensions, respectively. We will sometimes write a as &(i, 0). For each QI = 0 , . . . ,d and 1 = 0,1,2, . . ., let @(A) be given by the l-th order derivative of @(A) = (ASzi ,A2St) . In other words, @(A) = (hi, 2ASt) , $?(A) = (0,2St), and #(A) = (0,O) for all 1 2 3. Finally, let Lal...ol,, denote the q-th order derivative of L w.r.t. z a l , . . . ,za”, with z‘lZ(zi, t ) . We will also write LI:.)..is for a k-fold derivative w.r.t. t and q-fold derivative w.r.t. 9 1 , . . . , zzci.

Proof of Result 1: Note that A(65, 6 t ; A) = L($,”(X)), so differentiation w.r.t. A requires application of the chain rule and, since this will yield products of A-dependent factors, of Leibnitz’s product rule for differentiation. By induction one easily verifies that

in which the Einstein convention is in effect for the a-type indices. The * on top of the summation sym- bols indicates that the 1-type indices are subject to a common constraint, viz.

74 Florack, et al.

4

* : Clj==rl, j=l

that is, the total number of derivations equals n. Now we make the scale and spatial summations

explicit: suppose that k of the a-type indices are zero, say 01, . . . , a k = 0 (after suitable rearrangement), so that the remaining q - k indices must refer to space, i.e. a k + 1 = i k + l , ... ,aq = 2,. In order to account for all terms in the summation we must take into account a combinatorial factor (3 and sum over all

possible values of k = 0, . . . , q. This leads to

* * k .

The q - k summations over l k + l , . . . $ contribute by only one effective term each, since @(A) is nonzero only if 1 = 1, for which it equals Sz2. Evaluating the corresponding sums yields

* * * *

k ,

subject to the constraint

k

j=1

Both components 4y(X) and @(A) contain a factor S t , so we may extract this and rewrite the expression into

in which X(X; k, n - q) stands for

X ( X ; k, n - q ) =

with @(A) defined by $?(A) = 2X and &(A) = 2 .

Finally, we must evaluate X ( X ; k , n - q). Again we may assume that the first j indices 11, . . . , l j are equal to 1, and the remaining indices l j + l , . . . 11, are equal to 2, provided we sum over all values of j = 0 , . . . , k with the appropriate combinato-

rial factor (t), and provided we respect the con-

straint **. For a given j this constraint becomes j .1+ (k - j ) . 2 = n - q + k, from which it follows that we only get an effective term if j = k - n + q (and since j 2 0 we may as well start the k-sum at k = n - q). Thus we find

and substitution finally yields

1 d" n! dXn L ( G ( N ) = --

k

q = l k=n-q

( 2 X ) k - " + q L g ; l , , , i s (+;(X))6zik+l. . .6zir'6tk

This completes the proof of Lemma 3 and thus of Result 1 (after some trivial index manipulations).

Acknowledgements

This work has been carried out as part of the na- tional priority research programme "3D Computer Vision", supported by the Netherlands Ministries of Economic Affairs and Education & Science through a SPIN grant, and by several industrial participants. The final revision of the manuscript has been pre- pared partially at INRIA Sophia-Antipolis, France, and partially at INESC Aveiro, Portugal, as part of the ERCIM Fellowship Programme, financed by the Commission of the European Communities. Janita Wilting is gratefully acknowledged for her critical comments and suggestions.


Notes

1. The classical algebraic literature sometimes uses a bracket formalism based on so-called “Wansvectants”, which makes the coordinate invariance of magic combinations of coefficients like these transparent.

2. This was done by visual inspection after a quick trial-and-error by taking a starting value 6t = -0.1 and doubling this until a visually reasonable result was obtained. This explains the somewhat odd value of E = 0.469; no attempt has been made to rigorously optiniise the choice of E . The qualitative results turn out to be rather insensitive to the precise value of E % 3.

References

Babaud, J., Witkin, A. P., Baudin. M. and Duda, R. 0. 1986. Uniqueness of the gaussian kernel for scale-space filtering. IEEE Truns. Puttern Anulysis und Muchine Intelligence, 8( 1):26-33.

Image reconstruction and the solution of inverse problems in medical imaging. In Todd-Pokropck, A. E. and Viergever, M. A., editors, Medicul Images: Formution, Hun- dling and Evuluution, NATO AS1 Series F98, Springer Verlag, Berlin, pp. 3-42.

Blom, J., ter Haar Romeny, B. M., Bel, A. and Koenderink, J. I. 1993. Spatial derivatives and the propagation of noise in gaussian scale-space. J . of’ Vis. Comm. und lm. Repr: , 4( I): I - 13.

Local Morse theory for solutions to the heat equation and Gaussian blurring. Internal report, Department of Mathematics, University of North Carolina, Chapel Hill, North Carolina, USA.

Florack, L. M. J., ter Haar Romeny, B. M., Koenderink, J. J. and Viergever, M. A. 1992. Scale and the differential structure of images. Image und Vision Computing, 10(6):376-388.

Florack, L. M. J., ter Haar Romeny, B. M., Koenderink, J. J. and Viergever, M. A. 1993. Cartesian differential invariants in scale-space. Journal of Muthemuticul Imuging und Vision.

Florack, L. M. J., ter Haar Romeny, B. M., Koenderink, J. J . and Viergever, M. A. 1994. Linear scale-space. Journal of Muthe- muticul lmuging und Vision, 4(4):325-35 1.

Hummel, R. A., Kiniia, B. B. and Zucker, S. W. 1987. Deblurring gaussian blur. Coinputer Vision. Gruphics, and Imuge Process- ing. 38:66-80.

Johansen, P., Skelboe, S., Grue, K. and Andersen, J. D. 19x6. Rep- resenting signals by their top points in scale-space. In Proceed- ings of’the 8-th Internutionul Conference on Puttern Recognition, Paris, pp. 215-217.

Kay, D. C. 1988. Tensor Culculus. Schaum’s Outline Series. McGraw-Hill Book Company, New York.

Kimia, B. B. and 2ucker.S. W. 1993. Analytic inverse of’ discrete Gaussian blur. Opticul Engineering, 32( 1 ): 166-176.

Koenderink. J. J. 1984a. The concept of local sign. In van Doorn, A. J . , van de Grind, W. A. and Koenderink, J. J, editors, Limits in Perception, VNU Science Press, Utrecht, pp. 495-547.

Barrett, H. H. 1992.

Damon, J. 1990.

3(4):327-348.

Koenderink, J. J. 1984b. The structure of images. Biol. Cybern.,

Koenderink, J. J. 1992. Local image structure. In Johansen, P., and Olsen, S., editors, Theory & Applications of’ Image Anul- ysis, volume 2 of Series in Machine Perception and Artificial Intelligence, World Scientific, Singapore, pp. 15-21.

Koenderink, J. J. and van Doorn, A. J. 1986. Dynamic shape. Biol. Cybern., 53:383-396.

Koenderink, J. J. and van Doorn, A. J. 1987. Representation of local geometry in the visual system. Biol. Cybern., 55:367-375.

Koenderink, J. J. and van Doorn, A. J. 1990. Receptive field families. Biol. Cybern., 63:291-298.

Koenderink, J. J. 1990. The brain a geometry engine. Psycholog- icul Research, 52:122-127.

Lawden, D. F. 1962. An Introduction to Tensor Calculus und Relativity. Spottiswoode Ballantyne & CO Ltd.

Lindeberg, T. 1990. Scale-space for discrete signals. IEEE Truns. Puttern Analysis und Muchine Intelligence, 12(3):234- 245.

Lindeberg, T. and Eklundh, J. 0. 1990. On the computation of a scale-space primal sketch. J . of Ws. Comm. und Itn. R e p ,

Lindeberg, T. 1992a. On the behaviour in scale-space of local extrema and blobs. In Johansen, P., and Olsen, S., editors, The- ory & Applications of Imuge Analysis, volume 2 of Series in Muchine Perception und Artificial Intelligence, World Scientific, Singapore, pp. 38-47.

Lindeberg, T. 1992b. Scale-space behaviour of local extrema and blobs. Journal qf Muthematicul Imuging and Vision, I (1):65-99.

Lipschutz, M. M. 1969. Diflerentiul Geometry. Schaum’s Outline Series. McGraw Hill, New York.

Lotw, H. 1884. Mikrokosmos. Hirzel, Leipzig. Salden, A. H., ter Haar Romeny, B. M. and Viergever, M. A. 1994.

Local and multilocal scale-space description. In 0, Y.-L., Toet, A., Heijmans, H. J. A. M., Foster, D. H. and Meer, P., editors, Proc. of the NATO Advanced Reseurch Workshop Shupe in Pic- ture - Muthemuticul Description of Shape in Greylevel Images, NATO AS1 Series F126. Springer Verlag, Berlin.

Schwartz, L. 1950-1951. Thiorie des Distributions, volume I, I1 of Actuulitis scientijiyues et industrielles; 1091,1122. Publica- tions de I’lnstitut de MathCmatique de I’ UniversitC de Strasbourg, Paris.

Spivak, M. 1970-1975. A Comprehensive Introduction to Difer- entiul Geometry, volume I-V. Publish or Perish, Inc., Houston, Texas.

Ter Haar Romeny, B. M., Florack, L. M. J., de Swart, M., Wilt- ing, J. J. and Viergever, M. A. 1994. Deblurring Gaussian blur. In Proc. Mathematical Methods in Medicul Imuging 11, volume 2299, San Diego, CA, pp. 25-26.

Digital image enhancement: A survey. Computer Vision, Graphics, und Imuge Processing, 24:363-381.

Witkin, A. P. 1983. Scale space filtering. In Proc. Internationul Joint Conjerence on Artificial Intelligence, Karlsruhe, W. Ger- many, pp. 1019-1023.

501363-370.

2( 1):55-78.

Wang, D. C. C., Vagnucci, A. H. and Li, C. C. 1983.

The Gaussian Scale-Space Paradigm and the Multiscale Local Jetmate.tue.nl/mate/pdfs/5162.pdf · 1)-dimensional scale-space neighbourhood. 2. Theory In order to appreciate the notion

Documents