TOPOICA_1-s2.0-S1566253505000801-main

www.elsevier.com/locate/inffus

Information Fusion 8 (2007) 131–142

Pixel-based and region-based image fusion schemes using ICA bases

Nikolaos Mitianoudis *, Tania Stathaki

Communications and Signal Processing Group, Imperial College London, Exhibition Road, SW7 2AZ London, UK

Received 14 January 2005; received in revised form 1 September 2005; accepted 1 September 2005Available online 17 October 2005

Abstract

The task of enhancing the perception of a scene by combining information captured by different sensors is usually known as image

fusion. The pyramid decomposition and the Dual-Tree Wavelet Transform have been thoroughly applied in image fusion as analysis andsynthesis tools. Using a number of pixel-based and region-based fusion rules, one can combine the important features of the input imagesin the transform domain to compose an enhanced image. In this paper, the authors test the efficiency of a transform constructed usingIndependent Component Analysis (ICA) and Topographic Independent Component Analysis bases in image fusion. The bases are obtainedby offline training with images of similar context to the observed scene. The images are fused in the transform domain using novel pixel-based or region-based rules. The proposed schemes feature improved performance compared to traditional wavelet approaches withslightly increased computational complexity.� 2005 Elsevier B.V. All rights reserved.

Keywords: Image fusion; Image segmentation; Independent component analysis; Topographic ICA

1. Introduction

Let I1(x, y), I2(x, y), . . . , IT(x, y) represent T images ofsize M1 · M2 capturing the same scene. Each image hasbeen acquired using different instrument modalities or cap-ture techniques, allowing each image to have differentcharacteristics, such as degradation, thermal and visualcharacteristics.

In this scenario, we usually employ multiple sensors thatare placed relatively close and are observing the samescene. The images acquired by these sensors, although theyshould be similar, are bound to have some translationalmotion, i.e. miscorrespondence between several pointsof the observed scene. Image registration is the processof establishing point-by-point correspondence between anumber of images, describing the same scene. In this study,we will assume that the input images Ii(x, y) have negligibleregistration problems, which implies that the objects in allimages are geometrically aligned [5].

1566-2535/$ - see front matter � 2005 Elsevier B.V. All rights reserved.

doi:10.1016/j.inffus.2005.09.001

* Corresponding author. Tel.: +44 207 594 6199; fax: +44 207 594 6234.E-mail address: [email protected] (N. Mitianoudis).

The process of combining the important features fromthese T images to form a single enhanced image If(x, y) isusually referred to as image fusion. Fusion techniques canbe divided into spatial domain and transform domain tech-niques [6]. In spatial domain techniques, the input imagesare fused in the spatial domain, i.e. using localised spatialfeatures. Assuming that g(Æ) represents the ‘‘fusion rule’’,i.e. the method that combines features from the inputimages, the spatial domain techniques can be summarised,as follows:

If ðx; yÞ ¼ gðI1ðx; yÞ; . . . ; IT ðx; yÞÞ ð1ÞThe main motivation behind moving to a transformdomain is to work in a framework, where the image�s sali-ent features are more clearly depicted than in the spatialdomain. Hence, the choice of the transform is very impor-tant. Let Tf�g represent a transform operator and g(Æ) theapplied fusion rule. Transform-domain fusion techniquescan then be outlined, as follows:

If ðx; yÞ ¼T�1fgðTfI1ðx; yÞg; . . . ;TfIT ðx; yÞgÞg ð2ÞThe fusion operator g(Æ) describes the merging of informa-tion from the different input images. Many fusion rules

132 N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142

have been proposed in literature [14–16]. These rules can becategorised, as follows:

• Pixel-based rules: the information fusion is performed ina pixel-by-pixel basis either in the transform or spatialdomain. Each pixel (x, y) of the T input images is com-bined with various rules to form the corresponding pixel(x, y) in the ‘‘fused’’ image IT. Several basic transform-domain schemes were proposed [14], such as:– fusion by averaging: fuse by averaging the corre-

sponding coefficients in each image (‘‘mean’’ rule)

TfIf ðx; yÞg ¼1

T

XT

i¼1

TfI iðx; yÞg ð3Þ

– fusion by absolute maximum: fuse by selecting thegreatest in absolute value of the corresponding coef-ficients in each image (‘‘max-abs’’ rule)

TfIf ðx; yÞg ¼ sgnðTfI iðx; yÞgÞmaxijTfI iðx; yÞgj ð4Þ

– fusion by denoising (hard/soft thresholding): performsimultaneous fusion and denoising by thresholdingthe transform�s coefficients (sparse code shrinkage[10]).

– high/low fusion, i.e. combining the ‘‘high-frequency’’parts of some images with the ‘‘low-frequency’’ partsof some other images.The different properties of these fusion schemes willbe explained later on. For a more complete reviewon pixel-based methods, one can always refer to Piella[15], Nikolov et al. [14] and Rockinger et al. [16].

• Region-based fusion rules: these schemes group imagepixels to form contiguous regions, e.g. objects andimpose different fusion rules to each image region. In[13], Li et al. created a binary decision map to choosebetween the coefficients using a majority filter, measur-ing activity in small patches around each pixel. In [15],Piella proposed several activity level measures, such asthe absolute value, the median or the contrast to neigh-bours. Consequently, she proposed a region-basedscheme using a local correlation measurement to per-forms fusion of each region. In [12], Lewis et al.produced a joint-segmentation map out of the inputimages. To perform fusion, they measured priority usingenergy, variance, or entropy of the wavelet coefficients toimpose weighting on each region in the fusion processalong with other heuristic rules.

In this paper, the authors examine the application ofIndependent Component Analysis (ICA) and Topographic

Independent Component Analysis bases as an analysis toolfor image fusion in both noisy and noiseless environments.The performance of the proposed transform in image fu-sion is compared to traditional fusion analysis tools, suchas the wavelet transform. Common pixel-based fusion rulesare tested together with a proposed ‘‘weighted combina-tion’’ scheme, based on the L1-norm. Finally, a region-

based approach that segments and fuses active and non-ac-tive areas of the image is introduced.

The paper is structured, as follows. In Section 2, weintroduce the basics of the Independent Component Anal-ysis technique and how it can be used to generate analysis/synthesis bases for image fusion. In Section 3, we describethe general method for performing image fusion using ICAbases. In Section 4, we present the proposed pixel-basedweighted combination scheme and a combinatory region-based scheme. In Section 5, we benchmark the proposedtransform and fusion schemes, using common fusion test-bed. Finally, in Section 6, we outline the advantages anddisadvantages of the proposed schemes together with somesuggestions about future work.

2. ICA and topographic ICA bases

Assume an image I(x, y) of size M1 · M2 and a windowW of size N · N, centered around the pixel (x0, y0). An‘‘image patch’’ is defined as the product between a N · N

neighbourhood centered around pixel (x0, y0) and the win-dow W

Iwðk; lÞ ¼ W ðk; lÞIðx0 � bN=2c þ k; y0 � bN=2c þ lÞ;8k; l 2 ½0;N � 1� ð5Þ

where bÆc represents the lower integer part and N is odd.For the subsequent analysis, we will assume a rectangularwindow, i.e.

W ðk; lÞ ¼ 1; 8k; l 2 ½0;N � 1� ð6Þ

2.1. Definition of bases

In order to uncover the underlying structure of animage, it is common practice in image analysis to expressan image as the synthesis of several other basis images.These bases are chosen according to the image propertieswe aim to highlight with this analysis. A number of baseshave been proposed in literature so far, such as cosine bases,complex cosine bases, Hadamard bases and wavelet bases.In this case, the bases are well defined in order to servesome specific analysis tasks. However, one can estimatearbitrary bases by training with a population of similarcontent images. The bases are estimated after optimisinga cost function that defines the bases� desired properties.

The N · N image patch Iw(k, l) can be expressed as a lin-ear combination of a set of K basis images bj(k, l), i.e.

Iwðk; lÞ ¼XK

j¼1

ujbjðk; lÞ ð7Þ

where uj are scalar constants. The two-dimensional (2D)representation can be simplified to an one-dimensional(1D) representation, by employing lexicographic ordering,in order to facilitate the analysis. In other words, the imagepatch Iw(k, l) is arranged into a vector Iw, taking all ele-ments from matrix Iw in a row-wise fashion. Assume that

N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142 133

we have a population of patches Iw, acquired randomlyfrom the original image I(x, y). These image patches canthen be expressed in lexicographic ordering, as follows:

IwðtÞ ¼XK

j¼1

ujðtÞbj ¼ ½ b1 b2 � � � bK �

u1ðtÞu2ðtÞ. . .

uKðtÞ

26664

37775 ð8Þ

where t represents the tth image patch selected from theoriginal image. The whole procedure of image patch selec-tion and lexicographic ordering is depicted in Fig. 1.Let B ¼ ½ b1 b2 � � � bK � and uðtÞ ¼ ½ u1ðtÞ u2ðtÞ � � �uKðtÞ�T. Then, Eq. (8) can be simplified, as follows:

IwðtÞ ¼ BuðtÞ ð9ÞuðtÞ ¼ B�1IwðtÞ ¼ AIwðtÞ ð10Þ

In this case, A ¼ B�1 ¼ ½ a1 a2 � � � aK �T represents theanalysis kernel and B the synthesis kernel. This ‘‘trans-form’’ projects the observed signal Iw(t) on a set of basisvectors bj. The aim is to estimate a finite set of basis vectorsthat will be capable of capturing most of the signal�s struc-ture (energy). Essentially, we need N2 bases for a complete

representation of the N2-dimensional signals Iw(t). How-ever, with some energy compaction mechanism, we canhave efficient overcomplete representations of the originalsignals using K < N2 bases.

The estimation of these K vectors is performed using apopulation of training image patches Iw(t) and a criterion(cost function), which is going to be optimised in orderto select the basis vectors. In the next paragraphs, we willestimate bases from image patches using several criteria.

2.1.1. Principal component analysis (PCA) bases

One of the transform�s targets might be to analyse theimage patches into uncorrelated components. Principal

component analysis (PCA) can identify uncorrelated vectorbases [8], assuming a linear generative model, like the onein (9). In addition, PCA can be used for dimensionality

I(x0,y0)

Fig. 1. Selecting an image patch Iw around pix

reduction to identify the K most important basis vectors.This is performed by eigenvalue decomposition of the datacorrelation matrix C ¼ EfIwIT

wg. Assume that H is a matrixcontaining all the eigenvectors of C and D a diagonal ma-trix containing the eigenvalues of C. The eigenvalue at theith diagonal element should correspond to the eigenvectorat the ith column of H. Then, the rows of the following ma-trix V provide an orthonormal set of uncorrelated bases,which are called PCA bases

V ¼ D�0:5H T ð11ÞThe above set forms a complete set of bases, i.e. we have asmany bases as the dimensionality of the problem (N2). AsPCA has good energy compaction properties, one can forma reduced (overcomplete) set of bases, based on the originalones. The eigenvalues can illustrate the significance of theircorresponding eigenvector (basis vector). We can order theeigenvalues in the diagonal matrix D, in terms of decreas-ing absolute value. The eigenvector matrix H should be ar-ranged accordingly. Then, we can select the first K < N2

eigenvectors that correspond to the K most importanteigenvalues and form reduced versions of D and H . The re-duced K · N2 PCA matrix V is calculated using (11) for Dand H . The input data can be mapped to the PCA domainvia the transformation:

zðtÞ ¼ V IwðtÞ ð12ÞThe number of bases K of the overcomplete set is chosen sothat the computational load of a complete representationcan be reduced. However, the overcomplete set should beable to provide an almost lossless representation of the ori-ginal image. Therefore, the choice of K is usually a trade-offbetween computational complexity and image quality.

2.1.2. Independent component analysis (ICA) bases

A more strict criterion than uncorrelatedness is to as-sume that the basis vectors or equivalently the transformcoefficients are statistically independent. Independent Com-

ponent Analysis (ICA) can identify statistically independent

Iw(t)Image Patch

Lexicographicordering

Iw(k,l)

N

Iw(t)

el (x0, y0) and the lexicographic ordering.

134 N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142

basis vectors in a linear generative model [11]. A number ofdifferent approaches have been proposed to analyse thegenerative model in (9), assuming statistical independencebetween the coefficients ui in the transform domain. Statis-tical independence can be closely linked with the non-Gaussianity. The Central Limit Theorem states that thesum of several independent random variables tends to-wards a Gaussian distribution. The same principal holdsfor any linear combination Iw of these independent randomvariables ui. The Central Limit Theorem also implies that ifwe can find a combination of the observed signals in Iw

with minimal Gaussian properties, then that signal willbe one of the independent signals. Therefore, statisticalindependence and non-Gaussianity can be interchangeableterms.

We can briefly outline some of the different techniquesthat can be used to estimate independent coefficients ui.Some approaches estimate ui by minimising the Kullback–

Leibler (KL) divergence between the estimated coefficientsui and several probabilistic priors on the coefficients. Otherapproaches minimise the mutual information conveyed bythe estimated coefficients or perform approximate diago-nalisation of a cumulant tensor of Iw. Finally, some meth-ods estimate ui by estimating the directions of the mostnon-Gaussian components using kurtosis or negentropy,as non-Gaussianity measures. For more on these tech-niques, one can refer to tutorial books on ICA, such as[1,11].

In this study, we will use an approach that optimisesnegentropy, as a non-Gaussianity measurement to identifythe independent components ui. This is also known as Fas-tICA and was proposed by Hyvarinen and Oja [7]. In thistechnique, PCA is used as a preprocessing step to select theK most important vectors and orthonormalise the datausing (12). Consequently, the statistical independent com-ponents can be identified using orthogonal projectionsaT

i z. In order to estimate the projecting vectors ai, we haveto minimise the following non-quadratic approximation ofnegentropy:

J GðaiÞ ¼ EfGðaTi zÞg � EfGðvÞg

� �2 ð13Þ

where Ef�g denotes the expectation operator, v is a Gauss-ian variable of zero mean and unit variance and G(Æ) ispractically any non-quadratic function. A couple of possi-ble functions were proposed in [9]. In our analysis, we willuse

GðxÞ ¼ affiffiffiffiffiffiffiffiffiffixþ �p

þ b ð14Þ

where a, b are constants and � is a small constant (� � 0.1)to tackle numerical instability, in the case that x! 0.Hyvarinen and Oja produced a fixed-point method, opti-mising the above definition of negentropy, which is alsoknown as the FastICA algorithm

aþi Efai/ðaTi zÞg � Ef/0ðaT

i zÞgai; 1 6 i 6 K ð15ÞA AðATAÞ�0:5 ð16Þ

where /(x) = �oG(x)/ox. We randomly initialise the up-date rule in (15) for each projecting vector ai. The new up-dates are then orthogonalised, using the symmetric likeorthogonalisation scheme in (16). These two steps are iter-ated, until ai have converged.

2.1.3. Topographical independent component analysis

(TopoICA) bases

In practical applications, one can very often observeclear violations of the independence assumption. It is pos-sible to find couples of estimated components such thatthey are clearly dependent on each other. This dependencestructure, however, is very informative and it would be use-ful to somehow estimate it [9].

Hyvarinen et al. [9] used the residual dependency of the‘‘independent’’ components, i.e. dependencies that couldnot be cancelled by ICA, to define a topographic order be-tween the components. Therefore, they modified the origi-nal ICA model to include a topographic order between thecomponents, so that components that are near to eachother in the topographic representation are relativelystrongly dependent in the sense of higher-order correla-tions or mutual information. The proposed model is usu-ally known as the Topographic ICA model. Thetopography is introduced using a neighbourhood functionh(i, k), which expresses the proximity between the ith andthe kth component. A simple neighbourhood model canbe the following:

hði; kÞ ¼1; if ji� kj 6 L

0; otherwise

�ð17Þ

where L defines the width of the neighbourhood. Conse-quently, the estimated coefficients ui are no longer assumedindependent, but can be modelled by some generative ran-dom variables dk, fi that are controlled by the neighbour-hood function and shaped by a non-linearity /(Æ) (similarto the one in the FastICA algorithm). The topographicsource model, proposed by Hyvarinen et al. [9], is thefollowing:

ui ¼ /XK

k¼1

hði; kÞdk

!fi ð18Þ

Assuming a fixed-width neighbourhood L · L and that theinput data are preprocessed by PCA, Hyvarinen et al. per-formed Maximum Likelihood estimation of the synthesiskernel B using the linear model in (9) and the topographicsource model in (18), making several assumptions for thegenerative random variables dk and fi. Optimising anapproximation of the derived log-likelihood, they formedthe following gradient-based topographic ICA rule:

aþi ai þ gEfzðaTi zÞrig; 1 6 i 6 K ð19Þ

A AðATAÞ�0:5 ð20Þ

where g defines the learning rate of the gradient optimisa-tion scheme and

Fig. 2. Comparison between ICA and the topographical ICA basestrained on the same set of image patches. We can observe the localcorrelation of the bases induced by the ‘‘topography’’.

N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142 135

ri ¼XK

k¼1

hði; kÞ/XK

j¼1

hðj; kÞðaTi zÞ2

!ð21Þ

As previously, we randomly initialise the update rule in(19) for each projecting vector ai. The new updates are thenorthogonalised and the whole procedure is iterated, until ai

have converged. For more details on the definition and der-ivation of the topographic ICA model, one can always referto the original work by Hyvarinen et al. [9].

2.2. Training ICA bases

In this paragraph, we describe the training procedure ofthe ICA and topographic ICA bases more thoroughly. Wehave to stress that the training procedure needs to be com-pleted only once. After we have successfully trained the de-sired bases, the estimated transform can be used for fusionof similar content images.

We select a set of images with similar content to the onesthat will be used for image fusion. A number of N · Npatches (usually �10000) are randomly selected from thetraining images. We apply lexicographic ordering to the se-lected images patches. We perform PCA on the selectedpatches and select the K < N2 most important bases,according to the eigenvalues corresponding the bases. Itis always possible to keep the complete set of bases. Then,we iterate the ICA update rule in (15) or the topographicalICA rule in (19) for a chosen L · L neighbourhood untilconvergence. Each iteration, we orthogonalise the basesusing the scheme in (16).

Some examples from trained ICA and topographic ICAbases are depicted in Fig. 2. We randomly selected 1000016 · 16 patches from natural landscape images. UsingPCA, we selected the 160 most important bases out ofthe 256 bases available. In Fig. 2(a), we can see the ICAbases estimated using FastICA (15). In Fig. 2(b), we canthe set of the estimated topographic ICA bases using therule in (19) and assuming a 3 · 3 neighbourhood for thetopographic model.

2.3. Properties of the ICA bases

Let us explore some of the properties of the ICA and thetopographical ICA bases and the transforms they consti-tute. Both transforms are invertible, i.e. they guarantee per-fect reconstruction. Using the symmetric orthogonalisationstep A A(ATA)�0.5, the bases remain orthogonal, i.e. thetransform is orthogonal.

We can examine the estimated example set of ICA andtopographical ICA bases in Fig. 2. The ICA and topo-graphical ICA basis vectors seem to be closely related towavelets and Gabor functions, as they represent similarfeatures in different scales. However, these bases havemore degrees of freedom than wavelets [9]. The DiscreteWavelet transform has only two orientations and theDual-Tree wavelet transform can give six distinct sub-bands at each level with orientation ±15�, ±45�, ±75�.

The ICA bases can get arbitrary orientations to fit thetraining patches.

One basic drawback of these transforms is that they arenot shift invariant. This property is generally mentioned tobe very important for image fusion in literature [14]. Piella[15] states that the fusion result will depend on the locationor orientation of objects in the input sources in the case ofmisregistration problems or when used for image sequencefusion. As we assume that the observed images are all reg-istered, the lack of shift invariance should not necessarilybe a problem. In addition, Hyvarinen et al. proposed toapproximate shift invariance in these ICA schemes, byemploying a sliding window approach [10]. This implies thatthe input images are not divided into distinct patches, butinstead every possible N · N patch in the image is analysed.This is similar to the spin cycling method, proposed byCoifman and Donoho [2]. This will also increase the com-putational complexity of the proposed framework. Wehave to stress that the sliding window approach is only nec-essary for the fusion part and not for the estimation ofbases.

The basic difference between ICA and topographic ICAbases is the ‘‘topography’’, as introduced in the latter bases.The introduction of some local correlation in the ICAmodel enables the algorithm to uncover some connectionsbetween the independent components. In other words,

136 N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142

topographic bases provide an ordered representation of thedata, compared to the unordered representation of the ICAbases. In an image fusion framework, ‘‘topography’’ canidentify groups of features that can characterise certain ob-jects in the image. One can observe the ideas comparingFig. 2(a) and (b). Topographic ICA seems to offer a morecomprehensive representation compared to the generalICA model.

Another advantage of the ICA bases is that the esti-mated transform can be tailored to the needs of the appli-cation. Several image fusion applications work with specifictypes of images. For example, military applications workwith images of airplanes, tanks, ships etc. Biomedical appli-cations employ Computed Tomography (CT), PositronEmission Tomography (PET), ultra-sound scan imagesetc. Consequently, one can train bases for specific applica-tion areas. These bases should be able to analyse thetrained data types more efficiently than a generictransform.

3. Image fusion using ICA bases

In this section, we describe the whole procedure of per-forming image fusion using ICA or topographical ICAbases, which is summarised in Fig. 3. We assume that aICA or topographic ICA transform Tf�g is already esti-mated, as described in a previous section. Also, we assumethat we have T M1 · M2 registered sensor images Ik(x, y)that need to be fused. From each image we isolate everypossible N · N patch and using lexicographic ordering,we transform it to a vector Ik(t). The patches� size N shouldbe the same as the one used in the transform estimation.Therefore, each image Ik(x, y) is now represented by apopulation of (M1 � N)(M2 � N) vectors Ik(t), "t 2 [1,(M1 � N)(M2 � N)]. Each of these representations Ik(t) istransformed to the ICA or topographic ICA domain repre-sentation uk(t). Assuming that A is the estimated analysiskernel, we have

T{}

T{}

T{}

OptionalDenoising

EstimatedTransform

TransformedImages

InputImages

uk(t)

Fig. 3. The proposed fusion system us

ukðtÞ ¼TfIkðtÞg ¼ AIkðtÞ ð22ÞOnce the image representations are in the ICA domain, onecan apply a ‘‘hard’’ threshold on the coefficients and per-form optional denoising (sparse code shrinkage), as pro-posed by Hyvarinen et al. [10]. Then, one can performimage fusion in the ICA or topographic ICA domain inthe same manner that is performed in the wavelet ordual-tree wavelet domain. The corresponding coefficientsuk(t) from each image are combined in the ICA domainto construct a new image uf(t). The method g(Æ) that com-bines the coefficients in the ICA domain is called ‘‘fusionrule’’

uf ðtÞ ¼ g u1ðtÞ; . . . ; ukðtÞ; . . . ; uT ðtÞð Þ ð23ÞWe can use one of the many proposed rules for fusion, asthey were analysed in the introduction section and in liter-ature [15,14]. Therefore, the ‘‘max-abs’’ and the ‘‘mean’’rules can be two very common options. However, onecan use more efficient fusion rules, as we will see in the nextsection. Once the composite image uf(t) is constructed inthe ICA domain, we can move back to the spatial domain,using the synthesis kernel B, and synthesise the imageIf(x, y) by averaging the image patches If(t) in the same or-der they were selected during the analysis step. The wholeprocedure can be summarised as follows:

(1) Segment all input images Ik(x, y) into every possibleN · N image patch and transform them to vectorsIk(t) via lexicographic ordering.

(2) Move the input vectors to the ICA/topographic ICAdomain, and get the corresponding representationuk(t).

(3) Perform optional thresholding of uk(t) for denoising.(4) Fuse the corresponding coefficient using a fusion rule

and form the composite representation uf(t).(5) Move uf(t) to the spatial domain and reconstruct the

image If(x, y) by averaging the overlapping imagepatches.

Fusionrule

T -1{}

Fused Image

ing ICA/topographical ICA bases.

N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142 137

4. Pixel-based and region-based fusion rules using

ICA bases

In this section, we describe two proposed fusion rules forICA bases. The first one is an extension of the ‘‘max-abs’’pixel-based rule, which we will refer to as the Weight Com-bination (WC) rule. The second one is a combination of theWC and the ‘‘mean’’ rule in a region-based scenario.

4.1. A weight combination (WC) pixel-based method

An alternative to common fusion methods, is to use a‘‘weighted combination’’ of the transform coefficients, i.e.

TfIf ðtÞg ¼XT

k¼1

wkðtÞTfIkðtÞg ð24Þ

There are several parameters that can be employed in theestimation of the contribution wk(t) of each image to the‘‘fused’’ one. In [15], Piella proposed several activity

measures. Following the general ideas proposed in [15],we propose the following scheme. As we process each im-age in N · N patches, we can use the mean absolute value(L1-norm) of each patch (arranged in a vector) in thetransform domain, as an activity indicator in each patch.

EkðtÞ ¼ kukðtÞk1 k ¼ 1; . . . ; T ð25Þ

The weights wk(t) should emphasise sources that featuremore intense activity, as represented by Ek(t). Conse-quently, the weights wk(t) for each patch t can be estimatedby the contribution of the kth source image uk(t) over thetotal contribution of all the T source images at patch t,in terms of activity. Hence, we can choose

wkðtÞ ¼ EkðtÞXT

k¼1

EkðtÞ,

ð26Þ

There might be some cases, wherePT

k¼1EkðtÞ is very small,denoting small energy activity in the corresponding patch.As this can cause numerical instability, we can use the‘‘max-abs’’ or ‘‘mean’’ fusion rule for those patches.

4.2. Region-based image fusion using ICA bases

In this section, we will use the analysis of the inputimages in the estimated ICA domain to perform some re-gional segmentation and then we will fuse these regionsusing different rules, i.e. perform region-based image fusion.During, the proposed analysis methodology, we havealready divided the image in small N · N patches (i.e. re-gions). Using the splitting/merging philosophy of region-based segmentation [17], we can find a criterion to mergethe pixels corresponding to each patch in order to formcontiguous areas of interest.

One could use the energy activity measurement, as intro-duced by (25), to infer the existence of edges in the corre-sponding frame. As the ICA bases tend to focus on theedge information, it is clear that great values for Ek(t), cor-respond to great activity in the frame, i.e. the existence of

edges. In contrast, small values for Ek(t) denote the exis-tence of almost constant background in the frame. Usingthis idea, we can segment the image in two regions: (i)‘‘active’’ regions containing details and (ii) ‘‘non-active’’regions containing background information. The thresholdthat will be used to characterise a region as ‘‘active’’ or‘‘non-active’’ can be set heuristically to 2meant{Ek(t)}.Since we are not interested in creating the most accurateedge-detector, we can allow some tolerance around the realedges of the image. As a result, we form the following seg-mentation map mk(t) from each input image:

mkðtÞ ¼1; if EkðtÞ > 2meantfEkðtÞg0; otherwise

�ð27Þ

The segmentation map of each input image is combined toform a single segmentation map, using the logical OR oper-ator. As mentioned earlier, we are not interested in forminga very accurate edge detection map, but instead we have toensure that our segmentation map contains all the edgeinformation

mðtÞ ¼ ORfm1ðtÞ;m2ðtÞ; . . . ;mT ðtÞg ð28Þ

Now that we have segmented the image into ‘‘active’’ and‘‘non-active’’ regions, we can fuse these regions using differ-ent pixel-based fusion schemes. For the ‘‘active’’ region,we can use a fusion scheme that preserves the edges,i.e. the ‘‘max-abs’’ scheme or the weighted combinationscheme and for the ‘‘non-active’’ region, we can use ascheme that preserves the background information, i.e.the ‘‘mean’’ or ‘‘median’’ scheme. Consequently, this couldform a more accurate fusion scheme, that pays attention tothe structure of the image itself, rather than fuse informa-tion generically.

5. Experiments

In this section, we test the performance of the proposedimage fusion schemes based on ICA bases. It is not ourintention to provide an exhaustive comparison of the manydifferent transforms and fusion schemes that exist in litera-ture. Instead, a comparison with fusion schemes usingwavelet packets analysis and the Dual-Tree (Complex)Wavelet Transform are performed. In these examples wewill test the ‘‘fusion by absolute maximum’’ (max-abs),the ‘‘fusion by averaging’’ (mean), the Weighted Combina-tion (weighted) and the Region-based (Regional) fusion,where applicable.

We present three experiments, using both artificial andreal image data sets. In the first experiment, we have theGround Truth image Igt(x, y), which enable us to performnumerical evaluation of the fusion schemes. We assumethat the input images Ii(x, y) are processed by the fusionschemes to create the ‘‘fused’’ image If(x, y). To evaluatethe scheme�s performance, we can use the followingSignal-to-Noise Ratio (SNR) expression to compare theground truth image with the fused image

138 N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142

SNRðdBÞ ¼ 10 log10

Px

PyIgtðx; yÞ2P

x

PyðIgtðx; yÞ � If ðx; yÞÞ2

ð29Þ

As traditionally employed by the fusion community, wecan also use the Image Quality Index Q0, as a performancemeasure [19]. Assume that mI represents the mean of theimage I(x, y) and all images are of size M1 · M2. As�1 6 Q0 6 1, the value of Q0 that is closer to 1, indicatesbetter fusion performance

Q0 ¼4rIgtIf mIgt mIf

ðm2Igtþ m2

IfÞðr2

Igtþ r2

IfÞ ð30Þ

where

r2I ¼

1

M1M2 � 1

XM1

x¼1

XM2

y¼1

ðIðx; yÞ � mIÞ2 ð31Þ

rIJ ¼1

M1M2 � 1

XM1

x¼1

XM2

y¼1

ðIðx; yÞ � mIÞðJðx; yÞ � mJ Þ ð32Þ

We trained the ICA and the topographic ICA bases using10000 8 · 8 image patches selected randomly from 10images of similar content to the ground truth or the ob-served scene. We used 40 out of the 64 possible bases toperform the transformation in either case. We comparedthe performance of the ICA and topographic ICA trans-forms (topoICA) with a Wavelet Packet decomposition1

and the Dual-Tree Wavelet Transform.2 For the WaveletPacket decomposition (WP), we used Symmlet-7 (Sym7)bases, with 5 level-decomposition using Coifman–Wicker-hauser entropy. For the Dual-Tree Wavelet Transform(DTWT), we used 4 levels of decomposition and the filtersincluded in the package. In the next pages, we will presentsome of the resulting fusion images. However, the visualdifferences between the fused images may not be very clearin the printed version of the paper, due to limitation inspace. Consequently, the reader is prompted to acquirethe whole set either by download3 or via email to us.

5.1. Experiment 1: Artificially distorted images

In the first experiment, we have created three images ofan ‘‘airplane’’ using different localised artificial distortions.The introduced distortions can model several differenttypes of degradation that may occur in visual sensor imag-ing, such as motion blur, out-of-focus blur and finally pixe-late or shape distortion, due to low bit-rate transmission orchannel errors. This synthetic example can be a good start-ing point for evaluation, as there are no registration errorsbetween the input images and we can perform numericalevaluation, as we have the ground truth image. We applied

1 We used WaveLab v8.02, as available at http://www-stat.stanford.edu/~wavelab/.

2 Code available online by the Polytechnic University of Brooklyn, NYat http://taco.poly.edu/WaveletSoftware/.

3 http://www.commsp.ee.ic.ac.uk/~nikolao/ElsevierImages.zip.

all possible combinations of transforms and the fusionrules (the ‘‘Weighted’’ and ‘‘Regional’’ fusion rules cannotbe applied in the described form for the WP and DTWTtransforms). Some results are depicted in Fig. 5, whereasthe full numerical evaluation is presented in Table 1.

We can see that using the ICA and the TopoICA bases,we can get better fusion results both in visual quality andmetric quality (PSNR, Q0). We observe the ICA bases pro-vide an improvement of �1.5–2 dB, compared to the wave-let transforms, using the ‘‘max-abs’’ rule. The topoICAbases seem to score slightly better than the normal ICAbases, mainly due to better adaptation to local features.In terms of the various fusion schemes, the ‘‘max-abs’’ ruleseems to give very low performance in this example usingvisual sensors. This can be explained, due to the fact thatthis scheme seems to highlight the important features ofthe images, however, it tends to lose some constant back-ground information. On the other hand, the ‘‘mean’’ rulegives the best performance (especially for the wavelet coef-ficient), as it seems to balance the high detail with the low-detail information. However, the ‘‘fused’’ image in this caseseems quite ‘‘blurry’’, as the fusion rule has oversmoothedthe image details. Therefore, the high SNR has to be cross-checked with the actual visual quality and image percep-tion, where we can clearly that the salient features havebeen filtered. The ‘‘weighted combination’’ rule seems tobalance the pros and cons of the two previous approaches,as the results feature high PSNR and Q0 (inferior to the‘‘mean’’ rule), but the ‘‘fused’’ images seem sharper withcorrect constant background information. In Fig. 4, wecan see the segmentation map created by (27) and (28).The proposed region-based scheme manages to capturemost of the salient areas of the input images. It performsreasonably well as an edge detector, however, it producesthicker edges, as the objective is to identify areas aroundthe edges, not the edges themselves. The region-basedfusion scheme produces similar results to the ‘‘Weighted’’fusion scheme. However, it seems to produce better visualquality in constant background areas, as the ‘‘mean’’ rule ismore suitable for the ‘‘non-active’’regions (Fig. 5).

5.2. Experiment 2: The ‘‘Toys’’ dataset

In the second experiment, we use the ‘‘Toys’’ example,which is a real visual sensor example provided by LehighImage Fusion group [4]. In this example, we have three reg-istered images with different focus points, observing thesame scene of toys (Fig. 6). In the first image, we havefocused on left part, in the second on the center part andin the third image on the right part of the image. Theground truth image is not available, which is very commonin many multi-focus examples. Therefore, SNR-type mea-surements are not available in this case.

Here, we can see that the ICA and TopoICA bases per-form slightly better than wavelet-based approaches. Also,we can see that the ‘‘max-abs’’ rule performs slightly betterthan any other approach, with almost similar performance

Fig. 4. Region mask created for the region-based image fusion scheme.The white areas represent ‘‘active’’ segments and the black areas ‘‘non-active’’ segments.

Table 1Performance comparison of several combinations of transforms and fusion rules in terms of PSNR (dB)/Q0 using the ‘‘airplane’’ example

WP (Sym7) DT-WT ICA TopoICA

Max-abs 13.66/0.8247 13.44/0.8178 14.48/0.8609 14.80/0.8739

Mean 22.79/0.9853 22.79/0.9853 17.41/0.9565 17.70/0.9580

Weighted – – 17.56/0.9531 17.70/0.9547

Regional – – 17.56/0.9533 17.69/0.9549

N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142 139

from the ‘‘Weighted’’ scheme. The reason might be that thethree images have the same colour information, however,

Fig. 5. Three artificially-distorted input images and various

most parts of each image are blurred. Therefore, the‘‘max-abs’’ that identifies the greatest activity seems moresuitable for a multi-focus example.

5.3. Experiment 3: Multi-modal image fusion

In the third example, we explore the performance inmulti-modal image fusion. In this case, the input imagesare acquired from different modality sensors to unveil dif-ferent components in the observed scene. We have usedsome surveillance images from TNO Human Factors,provided by Toet [18]. More of these can be found in theImage Fusion Server [3]. The images are acquired bythree kayaks approaching the viewing location from faraway (Fig. 7). As a result, their corresponding image size

fusion results using various transforms and fusion rules.

Fig. 6. The ‘‘Toys’’ data-set demonstrating several out-of-focus examples and various fusion results with various transforms and fusion rules.

140 N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142

varies from less than 1 pixel to almost the entire field ofview, i.e. they are minimal registration errors. The firstsensor (AMB) is a Radiance HS IR camera (Raytheon),the second (AIM) is an AIM 256 microLW camera andthe third is a Philips LTC500 CCD camera. Consequently,

we get three different modality inputs for the sameobserved scene. However, the concept of ground truth isnot really meaningful in this case and therefore, we cannothave any numerical performance evaluation for thisexample.

Fig. 7. Multi-modal image fusion: three images acquired through different modality sensors and various fusion results with various transforms and fusionrules.

N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142 141

In this example, we can witness some effects ofmisregistration in the fused image. We can see thatall four transforms seem to have included mostsalient information from the input sensor images, espe-

cially in the ‘‘max-abs’’ and ‘‘weighted’’ schemes. How-ever, it seems that the fused image created using theICA and the TopoICA bases looks sharper and lessblurry.

142 N. Mitianoudis, T. Stathaki / Information Fusion 8 (2007) 131–142

6. Conclusion

In this paper, the authors have introduced the use ofICA and topographical ICA bases for image fusion appli-cations. These bases seem to construct very efficient tools,which can compliment common techniques used in imagefusion, such as the Dual-Tree Wavelet Transform. The pro-posed method can outperform wavelet approaches. Thetopographical ICA bases offer more accurate directionalselectivity, thus capturing the salient features of the imagemore accurately. A weighted combination image fusionrule seemed to improve the fusion quality over traditionalfusion rules in several cases. In addition, a region-based ap-proach was introduced. At first, segmentation into ‘‘active’’and ‘‘non-active’’ areas is performed. The ‘‘active’’ areasare fused using the pixel-based weighted combination ruleand the ‘‘non-active’’ areas are fused using the pixel-based‘‘mean’’ rule.

The proposed schemes seem to increase the computa-tional complexity of the image fusion framework. Theextra computational cost is not necessarily introduced bythe estimation of the ICA bases, as this task is performedonly once. The bases can be trained offline using selectedimage samples and then employed constantly by the fusionapplications. The increase in complexity comes from the‘‘sliding window’’ technique that is introduced to achieveshift invariance. Implementing this fusion scheme in a morecomputationally efficient framework than MATLAB willdecrease the time needed for the image analysis and synthe-sis part of the algorithm.

For future work, the authors would be looking at evolv-ing to a more autonomous fusion system. The fusionsystem should be able to select the essential coefficientsautomatically, by optimizing several criteria, such as activ-ity measures and region information. In addition, theauthors would like to explore the nature of ‘‘topography’’,as introduced by Hyvarinen et al., and form more efficientactivity detectors, based on topographic information.

Acknowledgements

This work is supported by the Data Information Fusionproject of the Defence Technology Centre, UK. Theauthors would like to thank the three anonymous reviewersfor their insightful comments and suggestions.

References

[1] A. Cichocki, S.I. Amari, Adaptive Blind Signal and Image Process-ing. Learning Algorithms and Applications, John Wiley &Sons, 2002.

[2] R.R. Coifman, D.L. Donoho, Translation-invariant de-noising,Technical report, Department of Statistics, Stanford University,Stanford, California, 1995.

[3] The Image fusion server. Available from: <http://www.imagefu-sion.org/>.

[4] Lehigh fusion test examples. Available from: <http://www.ece.le-high.edu/spcrl/>.

[5] A. Goshtasby, 2-D and 3-D Image Registration: For Medical,Remote Sensing, and Industrial Applications, John Wiley & Sons,2005.

[6] P. Hill, N. Canagarajah, D. Bull, Image fusion using complexwavelets, in: Proceedings of the 13th British Machine VisionConference, Cardiff, UK, 2002.

[7] A. Hyvarinen, Fast and robust fixed-point algorithms for indepen-dent component analysis, IEEE Transactions on Neural Networks 10(3) (1999) 626–634.

[8] A. Hyvarinen, Survey on independent component analysis, NeuralComputing Surveys 2 (1999) 94–128.

[9] A. Hyvarinen, P.O. Hoyer, M. Inki, Topographic independentcomponent analysis, Neural Computing 13 (2001).

[10] A. Hyvarinen, P.O. Hoyer, E. Oja, Image denoising by sparse codeshrinkage, in: S. Haykin, B. Kosko (Eds.), Intelligent Signal Process-ing, IEEE Press, 2001.

[11] A. Hyvarinen, J. Karhunen, E. Oja, Independent ComponentAnalysis, John Wiley & Sons, 2001.

[12] J.J. Lewis, R.J. O�Callaghan, S.G. Nikolov, D.R. Bull, C.N.Canagarajah, Region-based image fusion using complex wavelets,in: Proceedings of the 7th International Conference on InformationFusion, Stockholm, Sweden, 2004, pp. 555–562.

[13] H. Li, S. Manjunath, S. Mitra, Multisensor image fusion using thewavelet transform, Graphical Models and Image Processing 57 (3)(1995) 235–245.

[14] S.G. Nikolov, D.R. Bull, C.N. Canagarajah, M. Halliwell, P.N.T.Wells, Image fusion using a 3-d wavelet transform, in: Proceedings ofthe 7th International Conference on Image Processing and itsApplications, 1999, pp. 235–239.

[15] G. Piella, A general framework for multiresolution imagefusion: from pixels to regions, Information Fusion 4 (2003) 259–280.

[16] O. Rockinger, T. Fechner, Pixel-level image fusion: the case of imagesequences, SPIE Proceedings 3374 (1998) 378–388.

[17] M. Sonka, V. Hlavac, R. Boyle, Image Processing, Analysisand Machine Vision, second ed., Brooks/Cole Publishing Company,1999.

[18] A. Toet, Targets and Backgrounds: Characterization and Represen-tation VIII, The International Society for Optical Engineering, 2002,pp. 118–129.

[19] Z. Wang, A.C. Bovik, A universal image quality index, IEEE SignalProcessing Letters 9 (3) (2002) 81–84.