RECURSIVE IMAGE ESTIMATION AND INPAINTING IN NOISE USING NON-GAUSSIAN MRF PRIOR A THESIS submitted by G. R. K. SAI SUBRAHMANYAM for the award of the degree of DOCTOR OF PHILOSOPHY DEPARTMENT OF ELECTRICAL ENGINEERING INDIAN INSTITUTE OF TECHNOLOGY, MADRAS JANUARY 2008
162
Embed
RECURSIVE IMAGE ESTIMATION AND INPAINTING IN … · RECURSIVE IMAGE ESTIMATION AND INPAINTING IN NOISE USING NON-GAUSSIAN MRF PRIOR A THESIS submitted by G. R. K. SAI SUBRAHMANYAM
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
RECURSIVE IMAGE ESTIMATION AND
INPAINTING IN NOISE USING
NON-GAUSSIAN MRF PRIOR
A THESIS
submitted by
G. R. K. SAI SUBRAHMANYAM
for the award of the degree
of
DOCTOR OF PHILOSOPHY
DEPARTMENT OF ELECTRICAL ENGINEERINGINDIAN INSTITUTE OF TECHNOLOGY, MADRAS
JANUARY 2008
At the lotus feet of my samrdha sadgure
Bhagavaan Sri Venkaiah Swamy
THESIS CERTIFICATE
This is to certify that the thesis entitled “RECURSIVE IMAGE ESTIMATION
AND INPAINTING IN NOISE USING NON-GAUSSIAN MRF PRIOR”
submitted by G. R. K. S. Subrahmanyam to the Indian Institute of Technology,
Madras for the award of the degree of Doctor of Philosophy is a bona fide record of the
research work carried out by him under our supervision. The contents of this thesis,
in full or in parts, have not been submitted to any other Institute or University for
the award of any degree or diploma.
Chennai 600 036 (Dr. A.N. Rajagopalan and Dr. R. Aravind)
Date: Jan. 09, 2008 Research Guides
ACKNOWLEDGEMENTS
I vow my great debt of gratitude and offer my heart felt pranamas to the lotus feet
of merciful Bhagavan Sri Venkayya Swami of Golagamudi who blessed me to perform
PhD. I express my gratitude with utmost reverence to my Guru Sri P. Subbaramayya
sir who took loving care regarding my academic, physical and spiritual welfare during
my PhD. To Sri Achraya Bharadwaja master who enlightened me through his holy
books and talks a lot to pursue a student carrier with noble motivations.
I would like to express my deep sense of gratitude and reverence to my thesis
adviser Prof. A. N. Rajagopalan for his constant guidance and encouragement. His
cooperation that is reachable even to address the minute aspects such as in algorithmic
coding and editing tex files with a great care are the motivating reminiscence for me.
I would also like to offer my deep admiration and gratitude to my co-adviser
Prof. R. Aravind for his understanding and support especially during my thesis work.
Without his in depth clarifications and discussions it would not have been possible to
make many points explicit in this thesis.
I specially thank Ibrahim whose work and program codes became an interesting
and easy starting points of my work. I would like to convey my warm regards to Suresh
whose seniority and co-operation enable me to proceed fastly. I thank Rajiv, Aranov,
Paramanad and our other IPCV lab colleagues for their timely help. I acknowledge
the unanimous authors from whose publications I need to crop figures.
Finally, I bow at the lotus feet of Sri Sainath of Shiridi who blessed me with home
food and satsang during my study through my mother. I convey my heart full love to
mother who took all pains to follow the verdict of swami by providing company and diet
for my recreative health, to my father unless whose consent and cooperation it could
not have happened, and best wishes to my brother for his spirit of encouragement.
- G. R. K. Sai Subrahmanyam
ABSTRACT
Keywords: Image estimation, Markov random field (MRF), discontinuity-adaptive
Rm,n : NSHP support of order M1 at location (m, n)
c : A clique
C : Collection of all cliques
F : Random field
f : Configuration of F
p(.) : Probability density function
Vc : Potential function of clique c
γ : A constant controlling the convexity of DA function
ρm,n : Scale parameter of DAMRF density
s(m, n) : Pixel intensity at location (m, n)
s : vector containing first-order NSHP support of (m, n)
η2(s(m, n), s) : Energy term at (m, n) with neighbors s
x(m, n) : State vector at location (m, n)
y(m, n) : Known measurement at location (m, n)
u(m, n) : State noise with variance σ2u
u(m, n) : Vector state noise
v(m, n) : Measurement noise with variance σ2v
v(m, n) : Vector measurement noise
xvii
a(i, j) : AR model coefficients
E : Expectation operator
q(.) : Sampler density
zl : lth sample of random variable z
L : Number of samples
µp : Estimate of predicted mean
σp : Estimate of predicted covariance
f(.) : Known system transition function
F : State transition matrix
h(.) : Known measurement degradation function
g(.) : Arbitrary nonlinear function
s(m, n) : Mean estimate of the pixel intensity at location (m, n)
sk : Mean estimate of the state vector at location k
Pk : Estimate of the state Covariance at location k
Pk/k−1 : Predicted state Covariance matrix at location k
X : State sigma point matrix
Y : Measurement sigma point matrix
w : UT weights
κ, λ : UT scaling parameters
P(m, n) : State Covariance at location (m, n)
Px : Covariance of x at location (m, n)
Pxy : Cross covariance of x and y at location (m, n)
x(m, n) : Mean estimate of the state vector at location (m, n)
e(m, n) : Degraded image pixel in the exposure domain
Ω : Inpainting region
δΩ : Boundary of the inpainting region
xviii
CHAPTER 1
INTRODUCTION
1.1 IMAGE DEGRADATIONS
Imaging systems introduce different types of degradations resulting in imperfect ren-
dition of the original scene. Blur is a spatial degradation caused by relative mo-
tion between scene and camera or by an out-of-focus optical system [19]. Noise is
a point-wise degradation and refers to stochastic variations in the image intensity.
The thermal motion of electrons in electronic components constitutes electronic noise.
Thermal noise is additive white Gaussian (AWG) and signal-independent [1]. Quanti-
zation noise that results from digitization can be modeled by correlated space-varying
Gaussian distribution [20]. Photon noise is non-additive and Poisson distributed but
approaches the normal distribution for large numbers [1]. The perturbation in mag-
netic resonance imaging systems is characterised by additive white Gaussian noise
(AWGN) [21]. Restoring an original image in the absence of blur is referred to as
image estimation.
Yet another type of degradation arises from the nonlinear input-output charac-
teristics of imaging sensors and scanners. In photographic films, the silver density
deposited on the film is logarithmically related to the light exposure [3, 4]. The ran-
dom grains that develop during the film formation process result in film-grain noise
which can be modeled as AWG in the film density domain [4]. Aerial images captured
by synthetic aperture radar (SAR) systems are affected by multiplicative noise known
as speckle [22]. It results from the interference of back-scattered radar signals and is
usually modeled by a Gamma distribution [12].
Image degradations are not limited to only the above and include situations (such
1
as in old films and photographs) where there is complete loss of information in some
regions of the image due to aging, scratches, etc [17, 23]. Sometimes important infor-
mation is lost because of occlusion by another object or by superimposed text [6, 24].
The process of filling-in lost information is called image inpainting [6].
In this thesis, we investigate and propose new methods for image estimation in
AWGN, film-grain noise reduction, SAR despeckling, and joint image inpainting and
noise filtering.
1.2 LITERATURE REVIEW
Developing effective image recovery techniques is important for improving image qual-
ity, and continues to be an active area of research. These techniques must not only
rely on knowledge of the degradation process and the statistics of the noise, but must
also effectively exploit contextual image characteristics. They differ in incorporating
the prior knowledge about the original scene into the estimation procedure. Image re-
covery in the presence of noise is difficult not only because noise can be non-Gaussian
and/or non-additive, but also due to the fact that preservation of edges is equally
important for good qualitative and quantitative performance.
1.2.1 Image Estimation in AWGN
Work on suppressing AWG noise began with frequency domain approaches which ex-
ploit the differences in the frequency or the spectral content between the original signal
and the noise for recovery. The classical Wiener filter (WF) [25] assumes stationar-
ity of signal and noise, and minimizes the mean-squared error between the original
and the restored image. This requires knowledge of the power spectral density of the
original image and noise. The filter attenuates frequencies depending on the value of
signal-to-noise ratio. The constrained least-squares filter [26] provides for additional
control over the restoration process.
The Kalman filter (KF) is well-known for estimation of signals in AWGN and is
2
based on dynamic state-space formulation [7]. The KF was extended to 2D in [8] and
applied to remove AWGN from images. The reduced-order model Kalman filter [27]
achieves equivalent performance but with reduced computational complexity. Effects
of distortion resulting from noise can be reduced by Kalman filtering, provided the
image parameters such as the auto-regressive (AR) coefficients are known. In general,
however, these parameters are a priori unknown, and can also vary spatially as a
function of the image coordinates. Image estimation by KF with space-varying AR
parameters is considered in [28]. Jeng and Woods [29] propose residual and normalized
image models using local statistics based on the observation that a residual image can
be better predicted (by the AR model) than the original gray level image. Many
algorithms explicitly incorporate image edge-information within the KF framework.
An edge-adaptive filter for restoration of noisy and blurred images based on multiple
models has been presented in [30]. A modified KF in which edge information is used
to improve step-response is presented in [31]. In [32], the parameters of the AR model
are continuously updated to model non-homogeneity of the image. Kadaba et al.
[11] observe that the error residuals in the AR model can be better fitted by Bernouli-
Laplacian distribution. They analytically derive a sub-optimal recursive Bayesian filter
that can partially incorporate the non-Gaussian nature of error residuals. In [33], they
extend their framework to incorporate Cauchy AR residuals.
Based on the prior models of the image and the likelihood density specified by
the degradation model, many maximum a posteriori (MAP) techniques have been
developed. A great deal of image processing work stems from prior stochastic models
for image structure given by Markov random fields (MRFs). Geman and Geman [34]
exploit the equivalence between MRFs and Gibbs distributions to model images. The
optimal image estimate is given as a fixed point of an iterative procedure that relies on
neighborhood-dependent updates. Several MAP-based approaches have been proposed
to incorporate edge-preserving priors in the estimation procedure [35, 36]. In [37], a
compound Gauss MRF model is proposed for the image and its MAP estimate is
obtained by simulated annealing. Roth et. al. [38] propose a Field of Experts (FoE)
3
model by learning the prior probability with generic images within a higher-order
MRF formulation. Edge-preserving image recovery using discontinuity-adaptive finite
Markov random fields has been considered in [39]. A combination of homogeneous and
inhomogeneous conditional densities has been used in [40] for Bayesian estimation of
images. There is also an increasing interest in handling non-Gaussian situations using
Monte-Carlo sampling methods [41, 42].
Yet another class of algorithms is based on partial differential equations (PDEs)
[43, 44]. These methods have their origins in heat equation (called as anisotropic
diffusion) [45]. The total variation of an image is minimized numerically in [46] subject
to constraints involving the statistics of the noise. In recent years, wavelet transform-
based approaches have been proposed which perform a multi-resolution analysis of the
degraded image to effectively differentiate and suppress noise in the wavelet domain
[47, 48]. An approach that exploits the sparseness and independent nature of wavelet
coefficients is presented in [49]. The importance of prior modeling of the wavelet
coefficients is brought out in [50].
1.2.2 Photographic Film-Grain
In this problem, the main issue is to handle sensor non-linearity. Existing methods
for film-grain noise removal include extensions of the Wiener filter (WF) [4], MAP-
based approaches [51], and wavelet techniques [52]. An adaptive Wiener filter based
on non-stationary first-order statistics of the image is proposed in [53]. Tekalp et al. [4]
incorporate the effect of sensor nonlinearity into the Wiener filter structure to recover
from blurred and noisy photographic images. Both these variations of the Wiener filter
[4,53] result in good improvements over the conventional Wiener filter for suppressing
film-grain noise. In [54], the noise model parameters are estimated using higher-order
statistics. The filter coefficients are obtained by minimizing a cumulant criterion based
on the extended Wiener-Hopf equations.
Andrews et al. [3] expand the nonlinear observation model into a Taylor series
and derived an approximate filter for recovering the original image. In [55], using
4
the D − log E curve of the film, and assuming Gaussian models for image and noise
statistics, a MAP estimate of the restored image is derived. The resulting non-linear
equations are solved by steepest-descent using fast transform computations, and sig-
nificant improvements are reported over the WF. Image estimation based on different
priors is considered in [56]. The authors employ numerical integration and solve higher-
order polynomials to find the MAP estimate. In [51], the image is modeled as Gaussian
and a signal-independent transformation is used to reduce the complexity of the MAP
solution. Moldovan et al. [57] propose a Bayesian approach using learned spatial prior
and a likelihood based on inhomogeneous beta distribution. This is used to de-grain
individual frames of archival films. The Particle filter (PF) has been recently pro-
posed for linear and non-linear image estimation in [58]. The PF discussed in [59]
outperforms the well-known modified WF (MWF) [4] in suppressing film-grain noise
in photographic images.
An undecimated wavelet domain technique for reducing film-grain noise is pro-
posed in [52]. Using a set of training images, the wavelet coefficients are thresholded
for different noise levels by optimizing a cost function which is related to visual quality.
In [60], film-grain noise reduction is performed both in spatial and wavelet domains.
The detail wavelet coefficients are adaptively shrunk using the local statistics derived
from the noisy image. The wavelet filter is shown to outperform its spatial counter-
part. Noise reduction in video sequences of old movies is accomplished in [61] with a
combination of wavelet and Wiener filtering techniques.
1.2.3 SAR Speckle
Research on multiplicative speckle suppression in SAR images began with adaptive
denoising techniques which rely on local statistics. Frost et al. [62] explore the property
of coefficient of variation and propose an exponentially weighted kernel for speckle
suppression. Adaptive speckle suppression filters proposed by Lee et al. [63] and Kuan
et al. [64] use the local statistics of the degraded image and the multiplicative model
of speckle noise. Enhanced versions proposed by Lopes et al. [65] filter independently
5
in homogeneous, heterogeneous and isolated point regions. Other well-known early
approaches to SAR despeckling include the Gamma-MAP filter [66] and the nonlinear
geometric filter [67]. In [68], speckle is suppressed by employing an adaptive block-
Kalman filter, which relies on block-varying AR parameters of the original image.
An adaptive MAP estimator with a heavy-tailed Rayleigh model has been suggested
in [12]. Speckle suppression by anisotropic diffusion using PDE-based approaches is
considered in [69, 70].
In recent years, wavelet techniques have been found to be effective for SAR image
denoising. Wavelet despeckling filters adaptively modify the wavelet coefficients of
the log-transformed speckled image. A wavelet method based on soft-thresholding has
been proposed in [15]. Xie at el. [71] have proposed a despeckling algorithm that fuses
Bayesian wavelet denoising with a regularizing Gaussian-mixture prior. Many SAR
despeckling methods report improved performance with non-Gaussian prior modeling
of the wavelet coefficients within the MAP framework [72,73]. The choice of prior for
encoding SAR data is discussed in [74].
1.2.4 Digital Inpainting
Digital inpainting provides a means for reconstruction of small identified damages and
is also useful for removing superimposed text such as dates, subtitles, publicity or
logos from still images or videos [6, 17]. The main issue in inpainting methods is to
recover image information at damaged edges and lost texture regions [75]. There are
two major categories of inpainting algorithms. The first category propagates infor-
mation along the isophote (lines of equal gray values) direction [6] by formulating
the problem in terms of partial differential equations or elastica models. The second
category relies on edge or structure information [17]. We provide a detailed review
of inpainting approaches in chapter 7. Traditional inpainting algorithms aim only at
filling-in identified damages or removing superimposed text or small objects in images
[6, 17]. To the best of our knowledge, simultaneous image recovery from (film-grain)
noise and damages has not been addressed in the literature. In traditional denoising,
6
the pixels contain information about underlying data as well as noise, while in image
inpainting, there is no information available in the region to be inpainted [6] which
compounds the problem further.
1.3 MOTIVATION
The classical problem of image estimation in AWGN is difficult because one must meet
the (contradictory) but twin objective of noise reduction and detail preservation. The
Kalman filter is an elegant recursive filter that has been traditionally employed for
this purpose. However, the homogeneous image model, generally employed in KF [8],
can only provide limited compromise between noise removal and edge preservation.
For better performance, exploiting the non-stationary feature of the KF, algorithms
have been proposed to model the images with space-varying AR state models [28].
The state-space models are formulated with respect to residual and normalized images
to benefit from the use of the local statistics of the image [29]. However, such an
extension requires that the autocorrelation coefficients of the noise-free image be known
accurately. Explicit incorporation of edge-preserving priors within the KF framework
is an interesting research issue [11, 33].
The ubiquitous film is an important image recording medium. The degradation
phenomenon associated with it is film-grain noise which is not only visibly objection-
able but also renders compression difficult due to high entropy. Because the observa-
tion model is nonlinear, the noise turns out to be multiplicative and non-Gaussian in
the (visual) exposure domain [4]. A recent technique which is based on the recursive
particle filter (PF) [58] has the capability to incorporate sensor nonlinearity into its
estimation procedure. But its performance is constrained by the use of a homogeneous
autoregressive (AR) model. Also, it is computationally intensive. The widely em-
ployed nonlinear extension of the Kalman filter, the Extended Kalman filter (EKF),
only linearizes the state and observation nonlinear models about the (current) mean
of the state. The EKF is known to introduce considerably large errors in the estimates
7
and is prone to be unstable. Handling sensor nonlinearity in the presence of noise is a
challenging problem indeed.
In synthetic aperture radar (SAR) imaging systems, adaptive speckle suppression
filters [62,63] are based on the local statistics of the degraded image. These are simple
but tend to over-smooth image texture. They also suffer from ineffective denoising
around edges. The adaptive block-Kalman filter uses a block-wise varying AR model
for suppressing speckle noise [68]. However, it requires the AR coefficients of the noise-
free image to be known accurately. Recently proposed wavelet domain [13] and MAP
approaches [12] have superior performance but require explicit optimization and/or
parameter estimation. Developing methods for reducing the effect of speckle while
preserving point and line targets is an active area of research. A direct application
of the Kalman filter or its linear extensions [8, 11, 28] can only deal with linear and
additive noise models. The EKF is also limited by the additive model assumption.
Inpainting methods typically propagate information either along the isophote di-
rection or explicitly rely on edge or structure information. Edge-based inpainting
methods [17] first reconstruct the edge information in the damaged region. Then the
filling-in is performed within each object guided by the reconstructed edges. For this
class of algorithms, there are two main issues to be addressed: i) how do we recon-
struct the edges in the region to be inpainted using the boundary information? and
ii) how do we utilize the reconstructed edges in propagating gray level values into the
region to be inpainted? Research that addresses the problem of dual degradation due
to film-grain noise and loss of information is still in its infancy.
1.4 OBJECTIVES AND SCOPE OF THE THESIS
A common goal underlying all image estimation techniques is that of noise reduc-
tion in conjunction with feature preservation. We propose to meet this goal within
a recursive framework. Recursive techniques have excellent memory and implementa-
tional advantages. Moreover, they permit spatial adaptivity to be easily incorporated
8
into the filter model [76]. The state equation of the Kalman filter imposes a strong
linear constraint on state evolution which hinders true transitions at image edges.
We first propose a novel methodology to incorporate a non-Gaussian Markov ran-
dom field (MRF) prior into the recursive Kalman filter for estimating images cor-
rupted by AWGN. We begin with the intuition that incorporating edge-preserving
conditional priors can considerably improve performance. Specifically, the incorpora-
tion of a non-Gaussian prior in the Kalman filter amounts to determining the first two
moments of the prior at each recursive step. We formulate a discontinuity-adaptive
MRF (DAMRF) prior using the continuous adaptive regularizer model proposed in
[77, 78] so as to suit recursive estimation. However, the image model becomes non-
Gaussian and intractable for direct moment estimation. In recent years, there has
been an increasing interest in the application of Monte Carlo methods for several sig-
nal and image processing problems [41,58]. The power of these methods lies in random
sampling to approximate complex multi-dimensional integrals with simple summations
[79]. In particular, the importance sampling (IS) method [79,80] provides a convenient
methodology to estimate the moments under any arbitrary probability density func-
tion (pdf), known up to a multiplication constant. This is achieved by using weighted
samples of another roughly approximate pdf from which it is easy to draw samples.
We employ importance sampling to estimate the first two moments of the DAMRF
non-Gaussian prior which effectively uses the samples of an appropriately chosen sam-
pler density. Since the first two moments of the DAMRF prior constitute the predicted
mean and covariance of the state, the update is carried out exactly as in traditional
KF to arrive at the final image estimates. The performance of the proposed filter is
demonstrated with many examples.
The unscented Kalman filter (UKF), recently proposed by Julier and Uhlmann
[81], has been well-studied in the 1D domain and has been found to be more accurate
and stable than EKF [81]. It also has the capability to handle multiplicative, non-
Gaussian noise in its estimation procedure [82,83]. The UKF inherits the KF structure
and is the extension of a deterministic sampling approach known as the unscented
9
transformation (UT) to minimum mean-square error (MMSE) estimation [81]. In
UKF, the recursive prediction and updation of the first two moments of the state are
based on the propagation of sigma points (deterministic samples) through state and
observation models. The UKF has been applied to a wide variety of 1D nonlinear
signal estimation problems [81, 84, 85]. The UKF has also been explored for tracking
in computer vision [86, 87].
We develop an extension of the UKF for nonlinear image estimation, beginning
with the homogeneous AR state model. The sigma points are generated by assuming
an initial mean and covariance for the state and are propagated through the state and
nonlinear observation models resulting in the prediction of state and measurement
sigma points. The UT is employed to determine the predicted first two statistics and
the update is performed as in the Kalman filter.
Since the performance of AR based UKF (ARUKF) is limited by the AR state
model, we explore the incorporation of an edge-preserving MRF prior into the UKF
framework. Using a similar methodology as applied to improve the KF, we formulate
the DAMRF state conditional density based on the DA potential function and the
estimated neighbors. We employ IS to estimate the mean and covariance of the state
from the DAMRF prior. Unlike in the KF extension, here we employ the predicted
state statistics to determine the predicted sigma points and (directly) propagate them
through the nonlinear observation model to obtain the predicted measurement sigma
points. We carry out the update as in the ARUKF. We refer to this formulation of
the UKF as the importance sampling UKF (ISUKF).
We specifically consider film-grain noise removal in photographic prints, and speckle
suppression in SAR images, to demonstrate the effectiveness of the proposed ISUKF.
The ISUKF outperforms the ARUKF and also the PF [88] for image estimation in
film-grain noise. We modify the formulation of the DA prior for SAR imagery and ap-
ply the ISUKF with a multiplicative measurement model to arrive at despeckled SAR
images. Experiments on images with simulated and real speckle reveal the efficacy of
the proposed filter over contemporary methods.
10
Finally, we consider the problem of inpainting damaged images by extending the
proposed recursive UKF framework to simultaneously perform film-grain noise removal
and inpainting. We observe that the main issue in inpainting within the filtering
framework is to infer the observations from noisy neighborhood pixels. In the pro-
posed inpainting algorithm, we consider global information in terms of edge image
reconstruction and inpaint locally to restore the image. Using the damaged image and
information about the location of damages, we adopt a constrained matching strategy
to reconstruct the damaged edges in the regions to be inpainted. We inpaint the miss-
ing pixels by propagating the boundary information judiciously using the reconstructed
edge image. We validate the proposed approach with several examples.
To recover an image from damages as well as noise, we propose to embed the
inpainting method in the ISUKF formulation. Since the UKF lacks observations at
damages during its update step, we fill-in the missing observations by invoking our
inpainting method. We demonstrate the ability of the proposed unified framework for
inpainting and denoising of damaged film-grain noisy photographic images for both
simulated and real degradations.
1.4.1 Contributions of the Thesis
• We propose a discontinuity-adaptive image estimation scheme within the Kalman
framework by non-Gaussian MRF modeling of the image prior. This is achieved
by using importance sampling to estimate the first two moments of the non-
Gaussian prior and incorporating them into the update step of the KF.
• The unscented Kalman filter (UKF) with an AR state model is developed for
non-linear image estimation that can account for exact image sensor nonlinearity
in the observation model. To further improve performance, we incorporate an
edge-preserving Markov prior into the recursive estimation procedure of the
UKF through importance sampling. We employ this for film-grain noise removal
in photographic images.
• Next, the problem of despeckling of synthetic aperture radar (SAR) images is
11
considered. The discontinuity-adaptive MRF prior is modified so as to effectively
process SAR images and the UKF is formulated to account for multiplicative
noise in the image estimation procedure.
• We develop an edge-based image inpainting method and embed it into the pro-
posed UKF framework to simultaneously suppress film-grain noise and inpaint
(known) damages within a recursive framework.
1.5 ORGANIZATION OF THE THESIS
In chapter 2, we review the theory of Markov random fields.
Chapter 3 incorporates a discontinuity-adaptive MRF prior into the Kalman filter to
address the limitations of the AR-based KF for linear image estimation.
Chapter 4 introduces the principle of unscented transformation, analyzes the accuracy
of UT, illustrates UT with examples, and considers its extension to recursive nonlinear
estimation.
In chapter 5, we develop the UKF for nonlinear image estimation. We emphasize the
incorporation of edge-preserving MRF prior into the UKF and employ it for film-grain
noise suppression.
In chapter 6, we consider the problem of speckle noise suppression using the UKF
framework by suitably tailoring the prior for SAR images.
In chapter 7, the problem of filling-in missing information in noisy images is tackled
by embedding an edge-guided inpainting method into the UKF framework.
Chapter 8 summarizes the thesis and suggests directions for future work.
12
CHAPTER 2
MARKOV RANDOM FIELDS
2.1 INTRODUCTION
The study of random processes is important because filters or algorithms must be
developed in accordance with the probabilistic characteristics of the class of images
being processed [89]. A natural extension of a random process to two dimensions
is known as a random field. A realization of a random field is generated when we
perform a random experiment at each spatial location and assign the outcome of that
experiment to that location [90]. A Markov random field (MRF) is a conditional
density and possesses the Markovian property i.e., the value of a pixel depends only
on the values of its neighboring pixels and on no other pixel [77]. The qualifying
feature of MRFs in image processing is that the information contained in the local,
physical structure of images is captured by means of the local conditional probability
distribution [91]. It is important to realise that at discontinuities or edges, the notion
of neighborhood dependency should be relaxed. How to incorporate the smoothness
constraint in MRFs while simultaneously modeling the edges is a challenging task
[39, 77]. The theory of MRFs is built on the concept of a neighborhood system and
requires the following terminology for its representation and study.
2.1.1 Lattice, Sites, and Labels
Many vision tasks can be posed as labeling problems in which the solution to a task is
a set of labels assigned to image pixels or features. A labeling is specified in terms of a
set of sites and a set of (corresponding) labels. Typically, image data are represented
by gray-level variations defined over a finite rectangular or square point lattice. A
13
lattice S is a discrete set of points or pixels. For a 2D image of size M × N , it is
denoted by
S = (i, j) | 1 ≤ i ≤ M, 1 ≤ j ≤ N.
A lattice node t is uniquely specified by its coordinates t = (i, j), where i is the
image row number and j is the column number. A site represents a point or a region
in the Euclidean space. A site can be an image pixel (lattice node) or an image feature
such as a corner point, a line segment or a surface patch. Sites can be spatially regular
or irregular. A label is an event associated with a site. A label can assume either
continuous or discrete values. For example, the value of the analog pixel intensity is
an example of a continuous label, whereas the quantized values of intensities in the
set 0, 1, ..., 255, or edge, non-edge in edge detection, constitute discrete labels.
Another essential property of a label set is the ordering of the labels. For example,
elements in the continuous label set (the real space) can be ordered by the relation
“smaller than”. Let L be the set of labels. When a discrete set, say 0,1,2,...,255,represents pixel intensities, it is an ordered-set because for intensity values we have
0 < 1 < ... < 255. When a label set contains different symbols such as texture types
or edge features, it is considered to be un-ordered.
The problem of labeling is to assign a label from the set L to each of the sites in
the lattice S. The set of all possible combinations of labels is called the configuration
space and is denoted by F. In image estimation, the set S of sites corresponds to image
pixels and the discrete set L of labels is the intensity levels. For a discrete labeling
problem of k sites and l labels, there exist a total number of lk possible labellings (of
configuration space). However, among all the possibilities, there are only few which
are optimal in terms of a criterion measuring the goodness (or inversely, the cost) of
solutions. This is the optimization approach to visual labeling.
2.1.2 Neighborhood System and Cliques
One of the important characteristics of image data is statistical dependence of the
gray level at a lattice node on its neighbors. The sites in S are related to each other
14
(a) (b)
Fig. 2.1: Symmetric neighborhood system for (a) first-order, and (b) second-order.
through a neighborhood system defined as
N = Ni,j | ∀ (i, j) ∈ S (2.1)
where Ni,j is the set of sites neighboring the site (i, j). A collection of subsets of a
lattice S defined as N = Ni,j | (i, j) ∈ S, Ni,j ⊂ S is a neighborhood system on
S if the neighboring relationship satisfies the following properties:
1. (i, j) /∈ Ni,j (a site is not its own neighbor), and
2. if (k, l) ∈ Ni,j, then (i, j) ∈ Nk,l (the neighborhood relationship is mutual).
Neighborhood systems that are commonly used are N 1 and N 2. Figs. 2.1 (a) and
(b) show first-order and second-order symmetric neighborhood systems, respectively.
In the first-order neighborhood system (N 1), which is also called a 4-neighborhood
system, every site (m, n) has four neighbors as shown in Fig 2.1 (a). In the second-
order neighborhood system (N 2), also called an 8-neighborhood system, there are eight
neighbors for every site (shown in Fig. 2.1 (b)).
If we introduce the notion of time in a 2D representation, which is very useful
in recursive analysis of images, at a current pixel (m, n), only the “past” pixels such
as those shown in Fig. 2.2 become neighbors. Such a neighborhood support, known
as non-symmetric half-plane (NSHP) support (at a pixel (m, n)) is mathematically
15
Row
s Columns
Fig. 2.2: Non-symmetric half-plane support at a pixel (m, n).
(a) (b)
Fig. 2.3: NSHP neighborhood system for (a) first-order, and (b) second-order.
Neighborhood system N , using Nm,n over the whole lattice S, is defined by Eq. 2.1.
Typically, we employ first and second-order NSHP neighborhood systems as shown in
16
Fig. 2.4: Cliques for NSHP neighborhood.
Figs. 2.3(a) and (b), respectively. MRFs with NSHP support are also referred to as
unilateral MRF [92] or causal MRFs [93].
A clique c of the pair (S, N ) is a subset of sites in S such that it consists either
of a single site c = (m, n) or of a pair of neighboring sites c = (m, n), (k, l), or
of a triplet of neighboring sites c = (m, n), (k, l), (i, j) and so on. The collections of
single-site, pair-site and triple-site cliques will be denoted by C1, C2 and C3, respectively,
where C1 = (m, n) : (m, n) ∈ S, C2 = (m, n), (k, l) : (k, l) ∈ Nm,n, (m, n) ∈S, C3 = (m, n), (k, l), (i, j) : (m, n), (k, l), (i, j) ∈ S and are neighbors of one
another and so on. Similarly, we can define higher-order cliques. The sites in a clique
are ordered i.e., (m, n), (k, l) is not the same as (k, l), (m, n). The collection of
all cliques for (S,N ) is C = C1 ∪ C2 ∪ C3,... where “...” denotes possible set of larger
cliques. The order or type of cliques for (S,N ) is determined by the size, shape and
orientation of the neighborhood.
Fig. 2.4 shows the clique types for the first-order causal neighborhood system of a
lattice at site (m, n), identified with a dot. The single-site, horizontal and vertical pair-
site cliques in Figs. 2.4 (a) and (b) constitute a first-order symmetric neighborhood
system. While the clique types for a first-order NSHP neighborhood system include
not only those in Figs. 2.4 (a) and (b) but also diagonal pair-site cliques in Fig. 2.4
(c), triple-sites in Fig. 2.4 (d), and quadruple-site (Fig. 2.4 (e)) cliques. We note that
clique types for a second-order symmetric non-causal neighborhood system contain all
those in Figs. 2.4 (a)-(e) and a fourth possible triple-site (which is not an NSHP clique
and hence is not shown in Fig. 2.4 (d)). As the order of the neighborhood system
increases, the number of cliques grows rapidly.
17
2.2 MARKOVIANITY
For the purpose of image analysis and processing, it is useful to have an underlying
model for the dominant characteristics of the given data. Although it is often difficult
to identify the physical mechanism which generated the observed data, an analytical
expression that captures the statistical properties of images can be used as a model
[94]. Specifically, the theory of MRF provides a consistent way of modeling context-
dependent entities [77]. The implicit assumption behind probabilistic approaches to
image analysis is that, for a given problem, there exists a probability distribution that
can capture to some extent the variability and interactions of different sets of relevant
image attributes [95].
A random field F is a collection of random variables arranged on the lattice S and
is defined as F = Fm,n, (m, n) ∈ S. The random variables Fm,n, (m, n) ∈ S take
values in the label set L. Specifically, for 8-bit quantized images, L = 0, 1, ..., 255.Let F = LM×N be the configuration space, where the random field F takes values. Here,
M×N is the dimension of the image. Let each random variable Fi,j take a value fi,j in
L, identified with the notation Fi,j = fi,j . The associated probability p(Fi,j = fi,j) is
abbreviated as p(fi,j). We use the notation (F1,1 = f1,1, ..., FM,N = fM,N) for the joint
event. For simplicity, this joint event is abbreviated as F = f where f = f1,1, ..., fM,Nis a configuration of F , corresponding to a realization of the field. The joint probability
p(F = f) is abbreviated as p(f). We define f c = fk,l : (k, l) ∈ S − (i, j). The
where Rm,n is the NSHP support for a site (m, n) as defined in Eq. (2.2).
In addition to the traditional discrete [96], continuous [77] and causal MRFs [93],
there exist several variants of MRFs such as discriminative RF [97], conditional RFs
[98], and double random fields [99]. The utility of MRF theory has also been extended
to 3D [94] for volumetric analysis of images and for modeling the temporal dimension in
videos. Reconstruction of non-texture surfaces is an example application of continuous
MRFs where we need to assign continuous labels for a sampling of the underlying
surface. In the restoration of a quantized image, we employ discrete MRFs. MRFs
with a symmetric neighborhood fall into the class of noncausal MRFs while the ones
with NSHP support belong to the class of causal MRFs.
2.3 CLIQUE POTENTIALS
The cliques defined in the earlier section are characterised by their strengths referred to
as clique potentials. The value of a clique potential depends on the local configuration
of the clique c and the chosen potential function. Lower the potential of a certain
clique, the closer is the current site to its neighbors within that clique. The sum of
19
clique potentials is a measure of the energy of the spatial activity at that site. The
energy function determines the probability of a current site given its neighbors.
2.3.1 Potential Function and Gibbs Random Field
A random field F = Fi,j defined on S is a Gibbs random field (GRF) with respect
to the neighborhood system N , if and only if, its joint distribution is of the form
p(f) =1
Zexp −U(f) (2.5)
where Z =∑
f∈Fexp −U(f) is a normalizing constant known as the partition func-
tion, and U(f) is the energy function defined as the sum of clique potentials over all
possible cliques C and given by
U(f) =∑
(m,n)∈S,c∈C1
Vc(fm,n) +∑
(m,n)∈S
∑
(k,l)∈N(m,n),c∈C2
Vc(fm,n; fk,l) + ... (2.6)
where Vc(fm,n) and Vc(fm,n; fk,l) are known as clique potentials such that Vc(.) ≥ 0.
This ensures that the Gibbs energy U(f) is non-negative. The distribution given by
Eq. (2.5) is called Gibbs distribution (GD). The expression for the joint distribution
in Eq. (2.5) has the physical interpretation that smaller the value of U(f), which is
the energy of a particular realization f , the more probable that realization is. A GRF
is said to be homogeneous if Vc(f) is independent of the relative position of the clique
c in S. It is said to be isotropic if it is independent of the orientation of c.
The clique potential Vc(.) is a function of the values of sites in clique c. It quantifies
the irregularity in the neighborhood. It helps in encoding prior information about the
image to be estimated to restrict the space of possible solutions. In fact, the power of
MRF lies in a careful selection of the clique potential Vc(f). Contextual constraints
on two labels are the lowest-order constraints to convey contextual information. They
are widely used not only because of their simple form and low computational cost, but
also due to the fact that they are usually sufficient to describe the local characteristics
of images. They are encoded in the Gibbs energy as pair-site clique potentials. The
20
cliques with only pair-wise interactions are known in MRF terminology as auto-models
[77, 100]. For these models, the clique potential Vc(.) = 0 for |C| > 2.
There exist several MRF models based on the choice of the potential function
and the label set. The choice of model is application-dependent. For instance, an
auto-logistic model [77], which has a binary label set, is appropriate for modeling
binary textures; multi-level logistic (or ”colour-blind”) models [77] with (multi-valued)
discrete label set are suitable for textures. The most widely used models are the
spatial autoregressive (AR) models or the auto-normal models (Gaussian MRFs) [40,
91]. These appear in a variety of applications such as texture feature extraction,
classification and segmentation. An extensive presentation and comparison of different
classes of MRF models can be found in [101].
2.3.2 MRF-Gibbs Equivalence
An MRF is characterized by its local (conditional) property whereas a GRF represents
global (joint) property. A theorem by Hammersley and Clifford [77,102] establishes the
equivalence between MRF and GRF and yields the joint distribution from conditional
distributions. The first part of this theorem states that any GRF is an MRF. Here, our
interest is to derive an expression for calculating the conditional probability p(fi,j/fc)
from potential functions based on the assumption that F is a GRF. For completeness,
we state the theorem but give the proof for only the part that is of interest to us.
Hammersley-Clifford Theorem: A random field F = Fi,j is a Markov
random field with respect to neighborhood N , if and only if, its joint density function
is a Gibbs distribution with cliques associated with N .
Proof : We now only prove that a GRF is an MRF [77,102], which enables us to
derive the conditional density from the clique potential functions.
Let p(f) be a Gibbs distribution on S with respect to the neighborhood system
N . The conditional probability can be written as
p(fi,j/fc) =
p(f)
p(f c)=
p(f)∑f ′i,j∈L
p(f ′)(2.7)
21
where f ′ = f1,1, · · · , fi,j−1, f′i,j, fi,j+1, · · · , fM,N is any configuration that agrees with
f at all sites except at site (i, j). Since we assume that F is a GRF, using Eq. (2.5)
and Eq. (2.6), we can write
p(fi,j/fc) =
exp−∑c∈C Vc(f)∑f ′i,j
exp−∑c∈C Vc(f ′)(2.8)
Dividing the set of cliques C into sets A consisting of cliques containing site (i, j) and
B with cliques not containing site (i, j), we can write
p(fi,j/fc) =
exp−∑c∈A Vc(f) exp−∑c∈B Vc(f)∑f ′i,j
exp−∑c∈A Vc(f ′) exp−∑c∈B Vc(f ′)(2.9)
Since by definition Vc(f) = Vc(f′) for any clique c that does not contain site (i, j),
canceling common terms we get
p(fi,j/fc) =
exp−∑c∈A Vc(f)∑f ′i,j
exp−∑c∈A Vc(f ′)(2.10)
Eq. (2.10) provides a formula for calculating the conditional density from the potential
functions, based on the assumption that F is a GRF. We note that Eq. (2.10) resembles
a Gibbs random field (Eq. 2.5), except that now the energy function is obtained by
summing the clique potentials of only the neighboring sites of the current site. We refer
[103] for proof of the converse statement. In the light of this equivalence, we simply
specify the local “interaction” factors to define, up to some multiplicative constant,
the joint probability distribution p(f). With such a setup, each variable only directly
depends on a few other “neighboring” variables. From a more global point of view, all
the variables are mutually dependent, but only through a combination of successive
local interactions [95].
2.4 MRF PRIORS
In a Bayesian framework, many image analysis tasks such as edge detection, template
matching, super resolution and restoration can be posed as inference problems [77,100].
The aim is to estimate the actual random signal f from its observation e. Bayesian
22
techniques provide a way to invert an observation model taking the prior knowledge
p(f) into account. This is done by applying the Bayes rule on the prior p(f) and the
likelihood p(f/e), and then optimizing the a posteriori expected cost function for a
given observation [100].
The choice of the clique potential Vc(f) is crucial as it embeds important prior
information about the image to be reconstructed. The conditional probabilities for
image neighborhood configurations, based on cliques, play a similar role to image
energy in variational approaches. The cliques in MRF encode a set of probabilistic
assumptions (priors) about the geometric properties of the signal, and thus are very
effective when the signal conforms sufficiently well to the prior. In the sequel, we
present how Vc(f) can be appropriately modified to include contextual prior knowledge
in the formulation of the local conditional density.
2.4.1 Gaussian MRF
A generic contextual constraint is smoothness (or continuity) which assumes that the
physical properties in a neighborhood of space or in an interval of time have some
coherence and generally do not change abruptly. For example, the surface of a table
is flat, a meadow presents a texture of grass, and a temporal event does not change
abruptly over a short period of time. Smoothness constraints are often expressed as
the prior probability or equivalently an energy term measuring the extent to which the
smoothness assumption is violated by a random process or field f .
For spatially and temporally continuous MRFs, the smoothness prior often involves
derivatives as in analytical regularization. The order of the derivative n determines
the number of sites involved in the cliques; for example, n = 1 corresponds to a pair-
site smoothness potential (common auto-model). The energy is the integral of the
potential function over the (entire) range of the signal [77].
In the discrete case, where the surface is sampled at discrete points and hence
the label set is also discrete, we use a first-order difference to approximate the first
derivative and use a summation to approximate the integral. Thus, the potential
23
function, considering a 1D signal is Vc(f) = V2(fi, fi′) = 12(fi−fi′)
2. The energy is the
potential summed over all the pair-site cliques i.e.,
U(f) =∑
i∈S
∑i′∈Ni,c∈C2
V2(fi; fi′) =∑
i
∑i′∈Ni
(fi − fi′)2.
In specifying prior clique potentials for piecewise continuous surfaces, only pair-site
clique potentials are normally used. In the simplest case of a flat surface, they can be
defined by [77] V2(fi, fi′) = g(fi − fi′) where g(.) is a function penalizing the violation
of smoothness caused by the difference fi − fi′. For the purpose of restoration, the
function g is generally even (i.e., g(η) = g(−η)) and nondecreasing (i.e., g′(η) ≥ 0).
The random field associated with quadratic smoothness prior is referred to as
Gauss Markov random field (GMRF). An image is usually modeled as a homogeneous
GMRF for simplicity and mathematical tractability. The clique potential function for
the GMRF, as discussed above, is a quadratic function represented as
g(η) = η2 (2.11)
which corresponds to the prior clique potentials in the MRF models. Specifically, for
an image s at site (m, n), the pair-site potential for a GMRF is
Vc(sm,n, sk,l) = g(s(m, n)− s(k, l)) =1
z(s(m, n)− s(k, l))2
where z is a normalizing constant. The advantage of GMRFs is that the potential (Eq.
2.11) is a convex function, and hence it favors global optimization.
Employing pair-site cliques with quadratic potential function, we can formulate
the energy function for GMRF over the sites containing (m, n) in its neighborhood as
η2(s(m, n), s) =1
2ρ2(m,n)
∑
(i,j)∈Nm,n
(s(m, n)− s(m− i, n− j))2. (2.12)
Here, s are (already estimated) neighbors and ρ(m,n) is a weighting parameter based
on the neighborhood. Note that we represent the energy function∑
c∈A Vc(s)(=∑
(k,l)∈Nm,ng(s(m, n) − s(k, l))) with η2(s(m, n), s). Assuming that the image s is
a GRF, we obtain the conditional probability density function (pdf) as
p (s(m, n)/s(m− i, n− j)) =1
(2πρ2(m,n)/mi)1/2
exp(−η2(s(m, n), s)
)(2.13)
24
where (i, j) ∈ Nm,n. The neighborhood becomes Rm,n in a causal recursive framework.
This defines a Gaussian intrinsic auto-regressive (AR) model with conditional mean
given by the mean of the neighboring values and conditional varianceρ2(m,n)
mi, where mi
is the number of neighbors and parameter ρ(m,n) controls the allowable variation of the
current pixel with respect to its neighboring pixel values [40]. In general, smaller the
variance among the neighbors, lower must be the value of ρ(m,n), allowing a prediction
closer to its neighborhood.
The Gaussian MRF model is effective in modeling smooth data, but it penalizes
edges. Since much of the information in an image is contained in edges, arbitrarily pe-
nalizing them is clearly disadvantageous. An image model must favor local smoothness
but at the same time should accommodate abrupt transitions such as edges.
2.4.2 Edge-preserving Priors
A solution to the over-smoothing nature of the GMRF prior is by choosing a more
general contextual prior model through g as
∑
c∈C
Vc(f) =∑
c∈C
g(dcf)
where the differential operator dcf is a local spatial activity measure of the image and
should have a small value in smooth regions and large value at edges.
In the literature, one approach is to detect discontinuities or edges explicitly and
preserve using methods such as line fields or compound GMRF (CGMRF). Geman and
Geman [34] proposed the use of line fields model for preserving edges. The potential
function is a truncated quadratic and is given as
g(η, α) = minη2, α (2.14)
The function in Eq. (2.14) is non-convex (truncated convex). The smoothness con-
straint is switched-off at points where the magnitude of the signal derivative exceeds
threshold√
α thus preserving edge information. But the use of the line field poten-
tial function renders it non-differentiable. An integration of GMRF and line fields
25
−50 0 500
5
10
15
20
25
−50 0 500
1
2
3
4
5
6
7
−50 0 500
1
2
3
4
5
6
−50 0 500
1
2
3
4
5
6
(a) (b)
(c) (d)
Fig. 2.5: Edge-preserving convex potentials. The x and y axes correspond to η andg(η), respectively.
is referred to as compound GMRF (CGMRF) [37]. A CGMRF consists of several
conditional Gauss-Markov sub-models with an underlying line field. Preserving dis-
continuities with this model requires the parameters of the CGMRF to be estimated.
Another approach is to replace the “square law” potentials by more “robust” functions
[100]. Optimization techniques such as simulated annealing with Metropolis-Hasting’s
algorithm or Gibbs sampler which are quite involved may be needed for a reliable
global minimum or maximum [100].
We now briefly discuss some well-known edge-preserving potential functions for
preservation of edges. These can be classified as (discontinuity-preserving) convex
and non-convex potentials. Among the convex ones, Bouman and Sauer [35] define
generalized Gaussian potentials as
g(η) = |η|r (2.15)
26
with r ∈ [1, 2] (r = 2 for GMRF). Figs. 2.5 (a) and (b) show this potential function for
r = 2 i.e., quadratic and r = 1.2, respectively. We observe that there is a significant
difference in the magnitude of the penalty between the two values of r.
Stevenson et al. [104] propose Huber MRF which has the potential function (shown
in Fig. 2.5 (c) for α = 0.6) and is given by
g(η) =
η2, |η| ≤ α
2α|η| − α2, |η| > α(2.16)
The function proposed by Green [105]
g(η) = 2α2 log cosh(η/α) (2.17)
is approximately quadratic for small η and linear for large η. Parameter α controls the
transition between the two behaviors. We depict the nature of this potential for four
different values of the parameter α (from 0.6 (solid) to 0.1 (doted)) in Fig. 2.5 (d).
Non-convex potential functions penalise the edges to a still smaller degree by flat-
tening their shape for large differences. Some of the non-convex potential functions
employed for edge-preservation include the one proposed by Geman and McClure [106]
(shown in Fig. 2.6 (a), for α = 1) and given by
g(η) = η2/(η2 + α2) (2.18)
Blake et. al. [107] suggest a potential function which adaptively switches off penalising
severe discontinuities (as depicted in Fig. 2.6 (b) for α = 2) and is given by
g(η) = (min|η|, α)2 (2.19)
Geman and Reynolds [108] propose
g(η) = |η|/(|η|+ α) (2.20)
for constrained restoration of discontinuities. A typical plot is shown in Fig. 2.6 (c).
Hebert and Leahy [109] propose
g(η) = log(1 + (η/α)2) (2.21)
27
−50 0 500
0.2
0.4
0.6
0.8
1
−50 0 500
1
2
3
4
−50 0 500
0.2
0.4
0.6
0.8
1
−50 0 500
0.5
1
1.5
2
2.5
3
3.5
(a) (b)
(c) (d)
Fig. 2.6: Discontinuity-preserving non-convex potentials. The x and y axes corre-spond to η and g(η), respectively.
which has been employed for 3D Bayesian reconstruction from Poisson noisy data.
The corresponding nonconvex potential function is plotted in Fig. 2.6 (d). Further
references to edge-preserving priors can be found in [36, 39].
2.5 DISCUSSION
A brief review of the terminology, formulation and utility of the Markov random fields
as an image prior was given in this chapter. The theory of MRF provides a convenient
and consistent way of modeling the context-dependent information. Along with the
simple GMRF models, potential functions adaptive to discontinuities were also dis-
cussed. Applications of MRFs to image processing and computer vision problems can
be found in [91, 110].
28
CHAPTER 3
IMPORTANCE SAMPLING KALMAN FILTER FOR
IMAGE ESTIMATION
3.1 INTRODUCTION
In this chapter, we address the problem of image estimation in additive white Gaussian
noise. This can be cast as a problem of state estimation from noisy measurements.
When the state transition and measurement equations are both linear, and the state
and measurement noises are independent and additive Gaussian, the Kalman filter
is optimal and gives the minimum mean square error (MMSE) estimate of the state.
Extension of the 1D recursive KF to images was first proposed by Woods and Rade-
wan [8] and is referred to as the reduced-update Kalman filter. We first discuss the
formulation, abilities and limitations of the traditional auto-regressive Kalman filter
(ARKF). To address the shortcomings of the ARKF, we consider statistical modeling
of the contextual prior and incorporate it within the KF framework.
We propose an interesting extension for handling edges by modeling the original
image with a non-Gaussian MRF prior. The edge preservation capability is implic-
itly incorporated using a discontinuity-adaptive state conditional density. If the state
transition equation is not known but an assumption on the state transition density
(possibly non-Gaussian) can be made, we can still use the Kalman filter update equa-
tions in the proposed framework. We review the discontinuity-adaptive MRF model
proposed by Li [77,78] and employ it to formulate a state conditional prior to specify
the state dynamics. We discuss the importance sampling (IS) technique to estimate
the moments of an arbitrary pdf. The novelty of our approach lies in obtaining the
29
predicted mean and variance of the non-Gaussian state conditional density by im-
portance sampling and incorporating them in the update step of the Kalman filter.
Experimental results demonstrate the effectiveness of the proposed method in filtering
noise while preserving edges.
3.2 AUTO-REGRESSIVE KALMAN FILTER
The observation model for an image degraded by additive noise is given by
y(m, n) = s(m, n) + v(m, n) (3.1)
where s(m, n) is intensity of the original image at location (m, n), y(m, n) is the
observation, and v(m, n) is zero-mean white Gaussian noise with variance σ2v , which
is independent of s(m, n). The problem is to estimate s(m, n) given y(m, n). This is
not straightforward because we need to preserve the discontinuities in s(m, n) while
filtering out the noise.
Typically, the original image is modeled as a 2-D autoregressive (AR) process.
This is due to the fact that in the absence of any a priori constraints, the solution
can be very noisy. The corresponding AR equation [8, 25] can be written as a state
transition equation in the form
s(m, n) =∑
(i,j)∈Rm,n
a(i, j)s(m− i, n− j) + u(m, n) (3.2)
Here, Rm,n is the NSHP model support (which is commonly used) given by Eq. (2.2),
the scalars a(i, j) are the AR model coefficients (assumed stationary) and computed
from a prototype image, and u(m, n) is uncorrelated zero-mean white-Gaussian noise
driving the AR process. The term u(m, n) can also be regarded as the modeling error
between the image and its predicted value. Specifically, we obtain the AR coefficients
a(i, j) by solving the Yule-Walker equations [111]
rk,l −∑
(i,j)∈Rm,n
a(i, j)rk−i,l−j = 0, (k, l) ∈ Rm,n
σ2u = r0,0 −
∑
(i,j)∈Rm,n
a(i, j)ri,j (3.3)
30
where σ2u is variance of the state noise and rk,l is the (k, l)th auto-correlation coefficient
of the original (or its prototype) image computed as
rk,l =1
(M − k)(N − l)
∑
(i,j),(i−k,j−l)∈S
s(i, j)s(i− k, j − l)
For simplicity, we rewrite the AR model (Eq. (3.2)) in matrix-vector form as
sk = Fsk−1 + uk (3.4)
For example, if we consider a first-order NSHP support with three-pixel neighborhood,
then sk = [s(m, n), s(m, n−1), s(m−1, n)]T and sk−1 = [s(m, n−1), s(m−1, n), s(m−1, n−1)]T . Matrix F contains the AR coefficients of the image. If a(0, 1), a(1, 0), a(1, 1)
are the AR coefficients corresponding to the three-pixel NSHP neighborhood and ob-
tained by solving Eq. (3.3), then F = [f1 f2 f3] where f1 = (a(0, 1) 1 0)T ,
f2 = (a(1, 0) 0 1)T and f3 = (a(1, 1) 0 0)T . The vector uk = [u(m, n) 0 0]T
where u(m, n) is process noise and is assumed to be independent and white Gaussian
with zero mean and with covariance defined by σ2u. The 3× 3 covariance matrix Q of
vector uk is formed by augmenting σ2u with zeros.
Based on Eq. (3.1), the measurement equation can be formulated as
yk = Hsk + vk (3.5)
Here, yk = y(m, n) is the given observation at (m, n) and H = [1 0 0]. The measure-
ment noise vk = v(m, n) with covariance matrix R = σ2v . The process and measure-
ment noise are assumed to be uncorrelated. Image estimation boils down to estimating
the state sk given the observation yk.
3.2.1 Filter Equations
For the state-space model given by Eqs. (3.4) and (3.5), the MMSE estimate of state
sk can be derived using the following Kalman filter recursive equations [8, 112].
We initialize the state s0 (with the local mean of the observations) and covariance
matrix P0 (with an identity matrix). Here, k − 1 represents (m, n − 1), sk−1 and
Pk−1 are the posteriori estimates of the state and error covariance of the previous step
available at time k, sk/k−1 and Pk/k−1 are the a priori estimates of the state and error
covariance at time k, yk is the new measurement at time k, Kk is the Kalman gain, and
sk and Pk are the posterior state and error covariance of the present step, respectively.
The above filter is referred to as the Auto-regressive Kalman Filter (ARKF). It
is important to observe that linear dependence implies statistical dependence but not
vice-versa. Our idea is to arrive at a more general framework wherein pixel dependen-
cies can be expressed statistically rather than by imposing a strong linear constraint
as in the AR model.
3.3 DISCONTINUITY-ADAPTIVE MRF
To restore the original image in the presence of noise, it is important to incorporate
as much knowledge as possible about the original image in the estimation process.
The Gauss MRF or AR model, that is commonly used in image restoration, is one
way of imposing smoothness constraint through the state equation to regularize the
solution. A homogeneous and linear AR model compromises on the ability to detect
sharp transitions [28]. In [28, 113], spatially varying 2D AR parameters are estimated
by windowing the observed image. Jeng and Woods [29] propose inhomogeneous AR
image models using local statistics. Kadaba et al. [11, 33] observe that Bernouli-
Laplacian and Cauchy distributions are a better fit for AR model residuals than the
Gaussian. In [37], a compound Gauss Markov random field is used to model images.
Edge-preserving Markov random fields have been proposed in [39].
32
The approach that we adopt to incorporate an edge-preserving MRF prior within
our recursive estimation scheme is based on the fact that statistical dependence in-
corporated using a Markov random field (MRF) model provides better flexibility in
incorporating contextual constraints through a suitably derived prior. Following the
discussions in chapter 2, we propose to employ a discontinuity-adaptive MRF model
to incorporate smoothness constraint while simultaneously preserving the edges.
3.3.1 Condition for DA Potentials
Li [77] proposes a continuous adaptive regularizer model in which, unlike the line field
model, the interaction decreases as the derivative magnitude becomes large and is
completely turned off at infinite magnitude of the signal derivative. A classical way of
finding solutions to illposed problems is based on regularization methods, where stabil-
ity and uniqueness of the solution are enforced by the introduction of prior smoothness
constraints. A more general approach, which includes the classical one as a special
case, is probabilistic regularization which considers the original image and its degraded
version as realizations of random fields. The prior knowledge about the solution is ex-
pressed in the form of a conditional or joint probability distribution that specifies the
desired dependencies among values at neighboring sites.
Without loss of generality, consider a 1-D signal f that is to be estimated. Let
η = f ′(x) and f (n) denote the nth derivative of f . A potential function g(f (n)(x))
quantifies the penalty against the irregularity in f (n−1)(x) and corresponds to prior
clique (a set of connected pixels) potentials in MRF models [77]. The clique potential
is usually chosen to satisfy two properties (i) g(η) = g(|η|), and (ii) the derivative
of g must be expressible as g′(η) = 2ηh(η) where h is a function which determines
the interaction among neighboring pixels. The magnitude |g′(η)| = |2ηh(η)| is the
strength with which the regularizer performs smoothing. A necessary condition for
any regularization model to be adaptive to discontinuities is
limη→∞
|g′(η)| = limη→∞
|2ηh(η)| = A (3.6)
where A ≥ 0 is a constant. The above condition with A = 0 completely prohibits
33
−20 −15 −10 −5 0 5 10 15 200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Signed difference measure, η
Inte
ract
ion
fu
nct
ion
, h
γ(η)
−20 −15 −10 −5 0 5 10 15 200
1
2
3
4
5
6
7
8
9
10
Signed difference measure, η
Str
en
gth
of
DA
−p
ote
ntia
l fu
nct
ion
, g γ(η
)
non−convexconvex region
(a) (b)
Fig. 3.1: DA model. (a) Interaction of neighbors as a function of difference. (b)Penalty imposed by the DA model with increasing difference in intensity values.
smoothing at discontinuities as η →∞ whereas with A > 0 the DA regularizer allows
limited (bounded) smoothing.
3.3.2 DAMRF Prior for Recursive Estimation
The DA models proposed by Li [77] satisfy all the conditions for discontinuity-adaptivity
and are robust to outliers [78,114]. They are parameterized by a free parameter γ and
allow better control over the shape of the potential function [115]. Among the DA
potentials suggested by Li in [77], we use the interaction function hγ(η) =1
1 + (η2/γ)where η is as defined before. The corresponding potential function takes the form
gγ(η) = γ log(1 + (η2/γ)) (3.7)
Fig. 3.1 (a), shows how the function hγ(η) specifies interaction based on the difference
value η while Fig. 3.1 (b) brings out the behavior of the corresponding potential func-
tion gγ(η). For this choice of hγ(η), the smoothing strength |ηhγ(η)| increases mono-
tonically as η increases within a band Bγ = (−√γ,√
γ). Outside the band, smoothing
decreases and becomes zero as η→∞. Since this enables to preserve image discon-
34
0 0.5 1 1.5 2 2.5 3 3.5
x 104
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Squared difference measure, η2
Sm
oo
thin
g s
tre
ng
th o
f D
AM
RF
, | 2
η h
γ(η)
|
50 100 150 200 2500
0.01
0.02
0.03
0.04
0.05
0.06
(30, 50)(80, 30)(160, 180)
Pixel intensity
DA
MR
F v
alu
es
(a) (b)
Fig. 3.2: (a) Plot shows how the smoothing strength of DAMRF varies with η2. (b)Heavy-tailed DAMRF distributions.
tinuities, it is also called a discontinuity-adaptive MRF (DAMRF). We observe that
the variation from the convex (smoothing) region to non-convex (i.e., lesser increment
in penalty to distinct pixels) is gradual in Fig. 3.1 (b). This choice of DA-potential
function is based on our observation that its performance is superior to other DA-
potentials proposed by Li [77, 78]. Fig. 3.2 (a) shows how the smoothing strength of
the DAMRF varies as a function of η2.
When neighborhood interaction in s is defined as above, the irregularities in the
solution are penalized by the corresponding potential function, and the conditional
pdf takes the form
p (s(m, n)/s(m− i, n− j)) =1
Zexp
(−γ log
(1 +
η2(s(m, n), s)
γ
))(3.8)
where Z is a normalization constant and the energy term
η2(s(m, n), s) =1
ρ2(m,n)
∑
(i,j)∈Rm,n
(s(m, n)− s(m− i, n− j))2
(i2 + j2)(3.9)
Here, (i, j) ∈ Rm,n where Rm,n is the NSHP support of order M1. Thus, s are the
NSHP neighbors of s(m, n). Parameter ρ2(m,n) controls the allowable variation of the
35
current pixel with respect to its neighboring pixel values [40] and plays an important
role in adaptively modifying the prior distribution. Since the potential function used is
discontinuity-adaptive, this local conditional prior is expected to render better edge-
preserving capability compared to the quadratic GMRF regularizer. Fig. 3.2 (b)
illustrates the non-Gaussian nature of DAMRF distributions. The pdf in Eq. (3.8)
can be rewritten as
p (s(m, n)/s(m− i, n− j)) =1
Z
(1 +
η2(s(m, n), s)
γ
)−γ
(3.10)
which has a maximum at
∑αij s(m− i, n− j)∑
αijwhere αij = 1/(i2 + j2) are the corre-
sponding weights and (i, j) ∈ Rm,n. Further, the pdf p is symmetric about the maxi-
mum (due to the quadratic nature of the argument η2). Thus, the mean value of the
pdf p(s(m, n)/s) is given by
E(p(s(m, n)/s)) =
∑i,j αij s(m− i, n− j)
∑i,j αij
. (3.11)
3.4 PRINCIPLE OF IMPORTANCE SAMPLING
In order to incorporate the DAMRF prior (formulated in section 3.3.2), into the
Kalman filter framework, we need to estimate the mean and covariance of this non-
Gaussian prior (in each recursive prediction step). Since it is analytically not possible
to compute the variance of this pdf, we resort to a Monte Carlo method known as
importance sampling.
Monte Carlo (MC) is the art of approximating an expectation by the sample mean
of a function of simulated random variables [80]. Since the simulations are with random
numbers, the more simulations we perform, the more accurate the approximation
becomes [116]. However, closer the approximation by a specific MC method, fewer
samples are needed to reach a certain level of accuracy. With appropriate MC methods,
even when using random sampling, it is possible to reduce the (error) variance for a
given number of sample points. This is based on the intuition that the efficiency of
random sampling methods can be enormously enhanced by a suitable choice of sample
36
points. For example, to estimate moments under a Gaussian distribution with random
samples, we need to sample more frequently around the mean of the Gaussian pdf
than employ uniform sampling. The MC procedure that enables us to perform such
tasks systematically is known as importance sampling.
Importance sampling (IS) is an MC method for determining the estimates under
a (difficult to sample) target function or pdf, provided its functional form is known
up to a multiplication constant [79]. The idea behind IS is that certain values of
the input random variables have more impact on the parameter being estimated than
others. If these “important” values are emphasized by sampling more frequently, then
the estimator error variance can be reduced [80]. Hence, the basic methodology is to
choose a distribution for generating random samples, which ‘encourages’ important
values. The way it does this is by bringing in a probability distribution function
[116] which is a function that attempts to tell which interval of the target function
should get more samples. It does this by having a higher probability in that area.
But this can result in a biased estimator. Hence, the outputs are corrected using
correction weights for each sample to ensure unbiasedness [116, 117]. The selection of
the sampling distribution known as importance function requires prior knowledge of
the target pdf (or function).
Let us consider a pdf p(z) from which it is very difficult to make any estimates of its
moments. From the functional form, we can estimate its non-zero support. Consider
a distribution q(z), a reasonable approximation to p(z), which is also known up to a
multiplicative constant, is easy to sample, and is such that the (non-zero) support of
q(z) includes the support of p(z). Such a density q(z) is called the sampler density
(referred to as importance function).
Mathematically, the idea of importance sampling can be presented as follows.
Suppose the density q(z) roughly approximates the target density p(z), with (non-
zero) support A (i.e.,∫
z∈Aq(z)dz = 1). The expected value of a function g under pdf
p(z) can be written as
Ep[g(z)] =
∫g(z)p(z)dz =
∫g(z)
(p(z)
q(z)
)q(z)dz = Eq
[g(z)
(p(z)
q(z)
)](3.12)
37
−20 −10 0 10 20 30 400
0.01
0.02
0.03
0.04
0.05
0.06
0.07Rayleigh (10)Gamma (2, 4)
Support
pd
f va
lue
s
0 5 10 15 20 25 30 35 400
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
q: Rayleigh (4) p: Gamma (2,4)
Support
pd
f va
lue
s
(a) (b)
Fig. 3.3: Choice of importance function. (a) Good choice: The support of the targetpdf is included in the sampler. (b) Bad choice: The support of the same target pdf isnot included in the sampler.
Using the above identity and IS with the samples of q, Ep[g(z)] is estimated as
∫g(z)p(z)dz ≈ 1
L
L∑
i=1
g(z(l))
(p(z(l))
q(z(l))
). (3.13)
where L samples z(l) are drawn from the distribution given by q(z). The (dis-
tribution) ratio p(z)/q(z) is known as the likelihood ratio [118], since it quantifies
the relative likelihood of a given sample z(l) under p compared to q. We note that
each sample drawn from q is weighted by the likelihood (or the correction weight) for
counter-balancing [116].
It is important to note that the success of this method in getting accurate estimates
is entirely dependent on selecting a good importance function q(z). Fig. 3.3 shows
good and bad q for a given p. Even when q(z) is roughly the same shape as p(z),
serious difficulties arise if q(z) decays-out faster than p(z) at the tails (for example,
as shown in Fig. 3.3 (b)). In such cases, the improbable samples at the tails of q
are given much higher-orders of the correction weight than usual making the estimate
biased [80]. For further information about the choice of importance function, refer to
38
0 10 20 30 40 50 60 70 800
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0.04Cauchy (2,1)Rayleigh (2)
Support
pd
f va
lue
s
−20 −10 0 10 20 30 400
0.01
0.02
0.03
0.04
0.05
0.06
0.07Rayleigh (10)Gaussian (5, 40)
Support
pd
f va
lue
s
(a) (b)
Fig. 3.4: Choice of importance function. (a) Good choice: Sampler support includes(non-symmetric) target pdf (b) Bad choice: Non-symmetric sampler cannot be em-ployed for IS of a symmetric target distribution.
[118, 119]. Fig. 3.4 illustrates the limitation of a non-symmetric sampler1.
3.4.1 Moment Estimation
Our aim is to determine the estimates under the pdf p, using samples from q. Since it
is difficult to draw samples from the target pdf p, we draw L samples, z(l)Ll=1 from
the sampler pdf q, i.e., they are chosen from a distribution q which concentrates in
the region where the function p is large. If these were under p, we can determine the
moments of p directly from them. In order to use these samples to determine the
estimates of the moments of p, we proceed as follows. When we use samples from
q to determine any estimates under p, in the regions where q is greater than p, the
estimates are over-represented. In the regions where q is less than p, they are under-
represented. To account for this, we use correction weights wl =p(z(l))
q(z(l))in determining
the estimates under p [see Eq. (3.13)]. While estimating the moments, the correction
weights must be normalized, if target p and sampler q (densities) are known only upto
1In Fig. 3.4 (a), to estimate the moments under a non-symmetric distribution using the symmetric
sampler, explicit care must be taken to use zero values of target pdf for the negative samples.
39
Table 3.1: Moment estimation using IS with a proper choice of importance function.
Target pdf Sampler density True IS (µp) True IS (σ2p)
Fig. 4.3: Posterior samples (a) Monte Carlo, and (b) UT.
relating z = [x; y] with r and θ and ∆z =
[δ
δx,
δ
δy
].
With the three methods, the calculated posterior moments in the y−direction are
given as follows:
MC: posterior mean of y = 0.9672 and covariance Pyy =
0.0631 0
0 0.0260
.
UT: posterior mean of y = 0.9667 and covariance Pyy =
0.0625 0
0 0.0380
.
Linearization: mean of y = 1.0000 and covariance Pyy =
0.0685 0
0 0.0004
.
Figure 4.4 shows the mean and 1 − σ contours for each of these methods. The
1− σ contour is the locus of points y|(y − y)P−1yy (y − y) = 1 and is a graphical rep-
resentation of the size and orientation of Pyy [130,131]. As can be seen, the linearized
transformation is biased and inconsistent. This is most severe in the range direction,
where linearization estimates that the position is 1.00m whereas in reality it is 0.967m.
Since it is a bias which arises from the transformation process itself, the same error
with the same sign will be committed each time a coordinate transformation takes
63
−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.30.9
0.92
0.94
0.96
0.98
1
1.02
1.04
Linearization
MC
UT
y
x
Fig. 4.4: Figure shows the posterior mean and uncertainty (1-σ) contour of covariance,determined by MC approach, i.e., true values (mean at ’*’), linearization (mean at ’+’),and unscented transformation (mean almost matches with that of MC mean).
place. Even if there were no bias, the transformation is inconsistent, since the 1 − σ
contour of covariance obtained from linearization is significantly different from the MC
covariance. In comparison, the UT mean (96.67) is quite close to the MC estimate, and
the UT 1− σ covariance contour is very similar to that of MC propagation. We note
that only 5 sigma points are required in UT as against 105 samples for MC method
which works based on the laws of large numbers (to propagate the whole pdf).
As observed, linearization predicts the posterior moments of a nonlinearly trans-
formed random variable by truncating their (posterior) Taylor series expansion after
the first two terms for the mean and after the first term for the covariance [81, 112].
This can introduce significant errors in prediction for many frequently occurring non-
linearities [84, 132]. When the first two moments required for the Kalman filter are
predicted by linearization, it leads to the most well-known application of the KF frame-
work to nonlinear systems, known as the extended Kalman filter (EKF). Even though
64
the EKF is one of the most widely used approximate solutions for nonlinear estimation
and filtering, it has serious limitations [81, 122].
On the other hand, predicting the posterior moments with Monte Carlo methods
requires a very large (for example, of the order of 103 − 105) number of random sam-
ples (generated from the prior distribution) in an attempt to accurately propagate the
probability density function through the nonlinear models. The Monte Carlo approx-
imation of the optimal Bayesian estimator [41, 83], known as the particle filter, also
provides a solution (that is more accurate than the linearization used in EKF) to the
nonlinear estimation problem [59,133]. However, it is computationally very intensive.
4.4 UT-BASED EXTENSION OF THE KALMAN FILTER
Julier and Ulhmann [81] investigated a reliable nonlinear extension of the Kalman
filter and came up with UT to predict the means and covariances arising from nonlin-
ear transformations, which serve as sufficient statistics in a Kalman framework. They
demonstrated superior performance of the new filter, referred to as the unscented
Kalman filter (UKF), over the widely used EKF on several highly nonlinear 1D es-
timation problems [81, 132, 134]. Later, Wan and Ven der Merwe applied the UKF
to nonlinear parameter estimation [84] and dual (simultaneous state and parameter)
estimation [129]. Its utility as a smoother was demonstrated in the ’Expectation step’
of the EM algorithm for efficient training of neural networks, and then for the subse-
quent design of a dual unscented filter. Ven der Merwe et al. [135] further developed
a computationally efficient and numerically robust square root implementation of the
UKF. The unscented transformation with reduced number of sigma points is proposed
in [136] and its use has been extended to more accurately estimate the means and
covariances in [137, 138]. The sigma point approaches such as the UKF and the CDF
[127] are classified in a unified general family of derivative-free Kalman filters for non-
linear estimation in [128]. The use of UKF and its Gaussian mixtures as a proposal
function in particle filters is reported in [42, 83] and [122,139], respectively.
65
A detailed discussion on the utilities of UT including its application to discontin-
uous transformations and multi sensor fusion is described in [125]. Since its invention,
it has been applied as a more accurate, simple, and stable replacement to the widely
used Extended Kalman filter in diverse applications [85, 140]. The similarity of the
UKF with the statistical linear regression based Kalman filter (LRKF) is discussed
in [141] to provide useful insights into the properties of the UKF. The UKF is also
analysed from a Bayesian perspective in [142] and is suggested as a better alternative
to EKF and even to Monte Carlo filters for simple models. In vision, its application
to visual tracking applications has been explored in [86, 87, 143].
The UKF inherits the Kalman filter structure; however, it employs the UT to
propagate the first two moments of the state random variable through non-linear state
and measurement transformations [81, 83, 121]. As in the (extended) Kalman filter,
the state distribution is approximated by a Gaussian random variable (GRV) but is
now represented using a minimal set of carefully chosen sigma points. These sample
points completely capture the true mean and covariance of the prior GRV, and, when
propagated through the true nonlinear system, capture the posterior mean and covari-
ance accurately to second order (third order for symmetric priors) in the Taylor series
expansion for any nonlinearity [81].
State estimation in a general recursive framework employs a nonlinear/non-additive
state space model to describe the dynamics of the system. The evolution of the state
(the quantity to be estimated) is described as a state equation
xk = f(xk−1,uk) (4.26)
and the degradation that relates the observation to the state is modeled as the obser-
vation equation
yk = h(xk,vk) (4.27)
where uk is state noise and vk is the noise in the observations.
The UKF results from applying the UT to recursive minimum mean-square error
(MMSE) estimation [121, 122]. Kalman derived [7] the recursive form of the linear
66
Bayesian update of state conditional mean, xk = E[xk/y1,y2, ...,yk] and its covari-
ance, Pk = E[(xk − xk/k−1)(xk − xk/k−1)T ] as
xk =(Prediction of xk) + (Gain)k × [ yk − (Prediction of yk)]
While this is a linear recursion, we need not assume linearity of the model. For MMSE,
considerably less noisy than MWF. Fig. 5.3(e) shows the output of the ARUKF. It
is marginally sharper than the PF output and has a higher ISNR value. The image
estimated using ISUKF is presented in Fig. 5.3(f). The noise level is quite low while the
petals come out sharp and crisp demonstrating the effectiveness of the non-Gaussian
prior incorporated in the ISUKF. The image is much closer to the original and has the
highest ISNR value among all these methods.
Fig. 5.4(a) shows a ‘house’ image. The image degraded by film-grain noise is
shown in Fig. 5.4(b). The images estimated by MWF, PF, ARUKF and ISUKF are
shown in Figs. 5.4(c), (d), (e) and (f), respectively. The ARUKF preserves sharp
details (such as grass and tree leaves) akin to MWF but is less noisy. The performance
82
0 5 10 15 20 251
2
3
4
5
6
7
8ISUKFARUKFPF MWF
ISN
R v
alu
es
of
resp
ect
ive
filt
ers
Image index 0 5 10 15 20 25
2
3
4
5
6
7
8
9ISUKFARUKFPFMWF
Image index
ISN
R v
alu
es
of
resp
ect
ive
filt
ers
(a) (b)
Fig. 5.6: Performance comparison on different images in terms of mean value of ISNRover 20 MC runs (a) at moderate noise (σ2
v = 0.05), and (b) at high noise (σ2v = 0.15).
of PF is comparable to that of ARUKF. However, the image estimated by ISUKF is
the best among all. The noise is effectively filtered while preserving fine details such
as window-panes, grass and leaves. This is also reflected in its high ISNR value.
Fig. 5.5(a) shows the ‘peppers’ image. It is severely degraded by film-grain noise
with variance σ2v = 0.1 as shown in Fig. 5.5(b). The image recovered by MWF is
given in Fig. 5.5(c). By applying the ARUKF, we achieve good noise reduction (Fig.
5.5(e)) and obtain a result that is visually comparable to that of PF (Fig. 5.5(d)). The
image estimated using ISUKF is presented in Fig. 5.5(f). We note that the output
of ISUKF is almost free from noise while the contours remain sharp. It also has the
highest ISNR value among all the methods.
In Figs. 5.6(a) and (b), we statistically compare the performance of proposed
ARUKF and ISUKF with MWF and PF with prior boosting [146] on a wide variety
of images including face images, natural scenes, film images and textures, both at
moderate noise and high noise. Except on very few images (highly textured) ISUKF
outperforms all other filters at all noise levels. The ARUKF quantitatively precedes
ISUKF in performance. At low noise (say σ2v < 0.04) PF results in poor performance
83
(a) (b)
(c) (d) (e)
Fig. 5.7: (a) Cropped portion of a frame from the movie ‘Das testament des Dr.Mabuse’. Output of (b) MWF, (c) PF, (d) ARUKF, and (e) ISUKF.
because the likelihood is very peaked. The performance of ARUKF is better than
MWF and comparable to PF. But the ISUKF with a DAMRF non-Gaussian prior
performs the best among all the filters. It is also computationally efficient than PF.
5.5.2 Real Examples
Fig. 5.7(a) shows a cropped portion of an image frame from the old classic movie ‘Das
testament des Dr. Mabuse’. The film-grain noise is real and shows up clearly on the
white coat, the face etc. Using a homogeneous region in the frame, the measurement
noise variance was found to be 0.04. The output of MWF, PF and the proposed
ARUKF, and ISUKF are shown in Figs. 5.7(b), (c), (d) and (e), respectively. We
observe that ARUKF performs on par with PF but the ISUKF is the most effective
84
(a) (b)
(c) (d) (e)
Fig. 5.8: (a) Face image with real film-grain noise. Image estimated using (b) MWF,(c) PF, (d) ARUKF, and (e) ISUKF.
even in real situations. The overall noise level is quite low and the folds on the cloth
become quite discernible in the output of the ISUKF.
Next, a face image with film-grain noise is shown in Fig. 5.8(a). Residual film-
grain noise is visible in the output of MWF (Fig. 5.8(b)). The PF result (Fig. 5.8(c))
is comparatively better. The result using ARUKF (Fig. 5.8(d)) is marginally sharper
than PF. The output of ISUKF (Fig. 5.8(e)) has all edges intact, and is least affected
by grains. Finer details such as lips and eye-brows are well-preserved. The image
obtained using ISUKF is visually striking in appearance and appears the most natural
among them all.
Fig. 5.9(a) shows yet another real image with film-grain noise. Figs. 5.9(b), (c),
85
(a) (b)
(c) (d) (e)
Fig. 5.9: (a) A real building image. Output image obtained using (b) MWF, (c) PF,(d) ARUKF, and (e) ISUKF.
(d) and (e) depict the output from MWF, PF, ARUKF and ISUKF, respectively. The
result obtained using ARUKF is comparable to PF. The ISUKF output is again the
best. The edges have been preserved well and the grainy appearance in the original
degraded image of Fig. 5.9(a) is greatly mitigated.
Finally, we consider the locomotive image shown in Fig. 5.10 (a). The result
obtained using the PF with 500 samples is shown in Fig. 5.10 (b). We note that
inspite of some blurring of the letters and edges, the PF could not effectively remove
the noise in the white and black homogeneous regions. The image estimated by the
ARUKF (Fig. 5.10 (c)) is more effective in removing noise but it tends to blur details.
However, the output of the ISUKF, shown in Fig. 5.10 (d), preserves the sharpness of
86
(a) (b)
(c) (d)
Fig. 5.10: Cropped portion of a locomotive captured with a film camera. (a) Originalwith real film-grain noise. Image estimated using (b) PF, (c) ARUKF, and (d) ISUKF.
the details very well (for example, the Tamil letters) and has very little noise.
Unlike the particle filter, in which a large number of samples must be propagated
throughout the algorithm in an attempt to faithfully represent the posterior distribu-
87
Table 5.1: Computational complexity comparison at each pixel.
Operation PF (S = 200) ARUKF (na = 5) ISUKF (L = 100, na = 2)
5. Using the predicted state and measurement sigma points, and the UT determin-
istic weights, we predict the mean and covariance of state and measurements
(x,Px, y,Py) and also the cross-covariance of the state and measurement Pxy.
6. Based on the state and measurement statistics and using the recent observa-
tion y(m, n), we employ the Kalman filter update to estimate the mean x and
covariance Px of the state.
The estimated state x(m, n) corresponds to the despeckled pixel.
1These are scalars as described in section 5.3.1. We have dropped the subscripts referring to pixel
location, for brevity.
96
(a) (b) (c) (d) (e)
Fig. 6.1: (a) Original image. (b) Noisy version. Output of (c) Enhanced Lee(FOM = 0.92, S/MSE = 20.51 dB), (d) Frost (FOM = 0.94, S/MSE = 20.51dB), and (e) the proposed method (FOM = 0.98, S/MSE = 22.83 dB).
We note that the propagation of the variance parameter ρ2(m,n) of the DAMRF
prior for SAR is different from that for film-grain noise in which ρ2(m,n) was directly
set to the previous covariance estimate. In this chapter, the variance parameter ρ2(m,n)
(in Eq. 3.9) is propagated only based on neighboring estimates and tuned with a free
scaling parameter ζ to effectively handle the wide variety of contextual features in
SAR images. The ISUKF leads to significantly better performance that the AR-based
UKF both in terms of (local) smoothing of uniform regions and in preserving point
and edge features.
6.4 EXPERIMENTAL RESULTS
In this section, we present results obtained using the proposed ISUKF method when
applied on SAR images. We compare the performance of the proposed approach
with both standard and recent methods. We note that in the proposed approach,
the parameter ζ of the DA model is image-dependent. Based on our experiments, it
typically takes a value between 0.01 and 0.03 for moderate noise.
Consider the edge image shown in Fig. 6.1(a). This image is degraded with
simulated 2-Look Gamma-distributed speckle noise (Fig. 6.1(b)). The outputs of the
Enhanced Lee [65], Frost [62] and the proposed ISUKF method are shown in Figs.
6.1(c), (d) and (e), respectively. The images in Figs. 6.1 (c) and (d) have a grainy
appearance in the white uniform region due to residual speckle. The proposed method
97
(a) (b) (c)
(d) (e) (f)
Fig. 6.2: (a) Original SAR image. (b) Degraded (σ2v = 0.04). Image estimated using
(c) the Enhanced Lee filter, (d) the Frost filter, (e) AR-based UKF, and (f) the ISUKF.
is not only effective in despeckling but also well preserves the transition between the
dark and bright regions. Our method yields an almost exact edge map (close to that of
the original) even though the noise level is high. This is also reflected in the S/MSE
and FOM values which are higher as compared to those of the standard filters (as
given in Table. 6.1 (a)).
The image used for the next experiment is shown in Fig. 6.2(a). The degraded
image with simulated speckle noise of variance σ2v = 0.04 is shown in Fig. 6.2(b). The
despeckled images obtained with the Enhanced Lee filter and the Frost filter are shown
in Figs. 6.2(c) and (d), respectively. We observe that dark regions are over-smoothed
while the high intensity regions have a grainy appearance. The image obtained using
the ARUKF is shown in Fig. 6.2(e). The output is visually inferior to that of the Frost
filter but it does not over-smooth dark regions. The output of the proposed ISUKF is
98
Table 6.1: Quantitative comparison with standard filters.
(a) Standard Filters (Fig. 6.1)
SMSE
dB FOM ENL
Enhanced Lee (K = 1) 20.51 0.92 305
Frost (K = 3) 20.51 0.94 361.15
ISUKF (ζ = 0.01) 22.83 0.98 3071.00
(b) Standard filters (Fig. 6.2)
SMSE
dB ECF ENL
Noisy (at (170, 160)) 0 0.242 24.90
Enhanced Lee (K = 8) 22.28 0.217 459.06
Frost (K = 12) 23.35 0.199 559.18
ARUKF 21.38 0.639 270.00
ISUKF (ζ = 0.025) 22.15 0.624 1041.00
shown in Fig. 6.2(f). Note that incorporating a non-Gaussian MRF prior in the UKF
gives a significant improvement over the ARUKF. The image in Fig. 6.2(f) is visually
closest to the original. Comparative metrics for this example are given in Table 6.1(b).
We also compared our method with the adaptive MAP technique in [12] which is
based on a heavy-tailed Rayleigh model. In Figs. 6.3 (a), (b) and (c), we reproduce
the original, degraded and the output image from [12], respectively. Even though the
method proposed in [12] reduces speckle, the output is blurred. The image estimated
using the proposed ISUKF approach (Fig. 6.3(d)) when used on the same degraded
image is sharp and even the fine details are recovered well. The output of our method
has higher S/MSE and ECF values as given in Table 6.2(a). Because the output
of [12] is blurred, its ENL value is marginally higher. Note that the output of the
proposed method is much closer to the original image (texture-wise also).
Next, we considered the case of a Horse track image that is affected by real speckle
noise (Fig. 6.4(a)). The output of the Frost filter, shown in Fig. 6.4(b), contains
noticeable residual speckle in uniform regions. Also, the edge boundaries are noisy.
99
(a) (b) (c) (d)
Fig. 6.3: (a) Original aerial image. (b) Degraded image. Image estimated using (c)Rayleigh prior MAP estimator [12], and (d) the proposed ISUKF.
(a) (b) (c)
Fig. 6.4: (a) The Horse track image. Estimated output image using (b) the Frostfilter, and (c) our method.
The image estimated by the ISUKF method is shown in Fig. 6.4(c). Our method
is significantly more effective in smoothing speckle over uniform regions (such as the
top-left and bottom-right gray regions); yet the edges are sharp and clear. Even the
small white blobs on the top-right corner are recovered well. Table 6.2(b) gives a
quantitative comparison with other filters. The cropped 60× 60 region that was used
to calculate the ENL is centered at (115, 55). The performance of our method is
evidently superior even in the real case.
Figure 6.5(a) shows the Bedfordshire image (in Southeast) England with real
speckle. The image estimated using a Bayesian wavelet filter which relies on edge
information is reproduced from [13] and is given in Fig. 6.5(b). The output obtained
100
Table 6.2: Quantitative comparison.
(a) Rayleigh MAP-based (Fig. 6.3)
SMSE
dB ECF ENL
Noisy (at (165, 100)) 0 0.67 13.44
Rayleigh MAP technique [12] 18.57 0.61 17.26
ISUKF (ζ = 0.03) 21.14 0.74 16.42
(b) Real speckle (Fig. 6.4)
µf σf ENL
Noisy (at (115, 55)) 110.96 28.73 14.92
Frost (K = 10) 110.46 8.92 153.10
Enhanced Lee (K = 6) 110.47 9.56 133.66
ISUKF (ζ = 0.01) 112.18 7.88 202.49
(a) (b) (c)
Fig. 6.5: (a) Bedfordshire image. Result using (b) edge-based Bayesian wavelet filter[13], and (c) the ISUKF method.
by the ISUKF filter on the same degraded image of Fig. 6.5(a) is shown in Fig. 6.5(c).
We note that the edges are recovered without any blurring effects while the homoge-
neous regions are almost free from noise. The isolated point targets (two white spots
in the dark homogeneous region on the left) become strikingly visible in Fig. 6.5(c).
A quantitative comparison of the two methods is given in Table 6.3(a).
101
Table 6.3: Quantitative comparison with wavelet filters.
(a) Wavelet edge-based (Fig. 6.5)
µf σf ENL
Noisy (at (90, 160)) 89.73 25.73 12.16
Bayesian wavelet filter [13] 90.85 13.87 42.87
ISUKF (ζ = 0.015) 89.25 13.03 46.91
(b) UDWL-MAP (Fig. 6.6)
µf σf ENL
Noisy (at (280, 100)) 52.10 8.73 35.61
UDWL [14] 54.06 3.04 316.06
ISUKF (ζ = 0.004) 51.96 2.83 337.9
(a) (b) (c)
Fig. 6.6: (a) Airport image. Result using (b) MAP estimator in UDWL domain [14],and (c) the proposed method.
We also compared our method with a very recent wavelet despeckling technique
[14] developed in the undecimated wavelet (UDWL) domain. Fig. 6.6(a) shows a real
SAR image of an airport. This is a very difficult example as it contains lots of weak
edges. The outputs of the method in [14] and the proposed method are shown in Figs.
6.6(b) and (c), respectively. From a visual comparison, it is quite evident that the
ability of the ISUKF method in capturing soft edges is superior to that of [14]. Even
102
(a) (b)
(c) (d)
Fig. 6.7: (a) Urban SAR image. (b) Degraded image. Image estimated using (c)wavelet-based filter [15], and (d) ISUKF.
the ENL for our method is higher (Table 6.3 (b)).
We next present few more results, only for visual comparison with some well known
existing methods, to demonstrate the effectiveness of the proposed ISUKF method on
different kinds of SAR images. Figs. 6.7(a), (b) and (c) show the original, degraded
and the image estimated using the wavelet filter with soft-thresholding (from [15]),
respectively. The image obtained by using the ISUKF is shown in Fig. 6.7(d). We
observe that the proposed method is more effective in bringing out the narrow sharp
details in the bottom middle regions while effectively removing the overall speckle.
Next, we compare the proposed method with an MRF-based approach which ob-
tains an estimate by simulated annealing with Metropolis sampler [16]. Figs. 6.8 (a)
103
(a) (b) (c)
Fig. 6.8: (a) Real SAR image. Image estimated using (b) simulated annealing andMetropolis estimator [16], and (c) ISUKF.
and (b) show the real noisy SAR image and the output of the method in [16], respec-
tively. The image estimated using the proposed ISUKF is shown in Fig. 6.8(c). Our
approach could effectively suppress speckle while preserving image texture and sharp
details.
The above examples clearly demonstrate the effectiveness of the proposed ISUKF
in suppressing speckle while simultaneously preserving point and fine features. It
generally leads to a better visual output and also compares favorably in quantitative
performance as well as in computational complexity with other methods. It has the
same computational complexity as given in chapter 5 (Table (5.1)), and executes in 18
seconds using Matlab on a Pentium 4 PC with 256 MB RAM for a 200 x 200 image.
6.5 DISCUSSION
In this chapter, we explored the applicability of ISUKF for despeckling SAR imagery.
The formulation of the DAMRF prior was modified to preserve fine features in a wide
range of SAR images. A small set of sigma points were used to capture and propagate
the first two moments of the state and measurement noise through the multiplicative
speckle model.
104
CHAPTER 7
JOINT INPAINTING AND DENOISING
Old films or photographs usually suffer from damages due to physical and/or chemical
effects resulting in stain, scratch, scribbling, noise, and digital drop-out in frames. In-
painting is a technique for modifying user-specified regions of an image undetectably.
It provides a means for reconstruction of known damaged portions. It serves a wide
range of applications, such as removing superimposed text like dates, subtitles, public-
ity or logos from still images or videos, and reconstructing scans of deteriorated images
by removing scratches or stains [6, 18].
In this chapter, we present a scheme which can simultaneously filter film-grain
noise while filling-in identified damaged portions in an image within the unscented
Kalman filter (UKF) framework. We observe that a key issue in handling inpainting
with a recursive filter lies in arriving at the observations in the regions to be inpainted
based on the surrounding available information. Care must be taken to preserve the
edges through the inpainting regions. We first demonstrate the validity of our scheme
on line and region-scratches, and text removal. We then incorporate our inpainting
scheme within the UKF framework and demonstrate simultaneous film-grain noise
reduction and inpainting capability of the proposed filter.
7.1 INTRODUCTION
Most inpainting techniques smoothly propagate image information from outside the
boundary into the inpainting region based on edges or isophotes (lines of equal gray
values) [6] and require the user to only specify the region to be inpainted. Kokaram
et al. [23] interpolate losses in films from adjacent frames using motion estimation and
auto-regressive models. The technique, however, cannot be applied to still images or to
105
films where the regions to be inpainted span several frames. Hirani and Totsuka [155]
combine global frequency and local spatial information in order to fill a given region
with a selected texture. The algorithm requires the user to select the texture to be
copied into the region to be inpainted. Mansnou and Morel [24] define dis-occlusion as
the recovery of hidden parts of objects in an image by interpolation from the vicinity
of the occluded area and perform inpainting by joining the points of the isophotes at
the boundary of the region to be inpainted.
Bertalmio et al. [6] formulate partial differential equations (PDEs) that smoothly
propagate the boundary information (the Laplacian of the image) in the direction of
the isophotes, estimated by the image gradient rotated by 90 degrees. Their algorithm
shows that both the gradient direction (geometry) and the gray-scale values (pho-
tometry) of the image should be propagated inside the region to be filled-in. They
demonstrate the generality of their method with various applications including region-
filling, text-removal, and special-effects restoration of old photographs. In [156], an
exemplar-based inpainting algorithm for filling-in large objects is proposed by com-
bining the principles of texture-synthesis and traditional inpainting. Another class
of inpainting algorithms which stresses on exploiting geometric image models in a
Bayesian framework has been proposed by Chan and Shen in [18,157]. The first model
introduced by them uses a total variation-based image model [158]. This model can
successfully propagate sharp edges into the damaged domain. However, because of a
regularization term, the model exacts a penalty on the length of edges, and, thus, the
inpainting model cannot connect contours across very large distances. Subsequently,
Chan et al. [159] introduced the Mumford-Shaw model that allows both for isophotes
to be connected across large distances, and their directions to be kept continuous across
edges in the inpainting region.
Oliveira et al. [160] have proposed a fast and simple inpainting method which
repeatedly convolves a Gaussian 3 × 3 filter over the missing regions to diffuse in-
formation. However, it requires specification of the diffusion barriers (high-gradient
areas) manually. Telea [161] uses intensity gradient and distance information of the
106
boundary pixels and weighs them by the isophote direction. Yet another inpaint-
ing algorithm [162] uses the Sobel edge operator’s magnitude and angle to compute
isophotes. An inpainting method based on belief propagation is proposed in [163].
An exemplar-based image completion method that unifies texture synthesis and image
inpainting with a priority belief propagation-based optimization is proposed in [164].
Rares et al. [17, 165] propose inpainting methods which use edge information both
for the reconstruction of the skeleton image structure in missing areas, as well as for
guiding the interpolation that follows.
In [166], a PDE model based on the Cahn-Hilliard equation [167] is proposed for
binary inpainting of degraded text. Image repairing methods in [168, 169] employing
texture segmentation followed by robust, non-iterative multi-dimensional tensor voting
[170] to globally infer the most suitable pixel value in a neighborhood by using the
MRF assumption. In recent years, there is great interest in extensions such as video
inpainting [171], video repair [172], video stabilization [173] and deinterlacing of video
sequences [174].
A very recent development in inpainting is simultaneous recovery from noise as well
as damages. In [175], a PDE-based method is proposed to fill-in missing information
while removing noise. Inside the inpainting domain, smoothing operation is carried
out by mean-curvature-flow to enable transportation of boundary information, while
outside of the inpainting domain smoothing is encouraged within homogeneous regions
and discouraged across boundaries. In this chapter, we first propose an inpainting
method that is guided by edge information. We next embed noise filtering into the
inpainting framework.
7.2 AN EDGE-BASED APPROACH
Since image information between different objects generally need not correlate and
are separated by edges, edge-based inpainting methods first reconstruct explicit edge
information in the region of damages. Filling-in is followed within each object guided
107
Fig. 7.1: A typical edge-based inpainting methodology (from [17]): (Left) Generalalgorithm outline and, (right) an illustration of the outputs for each stage.
by the reconstructed edges. The user is required to specify the region to be inpainted.
Edge-based inpainting algorithms involve mainly two steps. i) reconstruction of the
edge image (in the masked regions) from damaged image edges and those available in
the unmasked regions, and ii) an edge-based pixel aggregation procedure. Consider an
image I that has lost information in region Ω(⊂ I), the region to be inpainted. Let
δΩ(⊂ I) represent the outer boundary of the inpainting region. Note that observations
are not available inside Ω and virtually anything could have existed there. One must
‘predict’ the image content in Ω based on appropriate assumptions about image prop-
erties such as local continuity and independence across objects separated by edges.
Digital inpainting algorithms are known to undetectably fill-in missing information
provided the damaged area is not too large in size.
The steps employed in a typical edge-based inpainting method and the correspond-
ing outputs are illustrated in Fig. 7.1. Following step 1 in the figure, the boundary
edges are detected and propagated through the damages. Once the edges in the dam-
108
(a) (b) (c) (d)
Fig. 7.2: (a) A real image superimposed with text. (b) Mask specifies the regionwhere the original image information is lost. (c) A damaged photograph, and (d) itsmask that specifies the regions to be filled-in.
aged region are reconstructed (as shown in the image corresponding to step 2), the
inpainting guided by the reconstructed edges yields the restored image shown in the
final step. In general, there can be more than one damaged region in an image.
In this section, we present a simple and effective edge-based inpainting method.
The proposed algorithm is similar to the one proposed by Rares et al. [17, 165] in the
sense that the major steps illustrated in Fig. 7.1 are performed, but it differs in the
methodology of implementation. Moreover, here our interest is not to just inpaint
the damaged images but to inpaint the damages while simultaneously filtering the
film-grain noise.
As stated earlier, we assume that the region to be filled in (Ω) is given a priori.
A convenient way to specify the regions to be filled-in is by an image mask which is
a binary image that distinguishes the regions that must be inpainted. For example,
for the image shown in Fig. 7.2 (a), the region to be inpainted (superimposed text)
is shown in the mask image (Fig. 7.2(b)) by white pixels. Fig. 7.2 (d) is another
example of a mask image that shows the regions to be inpainted in Fig. 7.2 (c).
7.2.1 Reconstruction of Edge Image
In any inpainting method, propagating the information in a uniform region is trivial.
The main difficulty arises at lost edges. As edges determine the topology, edge re-
109
construction needs to incorporate global image structure information in the inpainting
procedure. It is useful to note that the structure of the original image inside the region
Ω is a continuation of the (edge) structure outside it as illustrated earlier in Fig. 7.1.
Edge information can be utilized to independently propagate the information between
any two different objects in the inpainting region. Moreover, once we reconstruct the
edge image, the gray level propagation can be local. Thus, it is intuitive and reason-
able to first reconstruct the edges in the missing data regions to reliably propagate the
boundary image information.
It is important to note that reconstruction of edges from a damaged image is
different from the usual edge-linking process.
• The region of inpainting can be large and can result in much larger breaks in
edges than those that typically arise from edge-detection methods.
• The end points of the broken edges across a damaged region may even be farther
than some possible end points on the same side of the mask.
• The location/distance between pixels and the direction of edges become more
important than gradient magnitude and pixel intensity.
• Since the mask is user-specified and causes the edge gaps, its location and di-
mensions should be used to make the edge-linking process effective.
We propose the following edge-reconstruction procedure which makes effective use
of image features of the end points of edges, the contextual dimensions of the damages,
and the location of the mask. It comprises of the following steps.
1. Preprocessing and edge detection: We first find the edge map over Ωc with a
Canny edge operator [176]. The threshold is set high so that only strong edges
are captured. This yields an edge image everywhere except at the masked (to-
be-inpainted) regions. Fig. 7.3 (b) shows the edge information in Ωc derived
from Fig. 7.3 (a).
2. End points and features:
110
(a) (b) (c) (d)
Fig. 7.3: Edge reconstruction: (a) Degraded image showing maximum matching areadimensions. (b) Edge map of the degraded image. (c) Located end points, and (d) thereconstructed edge image.
(a) By searching over the boundary of the masked regions, the end points of
edges are located as points with exactly one neighboring edge pixel within
their eight point neighborhood. The end points thus detected are marked
with dots as shown in Fig. 7.3 (c).
(b) For each end point at location (ik, jk), its gradient direction Dk, magnitude
Gk, and the corresponding pixel intensity Ik are collected to form a feature
vector Fk = [ik, jk, Dk, Gk, Ik].
3. Matching: Our aim is to link edges across (possibly large) damaged regions.
(a) The end points are matched using an area constraint. The local matching
area for an end point is chosen to be a rectangle whose dimensions are set
to the largest breaks Mr and Mc along height and width, respectively, in
the strong edges due to damages. In Fig. 7.3 (a), we show selection of Mr
and Mc for the given degraded image.
(b) Two end points are considered for matching, only if i) the row and column-
wise absolute difference of their spatial locations is less than Mr and Mc,
respectively, and ii) the region between the two points belongs to Ω, the
inpainting region i.e., they are separated by the same damage.
111
(c) Ratio-test: Consider the end point p1 (with coordinates (i1, j1)) in Fig. 7.3
(c). Note that end point p2 (with coordinates (i2, j2)) as well as p3 (with
coordinates (i3, j3)) satisfy the matching area constraint for p1. In order
to decide to which of these two points should p1 be matched, the damaged
contents of the region between two end points is inferred as follows.
Inside the rectangular region whose opposite corners are determined by the
pixels (i1, j1) and (i2, j2) to be matched, the ratio of masked pixels to the
available pixels is determined. If the ratio is higher than a threshold r, we
decide that the region between these end points is a damaged region and
consider only such points for matching. We note that the rectangular box
enclosing p1 and p3 in Fig. 7.3 (c) includes a larger fraction of available
pixels, in contrast to the rectangular box enclosing p1 and p2. Hence, p2 is
a better candidate-match for p1 than p3.
(d) Based on (b) and (c), we perform constrained matching as follows: Let
Fpmand Fpn
denote the feature vectors corresponding to points pm and
pn, respectively. If |im − in| < Mr and |jm − jn| < Mc and the pixels
in the rectangle satisfy the ratio test (c) then we compute the distance
|Fpm− Fpn
|2.
(e) The end points with the minimum distance are selected as the matches i.e.,
L(pi) = arg mink |Fpi− Fpk
|2 among all candidate end points pk.
This mask-dependent local matching strategy suppresses false matches very ef-
fectively as compared to naive direct matching which uses only the feature vec-
tors of the end points.
4. Edge linking: Only the mutually matched end points which satisfy pk = L(pm)
and pm = L(pk) are connected to obtain the reconstructed edge map from
the degraded image. In Fig. 7.3 (d), we show the reconstructed edge image
corresponding to Fig. 7.3(a) using the above steps.
112
7.2.2 The Proposed Method
Once the edge image is reconstructed, inpainting can proceed by using suitably-derived
boundary neighbors. In our method, the edge information is effectively used to define
the neighboring pixels so as to limit the propagation of information within each object.
The algorithmic steps of the proposed inpainting method are as follows:
If (m, n) ∈ Ω,
1. S = φ. Take a window W with initial size 3× 3 about the location (m, n).
2. Collect valid neighbors within the window W and append as
S = S ∪ I(i, j); (i, j) ⊂ W, (i, j) /∈ ΩNote that the valid neighbors are pixels that belong to Ωc.
3. Check independently in all four directions whether the end pixel of the window
encounters an edge pixel of the reconstructed edge image; otherwise, increment
the size of window W in that direction. This limits the propagation of the
information within the object and renders it independent of other objects.
Let wl be the width of the window W between pixel (m, n) and the extreme
pixel on the left. We perform wl = wl +1, if (m, n−wl) /∈ E; else wl = wl where
E is the domain of non-zero edge pixels. Similarly, update the window length
along right (wr), top (wt) and bottom (wb) directions also.
4. Stop pixel collection when a certain number of available neighbors are accumu-
lated. i.e., stop if Cardinality S = T , else repeat steps (2) and (3). We set
the threshold T as 25.
Finally, the inpainted observation yp(m, n) = medianS.Here, the median is preferred over the average value of the pixels in set S for
better noise robustness and edge-preservation. Also, note that due to the appending
operation employed in step 2 of the algorithm, the computed median is actually a
weighted median that gives higher priority to the nearest valid neighbors. Schematic
representation of the proposed edge-based inpainting approach is shown in Fig. 7.4.
113
Fig. 7.4: Proposed inpainting algorithm.
7.3 RESULTS FOR INPAINTING
In this section, we demonstrate the performance of the proposed inpainting algorithm
on various synthetic as well as real degradations and compare it with well-known
existing techniques.
We begin with inpainting scratches on a synthetic image. Fig. 7.5(a) shows an
original brick image and Fig. 7.5(b) shows the degraded version with scratches (result-
ing in even some occluded edges). The output using the proposed inpainting scheme
is shown in Fig. 7.5(c). Even though this is a simulated example, the edge map was
reconstructed from the degraded image since the original image is never available in
real situations. We note that the inpainted image is quite close to the original image
even at the edges.
We next consider the degraded image shown in Fig. 7.6(a). The inpainting result
using the proposed algorithm is shown in Fig. 7.6(b). For comparison, we have
reproduced the output of the method described in [6] for the same image. Our result
Fig. 7.6: A synthetic image (a) with a ring mask. (b) Inpainted output. (c) Resultof Bertalmio [6].
is sharper at edges than that of [6] (Fig. 7.6(c)) but the method in [6] preserves broken
curvature better.
Next, we consider Fig. 7.7(a) which shows an original peppers image. The dam-
aged version with thick line scratches and patches is shown in Fig. 7.7(b). Using the
edge map of the degraded image (Fig. 7.7 (c)), which is known only at unmasked
regions, we arrive at the reconstructed edge map shown in Fig. 7.7(d). The final
inpainted image is given in Fig. 7.7(e). We note that the final result is quite good.
In Fig. 7.8(a), we consider the case of a real image of an old damaged painting.
The edge image in the unmasked regions is shown in Fig. 7.8(b). The reconstructed
115
(a) (b)
(c) (d) (e)
Fig. 7.7: Peppers image. (a) Original, (b) scratched, (c) edges from degraded image,(d) reconstructed edges, and (e) inpainted image using reconstructed edge map.
edge map using our edge reconstruction method is shown in Fig. 7.8(c) and the final
inpainted output using the proposed algorithm is given in Fig. 7.8(d). We note that
except at the middle right folds of the coat, where the edge reconstruction is not
perfect, our result compares favorably with the output Fig. 7.8(e) of the PDE-based
iterative approach of [6].
Next, we present inpainting results on a real damaged photograph (Fig. 7.9(a))
taken from [6]. The output of our algorithm and that of [6] are shown in Figs. 7.9(b)
and (c), respectively. While both the methods exhibit comparable performance overall,
our algorithm does better near the eyes while the result of [6] is better in inpainting
the damage near the bottom-hand.
116
(a) (b)
(c) (d) (e)
Fig. 7.8: An old painting (a) Degraded image. (b) Edge map from degraded image.(c) Reconstructed edge map. (d) Inpainted result using reconstructed edge map. (e)Output cropped from [6].
We next consider inpainting of superimposed text. Fig. 7.10(a) shows a bird image
with text written on it. The inpainting result of the proposed algorithm is shown in
Fig. 7.10(b). We also show the result of Chan and Shen [18] in Fig. 7.10(c) for
comparison. Our method, despite being less complex, is quite close to that of [18] in
performance.
Finally, we apply the proposed algorithm on a horse-cart image (Fig. 7.11(a))
superimposed with dense text. Fig. 7.11 (b) shows the inpainted image obtained with
our algorithm. The image inpainting result of [6] is shown in Fig. 7.11(c). Again, our
approach performs comparably.
117
(a) (b) (c)
Fig. 7.9: A real image (a) with mask on damaged region. (b) Inpainted result usingour method. (c) Result of Bertalmio [6].
(a) (b) (c)
Fig. 7.10: (a) A bird image superimposed with text. Inpainted result using (b)proposed method, and (c) method in [18].
7.4 INPAINTING IN PRESENCE OF NOISE
In chapter 5, we explored UKF for film-grain noise suppression. Old photographic
images and movies not only suffer from damages due to scratch, stain or blotches,
but also from film-grain noise. Text inpainting in the presence of film-grain noise is
also important for restoration and compression of photographic images and movies
containing subtitles and markers. In contrast to filtering, we must entirely rely on the
surrounding available pixels to estimate a pixel in image painting. This can be partially
accomplished during the prediction step of a recursive filter. Reliable prediction augurs
118
(a)
(b) (c)
Fig. 7.11: Horse-cart image (a) with superimposed text. (b) Inpainted output usingproposed method. (c) Output of [6].
well for simultaneous inpainting and filtering. Moreover, the prediction must not
depend on the missing observations in order to identify the state model parameters.
Importance sampling unscented Kalman filter, proposed in chapter 5, has a reliable
prediction step (without being dependent on observations) which suits our current
requirement. However, for the update step (in the damaged regions) we derive ‘virtual’
observations from the boundary topology and gray levels. This is in contrast to using
’known’ observations in traditional restoration.
119
7.4.1 ISUKF for Simultaneous Inpainting and Filtering
In this subsection, we propose a unified framework for inpainting images in the presence
of film-grain noise. The proposed inpainting process is embedded within the recursive
ISUKF framework to simultaneously combat noise while also accounting for missing
data. Prediction is based on the already estimated NSHP pixels as before. The issue in
performing inpainting within the filter framework is to handle missing observations. To
address this, we propose to reconstruct a missing observation by using the (available)
surrounding observations in our inpainting method. Inpainting is invoked only when we
encounter a missing pixel during the sequential filtering process. We first reconstruct
the edge image from the damaged noisy image, exactly as in section 7.2.1. We set a
high threshold for finding edges to mitigate the effect of noise on edge reconstruction.
We use the reconstructed edge map to guide the inpainting process as described in
section 7.2.2.
Details of the proposed method to sequentially filter and recover the intensity of
the original image at each pixel (m, n) in a raster-scan order from left-to-right are as
follows:
1. Prediction is based on the DAMRF prior exactly as in ISUKF (chapter 5). We
first formulate the DAMRF conditional density following step (1) and predict
the state mean and covariance by importance sampling of this non-Gaussian pdf
as in step (2) of section 5.3.1.
2. Inpainting is performed only for the missing observation pixels. We use the
output of our inpainting algorithm for ’reconstructing’ the observation pixel
intensity at (m, n) based on the available neighbors in the observation image.
3. UKF update step is as in ISUKF (step (3), of section 5.3.1). Note that for
missing data, we have the inpainted observations.
(a) We predict the state sigma points using Eq. (4.1) based on the predicted
mean and covariance.
120
Fig. 7.12: Proposed framework for inpainting in the presence of noise.
(b) We use the film-grain nonlinearity (Eq. 5.8) to predict the measurement
sigma points using the predicted state and measurement noise sigma points
as in step 3(b) of section 5.3.1.
(c) Following step 3 (c) of section 5.3.1, we update the mean and covariance of
the state at each pixel. In order to do this, we require the observations from
the degraded image. In case of missing observations, we use the inpainted
output from step 2 (above).
4. The updated state is the estimated pixel. This is used in the formulation of the
DAMRF prior and also for inpainting subsequent missing observations.
Figure 7.12 gives a flow diagram of the proposed filter.
7.5 RESULTS FOR JOINT RECOVERY
In this section, we recover damages in presence of both synthetic and real film-grain
noisy images. We begin with a synthetic image with two ellipses, a white region to
121
(a) (b)
Fig. 7.13: (a) Image degraded by film-grain noise and a big patch. (b) Output of theproposed method.
be inpainted, and corrupted by simulated film grain noise, as shown in Fig. 7.13(a).
The proposed recursive filter not only suppresses film-grain noise but also inpaints
efficiently the damaged region as shown in Fig. 7.13(b).
Next, we show a boat image and its degraded version in Figs. 7.14 (a) and (b),
respectively. Note that the masts have been scratched through. The image estimated
by the proposed algorithm and shown in Fig. 7.14(c) recovers most of the details,
except at the very narrow poles where the neighboring information is very different.
In the next example, we apply the proposed approach to suppress film-grain noise
while removing dense superimposed text. An original peppers image is shown in Fig.
122
(a) (b) (c)
(d) (e) (f)
Fig. 7.15: Peppers (a) Original. (b) Degraded and superimposed with text. (c)Reconstructed edge map. (d) Image recovered by the proposed filter (ISNR = 16.16dB). (e) Degraded with high noise. (f) Recovered result using reconstructed edge map(when input is (e)) (ISNR = 14.38 dB).
7.15(a). It is superimposed with dense text and corrupted (initially) with less degree
of film-grain noise as shown in Fig. 7.15(b). The reconstructed edge image is shown in
Fig. 7.15 (c). The recovered image using the reconstructed edge map is shown in Fig.
7.15(d). Next, we introduced a high level of film-grain noise as shown in Fig. 7.15(e)
and the corresponding inpainted and denoised result is shown in Fig. 7.15 (f). For
both levels of noise, we observe that the outputs are quite close to the original (Fig.
7.15(a)). Here we have given the ISNR values for a quantitative comparison.
Next, we consider a face image with real film-grain noise but simulated scratches
as shown in Fig. 7.16(a). The result of the proposed filter (Fig. 7.16(b)) looks quite
natural, recovering the fine details near the eyebrows, hair and nose. Scratches on
123
(a) (b)
(c) (d)
Fig. 7.16: Face with real film-grain. (a) Scratched version. (b) Inpainted and filteredresult (when input is (a)). (c) Degraded by patches. (d) Image recovered by our filter(when input is (c)).
the ears and near the mouth are also well recovered. Even when the face image is
damaged with patches and thick stripes (Fig. 7.16(c)), the recovered image by the
proposed filter (shown in Fig. 7.16(d)) recovers most of the details, except for small
artifacts near the forehead and mouth.
A cropped portion of a frame from the film “Dr. Mabuse” with real film-grain noise
is synthetically damaged at many places as shown in Fig. 7.17 (a). The results of the
proposed edge reconstruction method is given in Fig. 7.17(c), which can reconstruct
most of the major breaks in Fig. 7.17(b). The image recovered by the proposed filter
is shown in Fig. 7.17(d). The edges at the shoulders and at the tie are well-recovered,
and the image details near eyes and nose are restored properly.
124
(a) (b)
(c) (d)
Fig. 7.17: Mabuse (a) with scratches. (b) Edge image from (a). (c) Reconstructededge image (when input is (b)). (d) Image recovered by the proposed filter.
We next demonstrate the capability of the proposed approach to recover an im-
age of scanned painting with real film-grain noise (Fig. 7.18(a)). It is scratched at
several places including at edges as shown in Fig. 7.18(b). The recovered image (Fig.
7.18(c)) using the proposed approach can suppress film-grain and effectively inpaint
while retaining the image texture and finer details such as the ship partitions.
We have also performed removal of scratches in an image captured on a photo-
graphic film. The image is that of a locomotive shown in Fig. 7.19 (a). In the output
of the proposed algorithm shown in Fig. 7.19 (b), not only the numbers and letters
are properly recovered, but also the noise is suppressed well.
Finally, we consider another real example that has film-grain noise, embedded
125
(a) (b) (c)
Fig. 7.18: (a) Scanned painting with real film-grain noise. (b) Scratched version, and(c) recovered image.
(a) (b)
Fig. 7.19: (a) An image captured using a film-camera, and (b) recovered output.
text and a vertical scratch as shown in Fig. 7.20(a). The image recovered using the
proposed approach is shown in Fig. 7.20(b). It appears natural and without grain.
We have successfully inpainted the titles and the mark in the middle of the face, and
also recovered quite well the facial details such as hair, nose and mouth.
126
(a) (b)
Fig. 7.20: (a) A child image with text, scratch and film-grain noise. (b) Inpaintedand filtered output of the proposed method.
These examples demonstrate that the proposed filter is quite effective in suppress-
ing film-grain noise while performing image inpainting. The proposed algorithm takes
(using Matlab) about 20 seconds to execute on an image of size 200× 200 pixels when
run on a Pentium-IV PC with 256 MB RAM.
7.6 DISCUSSION
We proposed a novel recursive filter based on the unscented Kalman filter to recover
an image degraded by both film-grain noise and scratches. We first proposed an edge-
based inpainting scheme that is suitable for incorporating in a recursive framework. As
our inpainting method relies on the edge image, we proposed a local mask-dependent
edge-linking method to link edges across damaged portions. We proposed a pixel
aggregation procedure to inpaint missing pixels.
To inpaint damages in images corrupted by film-grain noise, we proposed a recur-
sive scheme based on the UKF which uses our inpainting method to derive the ob-
servations in the damaged regions. Prediction was based on a discontinuity-adaptive
MRF prior which preserves the edges while achieving good noise-reduction in uniform
regions.
127
CHAPTER 8
CONCLUSIONS
In this thesis, we addressed the problem of recovering an image from its degraded
observation through edge-preserving novel extensions of the Kalman filter and its vari-
ants. We first investigated the problem of filtering AWGN. The original image was
modeled with a non-Gaussian MRF prior based on a discontinuity-adaptive poten-
tial function. This conditional prior implicitly models the non-homogeneous nature of
images with local statistical information. We brought in the principle of importance
sampling to predict the mean and covariance of the state conditional prior, which are
then fed to the update step of the Kalman filter to arrive at the final image estimate.
We next examined the problem of handling non-linearity in photographic films and
proposed methods within the framework of the UKF. As a first step, we modeled the
image by a homogeneous AR state model with NSHP support. Assuming the initial
statistics of the state, we determined sigma points and transformed them through the
AR state model followed by the observation model. Employing UT, we predicted the
state and measurement statistics for use in the update stage of UKF.
We further relaxed the linearity constraint imposed by the homogeneous AR model
in UKF and incorporated an edge-preserving MRF image prior to capture local statis-
tics. The principle of importance sampling was adopted to predict the mean and
covariance of the state from the prior, which allowed us to determine the predicted
sigma points directly from the prior. These were propagated through the film-grain
observation model to obtain measurement sigma points and subsequently the statis-
tics required for the update step of the UKF. Experimental results on film-grain noise
reduction in photographic images were given to demonstrate the effectiveness of the
UKF-based algorithms.
128
We next addressed the problem of suppression of speckle noise in SAR images.
Here, one had to cope with multiplicative noise and yet preserve critical point and
line features. In order to achieve these goals, we tailored the DAMRF prior for SAR
imagery. The predicted sigma points using this prior were propagated through the
multiplicative model and were subsequently used to arrive at the image estimates.
Finally, we considered the problem of digital inpainting of noisy photographs. We
first developed an edge-based inpainting method which relies on the reconstructed
edge image. The end points of edges are matched based on a mask-based constrained
local matching strategy. Missing pixels are inpainted by propagating the boundary
information which is guided by the reconstructed edge map. Embedding the developed
inpainting method within the ISUKF-based filtering framework, we proposed a method
to simultaneously suppress film-grain noise while inpainting missing pixels.
All the proposed approaches were validated on synthetic as well as real examples.
They were also compared with existing methods to demonstrate the effectiveness of
the proposed methods.
8.1 SUGGESTIONS FOR FUTURE WORK
There are several directions for pursuing further research. We cite below a few of them.
• Extending the proposed ISKF and ISUKF for image restoration in the presence
of blur.
• Investigating the utility of the proposed ISUKF framework for other noise mod-
els and nonlinear restoration problems.
• Improving the performance of ARUKF and ISUKF through more accurate pre-
diction of the means and covariances using higher-order UT methods [137,138].
• Extending the inpainting ISUKF framework for occlusion-free tracking in videos.
• Exploring Gaussian-mixture UKF models for tracking multiple objects or people
in image sequences.
129
BIBLIOGRAPHY
[1] A. C. Bovik, Handbook of Image and Video Processing. Academic Press, 2005.
[2] A. K. Katsaggelos, “Recent trends in image restoration and enhancement techniques,”in Proc. IEEE Asia Pacific Conf. on Circuits and Systems, pp. 458–459, 1996.
[3] H. C. Andrews and B. R. Hunt, Digital image restoration. New Jersey: Prentice-Hall,Inc, 1977.
[4] A. M. Tekalp and G. Pavlovic, “Image restoration with multiplicative noise: Incorpo-rating the sensor nonlinearity,” IEEE Trans. Signal Process., vol. 39, pp. 2132 – 2136,1991.
[5] J. W. Goodman, “Some fundamental properties of speckle,” J. Opt. Soc. Amer., vol. 66,pp. 1145–1150, 1976.
[6] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” in Proc.SIGGRAPH, Computer Graphics Proceedings, pp. 417 – 424, 2000.
[7] R. E. Kalman, “A new approach to linear filtering and prediction problems,” Trans.ASME, Ser. D, Journal of Basic Engineering, vol. 82, pp. 34–45, 1960.
[8] J. W. Woods and C. H. Radewan, “Kalman filtering in two dimensions,” IEEE Trans.Inform. Theory, vol. 23, pp. 473 – 482, 1977.
[9] F. Daum, “Nonlinear filters: beyond the Kalman filter,” IEEE Aerospace and Elec-tronic Systems Magazine, vol. 20, pp. 57–69, 2005.
[10] Z. Chen, “Bayesian filtering: From Kalman filters to particle filters, and beyond.”Technical Report, 2003.
[11] S. R. Kadaba, S. B. Gelfand, and R. L. Kashyap, “Recursive estimation of images usingnon-Gaussian autoregressive models,” IEEE Trans. Image Processing, vol. 7, pp. 1439– 1452, 1998.
[12] A. Achim, E. E. Kuruoglu, and J. Zerubia, “SAR image filtering based on the heavy-tailed Rayleigh model,” IEEE Trans. Image Processing, vol. 15, pp. 2686–2693, 2006.
[13] M. Dai, C. Peng, A. K. Chan, and D. Loguinov, “Bayesian wavelet shrinkage with edgedetection for SAR image despeckling,” IEEE Trans. Geoscience and Remote Sensing,vol. 42, pp. 1642–1648, 2004.
[14] F. Argenti, T. Bianchi, and L. Alparone, “Multiresolution MAP despeckling of SARimages based on locally adaptive generalized Gaussian pdf modeling,” IEEE Trans.Image Processing, vol. 15, pp. 3385–3399, 2006.
130
[15] L. Gagnon and A. Jouan, “Speckle filtering of SAR images - a comparative studybetween a complex-wavelet-based and standard filters,” Proc. SPIE, vol. 3169, pp. 80–91, 1997.
[16] O. Lankoande, M. M. Hayat, and B. Santhanam, “Speckle modeling and reduction insynthetic aperture radar imagery,” in Proc. IEEE Int. Conf. Image Processing (ICIP),pp. III:317–320, 2005.
[17] A. Rares, M. Reinders, and J. Biemond, “Edge-based image restoration,” IEEE Trans.Image Processing, vol. 14, pp. 1454–1468, 2005.
[18] T. Chan and J. Shen, “Mathematical models for local non-texture inpaintings,” SIAMJournal on Applied Mathematics, vol. 62, pp. 1019–1043, 2001.
[19] A. K. Katsaggelos, “Iterative image restoration algorithms,” Optical Engineering,vol. 28, pp. 735 – 748, 1989.
[20] M. A. Robertson and R. L. Stevenson, “DCT quantization noise in compressed images,”IEEE Trans. Circuits and Systems for Video Technology, vol. 15, pp. 27–38, 2005.
[21] H. Soltanian-Zadeh, J. P. Windham, and A. E. Yagle, “A multi-dimensional non-linearedge-preserving filter for magnetic resonance image restoration,” IEEE Trans. ImageProcessing, vol. 4, pp. 147–161, 1995.
[22] F. T. Ulaby, “Radar signatures of terrain: Useful monitors of renewable resources,”Proc. IEEE, vol. 70, pp. 1410–1433, 1982.
[23] A. Kokaram, R. Morris, W. Fitzgerald, and P. Rayner, “Interpolation of missing datain image sequences,” IEEE Trans. Image Processing, vol. 11, pp. 1509–1519, 1995.
[24] S. Masnou and J. Morel, “Level-lines based disocclusion,” in Proc. IEEE Int. Conf.Image Processing (ICIP), pp. 259–263, 1998.
[25] A. K. Jain, Fundamentals of digital image processing. India: Prentice Hall, 2000.
[26] T. Berger, J. O. Stromberg, and T. Eltoft, “Adaptive regularized constrained leastsquares image restoration,” IEEE Trans. Image Processing, vol. 8, pp. 1191–1203,1999.
[27] D. Angwin and H. Kaufman, “Image restoration using a reduced order model Kalmanfilter,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing (ICASSAP),pp. 1000–1003, 1988.
[28] H. Kaufman, J. W. Woods, D. Subrahmanyam, and A. M. Tekalp, “Estimation andidentification of two-dimensional images,” IEEE Trans. Automatic Control, vol. 28,pp. 745–756, 1983.
[29] F. C. Jeng and J. W. Woods, “Inhomogeneous Gaussian image models for estimationand restoration,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 36, pp. 1305–1312, 1988.
131
[30] A. M. Tekalp, H. Kaufman, and J. Woods, “Edge-adaptive image restoration withringing suppression,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 892–899, 1989.
[31] J. Biemond and J. Gerbrands, “An edge preserving recursive noise smoothing algorithmfor image data,” IEEE Trans. Systems, Man and Cybernetics, vol. 9, pp. 622–627, 1979.
[32] H. R. Keshavan and M. D. Srinath, “Sequential estimation technique for enhancementof noisy images,” IEEE Trans. Computers, vol. 26, pp. 971–988, 1977.
[33] Y. C. Chang, S. R. Kadaba, P. C. Doerschuk, and S. B. Gelfand, “Image restorationusing recursive Markov random field models driven by Cauchy distributed noise,” IEEETrans. Signal Process. Letters, vol. 8, pp. 65–66, 2001.
[34] S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions, and the Bayesianrestoration of images,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 6, pp. 721–741, 1984.
[35] C. Bouman and K. Sauer, “A generalized Gaussian image model for edge-preservingMAP estimation,” IEEE Trans. Image Processing, vol. 2, pp. 296–310, 1993.
[36] P. Charbonnier, L. Blanc-Feraud, G. Aubert, and M. Barlaud, “Deterministic edge-preserving regularization in computed imaging,” IEEE Trans. Image Processing, vol. 6,pp. 298–311, 1997.
[37] F. C. Jeng and J. W. Woods, “Compound Gauss-Markov random fields for imageestimation,” IEEE Trans. Signal Process., vol. 39, pp. 683–697, 1991.
[38] S. Roth and M. J. Black, “Fields of experts: A framework for learning image priors,” inProc. IEEE Int. Conf. Computer Vision and Pattern Recognition, pp. 860–867, 2005.
[39] M. Ceccarelli, “A finite Markov random field approach to fast edge-preserving imagerecovery,” Image and Vision Computing, vol. 25, pp. 792–804, 2007.
[40] R. G. Aykroyd, “Bayesian estimation for homogeneous and inhomogeneous Gaussianrandom fields,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 20, pp. 533–539,1998.
[41] M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, “A tutorial on particle filtersfor on-line nonlinear/non-Gaussian Bayesian tracking,” IEEE Trans. Signal Process.,vol. 50, pp. 174 – 188, 2002.
[42] Y. Rui and Y. Chen, “Better proposal distributions: Object tracking using unscentedparticle filter,” in Proc. IEEE Computer Society Conf. Computer Vision and PatternRecognition, pp. 786–794, 2001.
[43] H. E. Knutsson, R. Wilson, and G. H. Granlund, “Anisotropic nonstationary imageestimation and its applications: Part I-restoration of noisy images,” IEEE Trans. Com-munication, vol. 31, pp. 388–397, 1983.
132
[44] G. Gilboa, N. Sochen, and Y. Y. Zeevi, “Image enhancement and denoising by complexdiffusion process,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 25, pp. 1020–1036, 2004.
[45] P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion,”IEEE Trans. Pattern Anal. and Machine Intell., vol. 12, pp. 629–639, 1990.
[46] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removalalgorithms,” Physica D, vol. 60, pp. 259–268, 1992.
[47] L. Kaur, S. Gupta, and R. C. Chauhan, “Image denoising using wavelet threshold-ing,” in Indian Conf. on Computer Vision, Graphics and Image Processing (ICVGIP),(India), 2002.
[48] S. G. Chang, B. Yu, and M. Vetterli, “Adaptive wavelet thresholding for image denois-ing and compression,” IEEE Trans. Image Processing, vol. 9, pp. 1532–1546, 2000.
[49] M. A. T. Figueiredo and R. D. Nowak, “Wavelet based image estimation: An empiricalBayes approach, using Jeffrey’s noninformative prior,” IEEE Trans. Image Processing,vol. 10, pp. 1322–1331, 2001.
[50] M. Jansen and A. Bultheel, “Geometric prior for noise free wavelet coefficient in imagede-noising,” in Lecture notes in Statistics (P. Mueller and B. Vidakovic, eds.), vol. 141,pp. 223–242, Spinger - verlag, 1999.
[51] A. S. Tavildar, H. M. Guptha, and S. N. Guptha, “Maximum a posteriori estimationin presence of film-grain noise,” Signal Process. (North Holland), vol. 8, pp. 363–368,1985.
[52] A. D. Stefano, P. R. White, and W. B. Collis, “Film grain reduction in colour imagesusing undecimated wavelet transform,” Image and Vision Computing, vol. 22, pp. 873–882, 2004.
[53] F. Naderi and A. A. Sawchuk, “Estimation of images degraded by film-grain noise,” J.Applied Optics, vol. 17, pp. 1228 – 1237, 1978.
[54] J. C. K. Yan and D. Hatzinakos, “Signal-dependent film grain noise removal and gen-eration based on higher-order statistics,” in Proc. of IEEE Signal Processing Workshopon Higher-Order Statistics, pp. 77–81, 1997.
[55] B. R. Hunt, “Bayesian methods in nonlinear digital image restoration,” IEEE Trans.Computers, vol. 26, pp. 219 – 229, 1977.
[56] G. K. Froehlich, J. F. Walkup, and T. F. Krille, “Estimation in signal-dependent film-grain noise,” Applied Optics, vol. 20, pp. 3619 – 3626, 1981.
[57] T. M. Moldovan, S. Roth, and M. J. Black, “Denoising archival films using a learnedBayesian model,” in Proc. IEEE Int. Conf. Image Processing (ICIP), pp. 2641–2644,2006.
133
[58] S. I. Sadhar and A. N. Rajagopalan, “Image recovery under non-linear and non-Gaussian degradations,” J. Opt. Soc. Amer. (A), vol. 22, pp. 604–615, 2005.
[59] S. I. Sadhar and A. N. Rajagopalan, “Image estimation in film-grain noise,” IEEETrans. Signal Process. Letters, vol. 12, pp. 238–241, 2005.
[60] F. Argenti, G. Torricelli, and L. Alparone, “MMSE filtering of generalised signal-dependent noise in spatial and shift-invariant wavelet domains,” Signal Processing,vol. 86, pp. 2056–2066, 2006.
[61] V. Bruni and D. Vitulano, “Old movies noise reduction via wavelets and Wiener filter,”Journal of Winter School of Computer Graphics (WSCG), vol. 12, pp. 65–69, 2004.
[62] V. S. Frost, J. A. Stiles, A. Josephine, K. S. Shanmugan, and J. C. Holtzman, “A modelfor radar images and its application to adaptive filtering of multiplicative noise,” IEEETrans. Pattern Anal. and Machine Intell., vol. 4, pp. 157–165, 1982.
[63] J. S. Lee, “Digital image enhancement and noise filtering by use of local statistics,”IEEE Trans. Pattern Anal. and Machine Intell., vol. 2, pp. 165–168, 1980.
[64] D. T. Kuan, A. A. Sawchuk, T. C. Strand, and P. Chavel, “Adaptive noise smoothingfilter for images with signal dependent noise,” IEEE Trans. Pattern Anal. and MachineIntell., vol. 7, pp. 165–177, 1985.
[65] A. Lopes, R. Touzi, and E. Nezzy, “Adaptive speckle filters and scene heterogeneity,”IEEE Trans. Geoscience and Remote Sensing, vol. 28, pp. 992–1000, 1990.
[66] A. Lopes, E. Nezry, R. Touzi, and H. Laur, “Maximum a posteriori speckle filteringand first order texture models in SAR images,” in Proc. IEEE Int. Geoscience andRemote Sensing Symposium, pp. 2409–2412, 1990.
[67] T. R. Crimmins, “Geometric filter for reducing speckle,” Optical Engineering, vol. 25,pp. 651–654, 1986.
[68] M. R. Azimi-Sadjadi and S. Bannour, “Two-dimensional adaptive block Kalman filter-ing of SAR imagery,” IEEE Trans. Geoscience and Remote Sensing, vol. 29, pp. 742–753, 1991.
[69] Y. Yu and S. T. Acton, “Speckle reducing anisotropic diffusion,” IEEE Trans. ImageProcessing, vol. 11, pp. 1260–1270, 2002.
[70] H. M. Salinas and D. C. Fernandez, “Comparison of PDE-based nonlinear diffusionapproaches for image enhancement and denoising in optical coherence tomography,”IEEE Trans. Medical Imaging, vol. 26, pp. 761–771, 2007.
[71] H. Xie, L. E. Pierce, and F. T. Ulaby, “SAR speckle reduction using wavelet denoisingand Markov random field modeling,” IEEE Trans. Geoscience and Remote Sensing,vol. 40, pp. 2196–2212, 2002.
[72] A. Achim, P. Tsakalides, and A. Bezerianos, “SAR image denoising via Bayesianwavelet shrinkage based on heavy-tailed modeling,” IEEE Trans. Geoscience and Re-mote Sensing, vol. 41, pp. 1773–1785, 2003.
134
[73] M. I. H. Bhuiyan, M. O. Ahmad, and M. N. S. Swamy, “Spatially-adaptive wavelet-based method using the Cauchy prior for denoising the SAR images,” IEEE Trans.Circuits and Systems for Video Technology, vol. 17, pp. 500–507, 2007.
[74] S. P. Luttrell and C. J. Oliver, “Prior knowledge in synthetic-aperture radar process-ing,” J. Phys. D, vol. 19, pp. 333–356, 1986.
[75] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, “Simultaneous structure and textureimage inpainting,” IEEE Trans. Image Processing, vol. 12, pp. 882–889, 2003.
[76] H. Kaufman and A. M. Tekalp, “Survey of estimation techniques in image restoration,”IEEE Control Systems Magazine, vol. 11, pp. 16 – 24, 1991.
[77] S. Z. Li, Markov Random Field Modeling in Computer Vision. New York, inc.:Springer-Verlag, 1995.
[78] S. Z. Li, “On discontinuity-adaptive smoothness priors in computer vision,” IEEETrans. Pattern Anal. and Machine Intell., vol. 17, pp. 576–586, 1995.
[79] D. J. C. MacKay, “Introduction to Monte Carlo methods,” in Learning in GraphicalModels (M. I. Jordan, ed.), NATO Science Series, pp. 175–204, Kluwer Academic Press,1998.
[80] E. C. Anderson, “Monte Carlo methods and importance sampling.” Lecture Notes,at ‘http://ib.berkeley.edu/labs/slatkin/eriq/classes/guest lect/mc lecture notes.pdf’,1999.
[81] S. Julier and J. Uhlmann, “A general method for approximating nonlinear transfor-mations of probability distributions.” Tech. rep., RRG, Dept. of Engineering Science,University of Oxford, 1996.
[82] S. J. Julier, J. K. Uhlmann, and H. F. Durrant-Whyte, “A new method for the nonlin-ear transformation of means and covariances in filters and estimators,” IEEE Trans.Automatic Control, vol. 45, pp. 477–482, 2000.
[83] R. V. Merwe, N. de Freitas, A. Doucet, and E. Wan, “The unscented particle fil-ter.” Technical report CUED/F-INFENG/TR 380, Cambridge University EngineeringDepartment, 2000.
[84] E. A.Wan and R. V. Merwe, “The unscented Kalman filter for nonlinear estimation,”in Proceedings of IEEE Symposium on Adaptive Systems for Signal Processing, Com-munications and Control (AS-SPCC), pp. 153–158, 2000.
[85] J. L. Crassidis and F. L. Markley, “Unscented filtering for spacecraft attitude estima-tion,” J. Guid. Control Dyn., vol. 26, pp. 536–542, 2003.
[86] P. Li, T. Zhang, and B. Ma, “Unscented Kalman filter for visual curve tracking,” Imageand Vision Computing, vol. 22, pp. 157–164, 2004.
[87] B. Stenger, P. Mendonca, and R. Cipolla, “Model-based hand tracking using an un-scented Kalman filter,” in Proc. British Machine Vision Conference, pp. 63–72, 2001.
135
[88] G. R. K. S. Subrahmanyam, A. N. Rajagopalan, and R. Aravind, “Unscented Kalmanfilter for image estimation in film-grain noise,” in Proc. IEEE Int. Conf. Image Pro-cessing (ICIP), pp. IV:17–21, 2007.
[89] E. R. Dougherty, Random processes for image and signal processing. New York:SPIE/IEEE Series on Imaging Science and Engineering, 1999.
[90] M. Petrou and P. G. Sevilla, Image processing: Dealing with texture. London, UK:John Wiley and Sons, 2006.
[91] A. Rangarajan and R. Chellappa, “Markov random field models in image processing,”in The handbook of brain theory and neural networks, NATO Science Series, pp. 564–567, MIT Press, 1998.
[92] N. Ahuja and B. Schachter, “Image models,” ACM Computing Surveys, vol. 13,pp. 373–397, 1981.
[93] S. C. Zhu, “Statistical modeling and conceptualization of visual patterns,” IEEE Trans.Pattern Anal. and Machine Intell., vol. 25, pp. 691–712, 2003.
[94] E. B. Ranguelova, Segmentation of textured images on three-dimensional lattices. PhDthesis, Dept. of Electronic and Electrical Engg., Univ. of Dublin, Trinity College,Dublin, 2002.
[95] P. Perez, “Markov random fields and images,” CWI Quarterly, vol. 11, pp. 413–437,1998.
[96] J. W. Woods, “Two-dimensional discrete Markovian fields,” IEEE Trans. Inform. The-ory, vol. 18, pp. 232–240, 1972.
[97] S. Kumar and M. Hebert, “Discriminative random fields,” Int. J. Computer Vision,vol. 68, pp. 179–201, 2006.
[98] H. M. Wallach, “Conditional random fields: An introduction.” University of Pennsyl-vania CIS, Technical Report MS-CIS-04-21, 2004.
[99] D. E. Melas and S. P. Wilson, “Double Markov random fields and Bayesian imagesegmentation,” IEEE Trans. Signal Process., vol. 50, pp. 357–365, 2002.
[100] M. A. T. Figueiredo, “Bayesian methods and Markov random fields.” Departmentof Electrical and Computer Engineering, Instituto Superior Tecnico, available at“www.lx.it.pt/˜mtf/FigueiredoCVPR.pdf”.
[101] D. Melas, A Bayesian Approach to the Segmentation of Textural Images. PhD thesis,Dept. of Electronic and Electrical Engg., Univ. of Dublin, Trinity College, Dublin,1998.
[102] J. Besag, “Spatial interaction and the statistical analysis of lattice systems,” J. RoyalStatist. Society (B), vol. 36, pp. 192–236, 1974.
[103] D. Griffeath, “Introduction to random fields,” in Denumerable Markov Chains (J. L. S.J. G. Kemeny and A. W. Knapp, eds.), pp. 425–458, New York: Springer-Verlag, 1976.
136
[104] R. Stevenson, B. Schmitz, and E. Delp, “Discontinuity-preserving regularization ofinverse visual problems,” IEEE Trans. Systems, Man and Cybernetics, vol. 24, pp. 455–469, 1994.
[105] P. J. Green, “Bayesian reconstruction from emission tomography data using a modifiedEM algorithm,” IEEE Trans. Medical Imaging, vol. 9, pp. 84–93, 1990.
[106] S. Geman, D. McClure, and D. Geman, “A nonlinear filter for film restoration andother problems in image processing,” Computer Vision Graphics and Image Processing,vol. 54, pp. 281–289, 1992.
[107] A. Blake and A. Zisserman, Visual Reconstruction. Cambridge, M.A.: M.I.T. Press,1987.
[108] S. Geman and G. Reynolds, “Constrained restoration and the recovery of discontinu-ities,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 14, pp. 367–383, 1992.
[109] T. Hebert and R. Leahy, “A generalized EM algorithm for 3-D Bayesian reconstructionfrom Poisson data using Gibbs priors,” IEEE Trans. Medical Imaging, vol. 8, pp. 194–202, 1989.
[110] S. Kapoor, P. Y. Mundkur, and U. B. Desai, “Depth and image recovery using a MRFmodel,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 16, pp. 1117 – 1122,1994.
[111] L. Khouas, C. Odet, and D. Friboulet, “3D furlike texture generation by a 2D autore-gressive synthesis,” in Proc. Inter. Conf. in Central Europe on Computer Graphics andVisualization (WSCG), pp. 171–177, 1998.
[112] S. Haykin, Kalman Filtering and Neural Networks. John Wiley and Sons, New York,inc.: Wiley-interscience, 2001.
[113] A. M. Tekalp, H. Kaufman, and J. W. Woods, “Fast recursive estimation of the pa-rameters of a space-varying autoregressive image model,” IEEE Trans. Acoust., Speech,Signal Processing, vol. 33, pp. 469–472, 1985.
[114] S. Z. Li, “Discontinuity-adaptive MRF prior and robust statistics: A comparativestudy,” Image and Vision Computing, vol. 13, pp. 227–233, 1995.
[115] S. Z. Li, “Robustizing robust M-estimation using deterministic annealing,” PatternRecognition, vol. 29, pp. 159–166, 1996.
[116] J. Pengelly, “Monte Carlo methods.” Students Tutorial, February 2002, available at‘http://csnet.otago.ac.nz/cosc453/student tutorials/monte carlo.pdf.’.
[117] M. Pagano and W. Sandmann, “Efficient rare event simulation: A tutorial on im-portance sampling.” Tutorial presented at Third International Working Conference onPerformance Modelling and Evaluation of Heterogeneous Networks, 2005.
[118] T. C. Hesterberg, Advances in Importance Sampling. PhD thesis, Stanford University,US, 1988.
137
[119] I. Beichl and N. F. Sullivan, “The importance of importance sampling,” Computing inSci. and Engg., vol. 1, pp. 71 – 73, 1999.
[120] P. H. Borcherds, “Importance sampling: an illustrative introduction,” Eur. J. Phys.,vol. 21, pp. 405–411, 2000.
[121] S. J. Julier and J. K. Uhlmann, “A new extension of the Kalman filter to nonlinearsystems,” in Proc. of AeroSense: 11th International Symposium on Aerospace/DefenseSensing, Simulation and Controls, vol. 3068, pp. 182–193, 1997.
[122] R. V. Merwe, Sigma-point Kalman filters for probabilistic inference in dynamic state-space models. PhD thesis, OGI Sch. of Sci. and Engg., Oreg. Health and Sci. Univ.,Portland, Oreg., 2004.
[123] B. D. Anderson and J. B. Moore, Optimal filtering. Englewood Cliffs, N. J: Prentice-Hall, 1979.
[124] B. Ristic, M. S. Arulampalam, A. Farina, and D. Benvenuti, “Performance boundsand comparison of nonlinear filters for tracking a ballistic object on re-entry,” in IEEProceedings: Radar, Sonar and Navigation, vol. 150, pp. 65–70, 2003.
[125] S. J. Julier and J. K. Uhlmann, “Unscented filtering and nonlinear estimation,” Proc.IEEE, vol. 92, pp. 401–422, 2004.
[126] S. J. Julier and J. K. Uhlmann, “The scaled unscented transformation,” in Proc. Amer.Control Conf., pp. 4555–4559, 2002.
[127] K. Ito and K. Xiong, “Gaussian filters for nonlinear filtering problems,” IEEE Trans.Automatic Control, vol. 45, pp. 910–927, 2000.
[128] R. V. Merwe and E. A. Wan, “Sigma-point Kalman filters for probabilistic inference indynamic state-space models,” in Proceedings of the Workshop on Advances in MachineLearning, 2003.
[129] R. V. Merwe, E. A. Wan, and A. T. Nelson, “Dual estimation and the unscentedtransformation,” in Advances in Neural Information Processing Systems, pp. 666–672,2000.
[130] I. A. Gura and R. H. Gersten, “Interpretation of n-dimensional covariance matrices,”American Institute of Aeronautics and Astronautics Journal, vol. 9, pp. 740–742, 1971.
[131] A. ACCESS, “Statistical analysis site.” http://www.aiaccess.net/e gm.htm.
[132] S. J. Julier, J. K. Uhlmann, and H. F. Durrant-Whyte, “A new approach for filteringnonlinear systems,” in Proc. Amer. Control Conf., pp. 1628–1632, 1995.
[133] B. Ristic, S. Arulampalam, and N. Gordon, Beyond the Kalman Filter: Particle Filtersfor Tracking Applications. New York, inc.: Artech House Radar Library, 2004.
[134] S. J. Julier, Comprehensive process models for high-speed land vehicles. PhD thesis,Robotics Research Group, Wadham Collage, University of Oxford, UK, 1997.
138
[135] R. V. Merwe and E. Wan, “The square-root unscented Kalman filter for state andparameter-estimation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing(ICASSAP), pp. 3461–3464, 2001.
[136] S. J. Julier and J. K. Uhlmann, “Reduced sigma point filters for the propagation ofmeans and covariances through nonlinear transformations,” in Proc. Amer. ControlConf., pp. 887–892, 2002.
[137] D. Tenne and T. Singh, “The higher order unscented filter,” in Proc. Amer. ControlConf., pp. 2441– 2446, 2003.
[138] Y. Wu, M. Wu, D. Hu, and X. Hu, “An improvement to unscented transformation,”in Australian Conference on Artificial Intelligence, pp. 1024–1029, 2004.
[139] R. V. Merwe and E. A. Wan, “Gaussian mixture sigma-point particle filters for sequen-tial probabilistic inference in dynamic state-space models,” in Proc. IEEE Int. Conf.Acoust., Speech, Signal Processing (ICASSAP), pp. 701–704, 2003.
[140] L. Angrisani, A. Baccigalupi, and R. S. L. Moriello, “Ultrasonic time-of-flight estima-tion through unscented Kalman filter,” IEEE Trans. Instrumentation and Measure-ment, vol. 55, pp. 1077–1084, 2006.
[141] T. Lefebvre, H. Bruyninckx, and J. D. Schutter, “Comment on ‘a new method forthe nonlinear transformation of means and covariances in filters and estimators (andauthor’s reply)’,” IEEE Trans. Automatic Control, vol. 47, pp. 1406–1409, 2002.
[142] A. Sitz, U. Schwarz, and J. Kurths, “The unscented Kalman filter, a powerful toolfor data analysis,” Int. J. of Bifurcation and Chaos in App. Sci. and Engg., vol. 14,pp. 2093–2105, 2004.
[143] J. J. LaViola, “A comparison of unscented and extended Kalman filtering for estimatingquaternion motion,” in Proc. Amer. Control Conf., pp. 2435–2440, 2003.
[144] S. I. Sadhar, Some new approaches for image restoration and blur identification usingparticle filter. PhD thesis, Dept. of Electrical Engg., IIT Madras, India, 2005.
[145] D. Zou, J. Tian, J. Bloom, and J. Zhai, “Data hiding in film grain,” in 5th InternationalWorkshop on Digital Watermarking, pp. 197–211, 2006.
[146] N. J. Gordon, D. J. Salmond, and A. F. M. Smith, “Novel approach to nonlinear/non-Gaussian Bayesian state estimation,” in IEE Proc. F: Radar and Signal Process.,vol. 140, pp. 107 – 113, 1993.
[147] R. Birk, W. Camus, E. Valenti, and W. McCandless, “Synthetic aperture radar imagingsystems,” IEEE AES Systems Magazine, vol. 10, pp. 15–23, 1995.
[148] K. Tomiyasu, “Tutorial review of synthetic-aperture radar (SAR) with applications toimaging of the ocean surface,” Proc. IEEE, vol. 66, pp. 563–587, 1978.
[149] O. Lankoande, M. M. Hayat, and B. Santhanam, “Speckle reduction of SAR imagesusing a physically based Markov random field model and simulated annealing,” Proc.SPIE, vol. 5808, pp. 210–221, 2005.
139
[150] M. Costanntini, A. Farina, and F. Zirilli, “The fusion of different resolution SARimages,” Proc. IEEE, vol. 85, pp. 139–146, 1997.
[151] M. Mastriani and A. E. Giraldez, “Kalman’s shrinkage for wavelet-based despecklingof SAR images,” Int. J. Intelligent Technology, vol. 1, pp. 190–196, 2006.
[152] F. Sattar, L. Floreby, G. Salomonsson, and L. Benny, “Image enhancement based ona nonlinear multiscale method,” IEEE Trans. Image Processing, vol. 6, pp. 888–895,1997.
[153] R. C. Gonzalez and R. E. Woods, Digital image processing. Asia: Pearson Education,2000.
[154] I. E. Abdou and W. K. Pratt, “Quantitative design and evaluation of enhance-ment/thresholding edge detectors,” Proc. IEEE, vol. 67, pp. 753–766, 1979.
[155] A. Hirani and T. Totsuka, “Combining frequency and spatial domain information forfast interactive image noise removal,” in Proc. SIGGRAPH, Computer Graphics Pro-ceedings, pp. 269–276, 1996.
[156] A. Criminisi, P. Perez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE Trans. Image Processing, vol. 13, pp. 1200–1212, 2004.
[157] J. Shen, “Inpainting and the fundamental problem of image processing,” SIAM News,vol. 36, 2003.
[158] T. Chan, S. Kang, and J. Shen, “Euler’s elastica and curvature based inpaintings,”SIAM Journal on Applied Mathematics, vol. 63, pp. 564–592, 2002.
[159] T. Chan and J. Shen, “Non-texture inpaintings by curvature-driven diffusions (CDD),”J. Vis. Commun. Image Res., vol. 12, pp. 436–449, 2001.
[160] M. M. Oliveira, B. Bowen, R. McKenna, and Y. S. Chang, “Fast digital image in-painting,” in Proc. of the Inter. Conf. on Visualization, Imaging and Image Processing(VIIP), pp. 261–266, 2001.
[161] A. Telea, “An image inpainting technique based on the fast marching method,” J.Graphics Tools, vol. 9, pp. 25–36, 2004.
[162] K. Ko and S. Kim, “Efficient inpainting of old film scratch using Sobel edge opera-tor based isophote computation.,” in Proc. IEEE Int. Conf. Comput. and Inf. Tech.,pp. 124–129, 2006.
[163] M. Yasuda, J. Ohkubo, and K. Tanaka, “Digital image inpainting based on Markovrandom field,” in CIMCA-IAWTIC, pp. 747–752, IEEE Computer Society, 2005.
[164] N. Komodakis and G. Tziritas, “Image completion using global optimization,” in Proc.IEEE Computer Society Conf. Computer Vision and Pattern Recognition, pp. 442– 452,2006.
140
[165] A. Rares, M. J. T. Reinders, and J. Biemond, “Image sequence restoration in thepresence of pathological motion and severe artifacts,” in Proc. IEEE Int. Conf. Acoust.,Speech, Signal Processing (ICASSAP), pp. 3365–3368, 2002.
[166] A. L. Bertozzi, S. Esedoglu, and A. Gillette, “Inpainting of binary images using theCahn-Hilliard equation,” IEEE Trans. Image Processing, vol. 16, pp. 285–291, 2007.
[167] A. N. Cohen, “The Cahn-Hilliard equation: mathematical and modeling perspectives,”Adv. Math. Sci. Appl., vol. 8, pp. 965–985, 1998.
[168] J. Jia and C. Tang, “Image repairing: Robust image synthesis by adaptive ND ten-sor voting,” in Proc. IEEE Computer Society Conf. Computer Vision and PatternRecognition, pp. 643–650, 2003.
[169] J. Jia and C. K. Tang, “Inference of segmented color and texture description by tensorvoting,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 26, pp. 771–786, 2004.
[170] G. Medioni, M. Lee, and C. Tang, A computational framework for feature extractionand segmentation. Elseviers Science, 2000.
[171] K. A. Patwardhan, G. Sapiro, and M. Bertalmo, “Video inpainting under constrainedcamera motion,” IEEE Trans. Image Processing, vol. 16, pp. 545–553, 2007.
[172] J. Jia, Y. Tai, T. Wu, and C. Tang, “Video repairing under variable illumination usingcyclic motions,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 28, pp. 832–839,2006.
[173] Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H. Shum, “Full-frame video stabilizationwith motion inpainting,” IEEE Trans. Pattern Anal. and Machine Intell., vol. 28,pp. 1150–1164, 2006.
[174] C. Ballester, M. Bertalmio, V. Caselles, L. Garrido, A. Marques, and F. Ranchin,“An inpainting-based deinterlacing method,” IEEE Trans. Image Processing, vol. 16,pp. 2476–2491, 2007.
[175] C. A. Z. Barcelos and M. A. Batista, “Image restoration using digital inpainting andnoise removal,” Image and Vision Computing, vol. 25, pp. 61–69, 2007.
[176] J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal.and Machine Intell., vol. 8, pp. 679–714, 1986.
141
LIST OF PAPERS BASED ON THESIS
A. Journal Papers
1. G. R. K. S. Subrahmanyam, A. N. Rajagopalan and R. Aravind, “Importance
sampling Kalman filter for image estimation”, IEEE Signal Processing Letters,
vol. 14, pp. 453-456, 2007.
2. G. R. K. S. Subrahmanyam, A. N. Rajagopalan and R. Aravind, “Importance
sampling unscented Kalman filter for film-grain noise removal”, IEEE Multime-
dia (to appear).
B. Conference Papers
1. G. R. K. S. Subrahamanyam, A. N. Rajagopalan and R. Aravind, “A new exten-
sion of Kalman filter to non-Gaussian priors”, Indian Conference on Computer
Vision Graphics and Image Processing (ICVGIP’2006), pp. 162-171, 2006.
2. G. R. K. S. Subrahmanyam, A. N. Rajagopalan and R. Aravind, “Unscented
Kalman filter for image estimation in film-grain noise”, IEEE International Con-
ference on Image Processing (ICIP’2007), pp. IV-17 - IV-20, 2007.