Filter factor analysis of an iterative multilevel regularizing method Marco Donatelli Department of Physics and Mathematics University of Insubria
Filter factor analysis of an iterative multilevelregularizing method
Marco Donatelli
Department of Physics and MathematicsUniversity of Insubria
Outline
1 Restoration of blurred and noisy imagesThe model problemProperties of the PSFIterative regularization methods
2 Multigrid regularizationMultigrid methodsIterative Multigrid regularizationComputational CostFilter factor analysis of the TL
3 Numerical experiments
4 Conclusions
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 2 / 30
Restoration of blurred and noisy images
Outline
1 Restoration of blurred and noisy imagesThe model problemProperties of the PSFIterative regularization methods
2 Multigrid regularizationMultigrid methodsIterative Multigrid regularizationComputational CostFilter factor analysis of the TL
3 Numerical experiments
4 Conclusions
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 3 / 30
Restoration of blurred and noisy images The model problem
Image restoration with Boundary Conditions
Using Boundary Conditions (BCs), the restored image f is obtainedsolving: (in some way ...)
Af = g + ξ
• g = blurred image,
• ξ = noise (random vector),
• A = two-level matrix depending on the point spread function (PSF)and the BCs.
The PSF is the observation of a single point (e.g., a star in astronomy).
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 4 / 30
Restoration of blurred and noisy images The model problem
Coefficient matrix structure
The matrix-vector product computed in O(n2 log(n)) ops for n × n imageswhile the inversion costs O(n2 log(n)) ops only in the periodic case.
BCs A
Dirichlet Toeplitzperiodic circulant
Neumann (reflective) Toeplitz + Hankelanti-reflective Toeplitz + Hankel
If the PSF is symmetric with respect to each direction:
BCs A
Neumann (reflective) DCT IIIanti-reflective DST I + low-rank
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 5 / 30
Restoration of blurred and noisy images Properties of the PSF
Generating function of PSF
• The eigenvalues of A(z) are about a uniform sampling of z .
PSF Generating function z(x)
• The ill-conditioned subspace is mainly constituted by themiddle/high frequencies.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 6 / 30
Restoration of blurred and noisy images Iterative regularization methods
Iterative regularization methods
Semi-convergence behavior
Some iterative methods (Landweber, CGNE, . . . ) have regularizationproperties: the restoration error firstly decreases and then increases.
Example
0 50 100 150 200 250 30010
−1
100
101
ReasonMarco Donatelli (University of Insubria) An iterative multilevel regularization method 7 / 30
Multigrid regularization
Outline
1 Restoration of blurred and noisy imagesThe model problemProperties of the PSFIterative regularization methods
2 Multigrid regularizationMultigrid methodsIterative Multigrid regularizationComputational CostFilter factor analysis of the TL
3 Numerical experiments
4 Conclusions
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 8 / 30
Multigrid regularization Multigrid methods
The algorithm
The choices
1 We apply only the pre-smoother simply called smoother.
2 Let Ri and Pi be the restriction and the prolongation operators atthe level i , respectively.
3 We use the Galerkin approach• Pi = RT
i
• Ai+1 = RAiRTi
4 Coarser grid of size 8 × 8 independent of the size of the finer grid.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 9 / 30
Multigrid regularization Multigrid methods
The Algebraic Multigrid (AMG)
• The AMG uses only information on the coefficient matrix.
• Different classic smoothers have similar behavior:in the initial iterations they are not able to reduce effectively the errorin the subspace generated by the eigenvectors associated to smalleigenvalues (ill-conditioned subspace)
⇓• To obtain a fast solver, the restriction is chosen in order to project
the error equation in such subspace.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 10 / 30
Multigrid regularization Multigrid methods
Image deblurring and Multigrid
• In the image deblurring the ill-conditioned subspace is related to highfrequencies, while the well-conditioned subspace is generated by lowfrequencies.
• In order to obtain a fast convergence the algebraic multigrid projectsin the high frequencies where the noise “lives” =⇒ noise explosionalready at the first iteration (it requires Tikhonov regularization[Donatelli, NLAA, 12 (2005), pp. 715–729]).
• In this case the low-pass filter projects in the well-conditionedsubspace (low frequencies) =⇒ it is slowly convergent but it can be agood iterative regularizer.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 11 / 30
Multigrid regularization Multigrid methods
Multigrid for structured matrices
Preserve the structure
• In order to apply recursively the MGM, it is necessary to keep thesame structure at each level (Toeplitz, . . . ).
• For every structure arising from the proposed BCs, there existprojectors that preserve the same structure.
Ri = KNiANi
(p), where
• KNi∈ R
Ni4×Ni is the cutting matrix that preserves the structure at
the lower level.
• p(x , y) is the generating function of the projector, which selects thesubspace where to project the linear system.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 12 / 30
Multigrid regularization Multigrid methods
Multigrid, structured matrices, and images
The cutting matrix Kniin 1D
circulant Toeplitz&DST − I DCT − III
[
1 01 0 ... ...
1 0
] [
0 1 00 1 0... ... ...
0 1 0
] [
1 1 01 1 0... ... ...
0 1 1
]
Low-pass filter: Low frequencies projection ⇒ noise reduction
2D ↔ p(x , y) = (1 + cos(x))(1 + cos(y))
ց Full weighting ր Bilinear interpolation
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 13 / 30
Multigrid regularization Iterative Multigrid regularization
Iterative multigrid regularization
The Multigrid as an iterative regularization method
If we have an iterative regularization method we can improve itsregularizing properties and/or accelerate its convergence using it assmoother in a Multigrid algorithm.
Regularization
The regularization properties of the smoother are preserved since it iscombined with a low-pass filter.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 14 / 30
Multigrid regularization Iterative Multigrid regularization
Two-Level (TL) regularization
Idea: project into the low frequencies and then apply an iterativeregularization method.
TL as a specialization of TGM
Smoother: iterative regularization method
Projector: low-pass filter
TL Algorithm
1 No smoothing at the finer level
2 At the coarser level to apply one step of the smoother instead ofto solve directly the linear system
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 15 / 30
Multigrid regularization Iterative Multigrid regularization
Multigrid regularization (applying recursively the TL)
V-cycle
Using a larger number of recursive calls (e.g. W -cycle), the algorithm“works” more in the well-conditioned subspace, but it is more difficult todefine an early stopping criterium.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 16 / 30
Multigrid regularization Computational Cost
Computational Cost
Assumptions: n × n images and m × m PSFs with m ≪ n.
• Let S(n) be the computational cost of one smoother iteration.
• The computational cost of one iteration of our multigridregularization method with γ recursive calls is
C (γ, n) ≈
13S(n), γ = 1S(n), γ = 23S(n), γ = 3
• if m ≈ n then S(n) = O(n2 log(n)).
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 17 / 30
Multigrid regularization Filter factor analysis of the TL
Filter factor of the Landweber method
• Imposing P-BCs A = Cn(z): A is a circulant matrix of size ngenerated by the function z .
• A = FnDn(z)FHn , where Fn = [eijxk ]n−1
k,j=0/√
n is the DFT matrix and
Dn(z) = diag([f (xk)]n−1k=0) with xk = 2πk
n.
• Taking x0 = 0 the jth approximation of f is
xj = Fn
j−1∑
i=0
(I − Dn(|z |2))iDn(z)FHn b = Cn(φj)C
−1n (z)b
where φj(x) = 1 − (1 − |z(x)|2)j , x ∈ (0, 2π] is the filter factor.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 18 / 30
Multigrid regularization Filter factor analysis of the TL
Filter factor of the TL method
• For TL with Landweber as smoother xj = Bnb with
Bn = Cn(p)KTn C n
2(g)KnCn(r),
where g(x) = 1−(1−|z(x)|2)j
z(x) , x ∈ (0, 2π], Kn is the cutting matrix andr , p and z are restriction, prolongation and PSF function at thecoarser level respectively.
• Bn = FnΠTn WnΠnF
Hn , where Πn is a permutation matrix and Wn is
the diagonal block matrix of size (n/2) × (n/2) with blocks ofdimension 2 × 2. For k = 0, . . . , n/2 − 1, the k-th diagonal block isgiven by
W(k)n =
1
2g(x2k)
[
p(xk)p(x(k+n/2))
]
[
r(xk) r(x(k+n/2))]
.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 19 / 30
Multigrid regularization Filter factor analysis of the TL
Filter factor of the TL method 2
• The block W(k)n has rank 1 and the nontrivial null eigenvalue λk is
λk =1
2g(x2k)
(
(pr)(xk) + (pr)(x(k+n/2)))
.
• The eigenvector associated to the null eigenvalue is
r(xk)
r(x(k+n/2))F
(k+n/2)n − F
(k)n .
This should be an high frequency (to filtering) ⇒ it provides acondition to choose r : e.g. nonnegative and decreasing in [0, pi ].
• The eigenvector associated to λk defines an analogous condition for p.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 20 / 30
Multigrid regularization Filter factor analysis of the TL
Comparison TL vs Landweber
Focus on the high frequencies for the filter factors of TL and Landweberfor j = 1000
0 1 2 3 4 5 60
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1x 10
−4
Landweber
TL
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 21 / 30
Multigrid regularization Filter factor analysis of the TL
Noise −→ 0 ?
In the noise free case the TL method does not compute the exact solution.
How to recover the high frequencies in the noise free case is a work inprogress . . .
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 22 / 30
Numerical experiments
Outline
1 Restoration of blurred and noisy imagesThe model problemProperties of the PSFIterative regularization methods
2 Multigrid regularizationMultigrid methodsIterative Multigrid regularizationComputational CostFilter factor analysis of the TL
3 Numerical experiments
4 Conclusions
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 23 / 30
Numerical experiments
An airplane
• Periodic BCs
• Gaussian PSF (A spd)
• noise = 1%
OriginalImage
Inner part 128 × 128 Observed image Restored with MGM
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 24 / 30
Numerical experiments
Restoration error: noise = 1%
ej = ‖f − f(j)‖2/‖f‖2 restoration error at the j-th iteration.
Minimum restoration error
Method minj=1,...
(ej) arg minj=1,...
(ej)
CG 0.1215 4
Richardson 0.1218 8
TL(CG) 0.1132 8
TL(Rich) 0.1134 16
MGM(Rich, 1) 0.1127 12
MGM(Rich, 2) 0.1129 5
CGNE 0.1135 178
RichNE 0.1135 352
Relative error vs. number of iterations
0.2Marco Donatelli (University of Insubria) An iterative multilevel regularization method 25 / 30
Numerical experiments
Noise = 10%
For CG and Richardson it is better to resort to normal equations.
Minimum restoration error
Method minj=1,...
(ej) arg minj=1,...
(ej)
CGNE 0.1625 30
RichNE 0.1630 59
TL(CGNE) 0.1611 48
TL(RichNE) 0.1613 97
MGM(RichNE,1) 0.1618 69
MGM(RichNE,2) 0.1621 26
MGM(Rich,1) 0.1648 3
MGM(Rich,2) 0.1630 1
Relative error vs. number of iterations
0.21Marco Donatelli (University of Insubria) An iterative multilevel regularization method 26 / 30
Conclusions
Outline
1 Restoration of blurred and noisy imagesThe model problemProperties of the PSFIterative regularization methods
2 Multigrid regularizationMultigrid methodsIterative Multigrid regularizationComputational CostFilter factor analysis of the TL
3 Numerical experiments
4 Conclusions
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 27 / 30
Conclusions
Possible generalizations
• Include the nonnegativity constraints.
• Improve the projector:
p(x , y) = (1 + cos(x))α(1 + cos(y))α, α ∈ N+.
• The γ regularization:
varying γ, the proposed multigrid is a direct (one step)regularization method with regularization parameter γ.
The computational cost increases with γ but not so much (e.g.γ = 8 ⇒ O(N1.5) where N = n2).
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 28 / 30
Conclusions
Summarizing . . . multigrid regularization method
• It is a general framework which can be used to improve theregularization properties of an iterative regularizing method.
• It leads to a smaller relative error and a flatter error curve withrespect to the smoother applied alone.
• It is fast and usually it obtains a good restored image also withoutresorting to normal equations.
• It can be combined with other techniques and it can lead to severalgeneralizations (e.g., nonnegativity constraints).
ReferenceM. Donatelli and S. Serra Capizzano, On the regularizing power ofmultigrid-type algorithms, SIAM J. Sci. Comput., 27–6 (2006) pp.2053–2076.
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 29 / 30
Conclusions
Future work
Theoretical
• A complete theoretical analysis of the regularization properties.
Applications:
• strictly nonsymmetric PSFs.
• Combination with techniques for edge enhancing (Wavelet, TotalVariation, . . . ).
Numerics/Simulations:
• A complete experimentation with all the proposed BCs (multigridmethods already exist for the arising matrices, see [Arico, Donatelli,Serra Capizzano, SIMAX, Vol. 26–1 pp. 186–214.]).
Marco Donatelli (University of Insubria) An iterative multilevel regularization method 30 / 30