Transform-based Non-local Methods for Image Restoration IT530, Lecture Notes.

Transform-based Non-local Methods for Image Restoration

IT530, Lecture Notes

2

Non-local Techniques

Natural images have a great deal of redundancy: patches from different regions can be very similar

NL-Means: a non-local pixel-based method (Buades et al, 2005)

•Awate and Whitaker (PAMI 2007)•Popat and Picard (TIP 1998)•De-Bonet (MIT Tech report 1998)•Wang et al (IEEE SPL 2003) Difference

between patches

3

Transform-based Techniques

DCT coefficientsof a “typical” natural image

Observed property of natural images:Sparse coefficients

in Fourier/DCT/Wavelet domain

4

Transform-based Techniques

Project Patch onto Transform Basis (DCT/Wavelet etc.)

Transform Coefficients

Manipulate coefficients of noisy patch coefficients of

filtered patch

Get filtered patch by inversion of the transform from filtered coefficients

Repeat for all patches in sliding window fashion: final filtered image obtained by averaging overlapping filtered patches.

Noisy Patch

Clean Patch

DCT Coefficients (absolute value)

Transform basis

Patch matrix

Non-local transform based techniques

• Machine learning approach.

• Make use of non-local self-similarity at the patch level to learn good transform bases for denoising.

Non-local PCA

• PCA is the most basic transform to learn!• Assume Gaussian noise (mean zero, known

variance).• Given a patch (called “reference patch”) in a noisy

image, search for similar patches elsewhere in the image.

• Given such a collection of patches, compute their PCA bases.

• Compute the eigen-coefficients of the reference patch.

Non-local PCA

• Manipulate these eigen-coefficients (by hard-thresholding or Wiener filter).

• Reconstruct the reference patch.• Repeat the procedure for every patch in the

noisy image in sliding window fashion.• Average the multiple hypotheses that appear

at a pixel.

Ref: Muresan and Parks, “Adaptive principal components for image denoising”, ICIP 2003.

8

Criterion: Patch Similarity

),0(~

),0(~

2

1

NzPP

NzPP

ref

ref

Two patches: identical

modulo noise )n(~2

||PP|| 222

221

)2

n,2

x(GammaInc)n;x(F

22

Cumulative of chi-squared is incomplete gamma

function.

2

n3)

2

n;99.0(F

221

2

n3

2

||PP|| 2

2

221

A patch P1 is considered similar to patch Pref if their squared

difference is less than or equal to 22n3

Non-local PCA

),0(~, Nnnxy iiii Noisy patch

The l-th coefficient of yi – computed by projection onto the l-th local PCA bases (obtained from similar patches)

We are modelling as zero-mean Gaussian random variable with variance

lix

l

Two-stage non-local PCA

Ref: Zhang et al, Two stage image denoising by principal components analysis with local pixel grouping

BM3D PCA

KSVD

BM3D PCA

KSVD

BM3D PCA

KSVD

Time complexity

• The patches of size p x p are treated as vectors of size p2 x 1.

• Hence the covariance matrices are of size p2 x p2.

• Eigen-analysis will have a time complexity of O(p6) per patch (not counting the time to search for similar patches).

• This is quite expensive.

Non-local collaborative filtering: Block Matching in 3D (BM3D)

• The state of the art method today.• Based on the idea of non-local similarity at the

patch-level.• Given a reference patch in the noisy image,

this method again collects similar patches.• But this time, the similar patches and the

reference patch are arranged in the form of a 3D stack (of say some K patches in all).

Non-local collaborative filtering: Block Matching in 3D (BM3D)

• The stack is projected onto 3D transform bases (typically 3D DCT, or tensor product of 2D DCT and 1D Haar wavelets).

• The 3D transform coefficients are manipulated – usually by hard thresholding using the universal threshold .

• All the patches in the entire stack are reconstructed using an inverse 3D transform.

• This is repeated for every patch in the image. The multiple answers appearing at any pixel are averaged.

)log(2 2Kp

BM3D – second stage

• The preceding steps form the first stage of BM3D.• The output image of the first step is used to

compute patch similarities (this will be more robust than computing the similarities in the noisy image).

• Patches from the first-stage image are then appropriately assembled into a stack.

• Corresponding patches from the noisy image are assembled into a second stack.

BM3D – second stage

• 3D transform coefficients of both the stacks are computed.

• The second stack is denoised using Wiener filtering as follows:

• This is again repeated in sliding-window fashion with averaging.

noisy22onestage

2onestage

stagesecond cc

cc

Note: BM3D does allow individual patches from the patch to be independently filtered. It filters the similar patches from the stack collectively, assuming a dependence between them! This is the reason for its superior performance.

Results

• State of the art denoising method.

• Impressive results even at high noise levels.

• Residual images produced are very noisy.

• Preserves textures and fine details/edges very well.

24

NL-SVD: PSNR 30.88SSIM 0.882

Noisy PSNR: 22.13

BM3D1: PSNR 31.02SSIM 0.884

HOSVD: PSNR 31.53SSIM 0.897

BM3D2: PSNR 31.66SSIM 0.903

NLMeans: PSNR 29.42SSIM 0.821

KSVD: PSNR 30.762SSIM 0.877

25

NL-SVDBM3D1BM3D2HOSVDKSVD

26

NL-SVD: PSNR 30.187SSIM 0.801

Noisy PSNR: 22.13

BM3D1: PSNR 30.395SSIM 0.809

HOSVD: PSNR 30.491SSIM 0.814

BM3D2: PSNR 30.8SSIM 0.824

NLMeans: PSNR 28.91SSIM 0.753

KSVD: PSNR 30.36SSIM 0.803

27

NL-SVDBM3D1BM3D2HOSVDKSVD

28Gaussian Noise sigma = 15