Limitation of Imaging Technology

EE465: Introduction to Digital Image Processing 1

Limitation of Imaging Technology Two plagues in image acquisition

Noise interference Blur (motion, out-of-focus, hazy

weather) Difficult to obtain high-quality

images as imaging goes Beyond visible spectrum Micro-scale (microscopic imaging) Macro-scale (astronomical imaging)

What is Noise? Wiki definition: noise means any

unwanted signal One person’s signal is another

one’s noise Noise is not always random and

randomness is an artificial term Noise is not always bad (see

stochastic resonance example in the next slide)EE465: Introduction to Digital

Image Processing 2

Stochastic Resonance

EE465: Introduction to Digital Image Processing 3

no noise heavy noiselight noise

EE465: Introduction to Digital Image Processing 4

Image Denoising Where does noise come from?

Sensor (e.g., thermal or electrical interference)

Environmental conditions (rain, snow etc.)

Why do we want to denoise? Visually unpleasant Bad for compression Bad for analysis

EE465: Introduction to Digital Image Processing 5

electrical interference

Noisy Image Examples

thermal imaging

ultrasound imagingphysical interference

EE465: Introduction to Digital Image Processing 6

(Ad-hoc) Noise Modeling Simplified assumptions

Noise is independent of signal Noise types

Independent of spatial location Impulse noise Additive white Gaussian noise

Spatially dependent Periodic noise

EE465: Introduction to Digital Image Processing 7

Noise Removal Techniques Linear filtering Nonlinear filtering

Recall

Linear system

EE465: Introduction to Digital Image Processing 8

Image Denoising Introduction Impulse noise removal

Median filtering Additive white Gaussian noise

removal 2D convolution and DFT

Periodic noise removal Band-rejection and Notch filter

EE465: Introduction to Digital Image Processing 9

Impulse Noise (salt-pepper Noise)

Definition

Each pixel in an image has the probability of p/2 (0<p<1) being contaminated by either a white dot (salt) or a black dot (pepper)

WjHi

jiXjiY

1,1

),(0

255),(

with probability of p/2with probability of p/2with probability of 1-p

noisy pixels

clean pixels

X: noise-free image, Y: noisy image

Note: in some applications, noisy pixels are not simply black or white, which makes the impulse noise removal problem more difficult

EE465: Introduction to Digital Image Processing 10

Numerical ExampleP=0.1

128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128

X

128 128 255 0 128 128 128 128 128 128 128 128 128 128 0 128 128 128 128 0 128 128 128 128 128 128 128 128 128 128 128 128 0 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 0 128 128 128 128 255 128 128 128 128 128 128 128 128 128 128 128 128 128 255 128 128 128 128 128 128 128 255 128 128

Y

Noise level p=0.1 means that approximately 10% of pixels are contaminated bysalt or pepper noise (highlighted by red color)

EE465: Introduction to Digital Image Processing 11

MATLAB Command>Y = IMNOISE(X,'salt & pepper',p)

Notes:

The intensity of input images is assumed to be normalized to [0,1].If X is double, you need to do normalization first, i.e., X=X/255;If X is uint8, MATLAB would do the normalization automatically The default value of p is 0.05 (i.e., 5 percent of pixels are contaminated) imnoise function can produce other types of noise as well (you need to change the noise type ‘salt & pepper’)

EE465: Introduction to Digital Image Processing 12

Impulse Noise Removal Problem

Noisy image Y

filteringalgorithm

Can we make the denoised image X as closeto the noise-free image X as possible?

^

X̂denoisedimage

EE465: Introduction to Digital Image Processing 13

Median Operator Given a sequence of numbers {y1,

…,yN} Mean: average of N numbers Min: minimum of N numbers Max: maximum of N numbers Median: half-way of N numbersExample ]56,55,54,255,52,0,50[y

54)( ymedian

]255,56,55,54,52,50,0[y

sorted

EE465: Introduction to Digital Image Processing 14

y(n)

W=2T+1

)](),...,(),...,([)(ˆ TnynyTnymediannx

1D Median Filtering

… …

Note: median operator is nonlinear

MATLAB command: x=median(y(n-T:n+T));

EE465: Introduction to Digital Image Processing 15

Numerical Example]56,55,54,255,52,0,50[yT=1:

],56,55,54,255,52,0,50,[ 5650y

]55,55,54,54,52,50,50[ˆ x

BoundaryPadding

EE465: Introduction to Digital Image Processing 16

x(m,n)

W: (2T+1)-by-(2T+1) window

2D Median Filtering

)],(),...,,(),...,,(),...,,(),...,,([),(ˆ

TnTmyTnTmynmyTnTmyTnTmymediannmx

MATLAB command: x=medfilt2(y,[2*T+1,2*T+1]);

EE465: Introduction to Digital Image Processing 17

Numerical Example 225 225 225 226 226 226 226 226 225 225 255 226 226 226 225 226 226 226 225 226 0 226 226 255 255 226 225 0 226 226 226 226 225 255 0 225 226 226 226 255 255 225 224 226 226 0 225 226 226 225 225 226 255 226 226 228 226 226 225 226 226 226 226 226

0 225 225 226 226 226 226 226 225 225 226 226 226 226 226 226 225 226 226 226 226 226 226 226 226 226 225 225 226 226 226 226 225 225 225 225 226 226 226 226 225 225 225 226 226 226 226 226 225 225 225 226 226 226 226 226 226 226 226 226 226 226 226 226

Y X̂

Sorted: [0, 0, 0, 225, 225, 225, 226, 226, 226]

EE465: Introduction to Digital Image Processing 18

Image ExampleP=0.1

Noisy image Y X̂denoisedimage

3-by-3 window

EE465: Introduction to Digital Image Processing 19

Image Example (Con’t)

3-by-3 window 5-by-5 window

clean noisy(p=0.2)

EE465: Introduction to Digital Image Processing 20

Reflections What is good about median

operation? Since we know impulse noise appears

as black (minimum) or white (maximum) dots, taking median effectively suppresses the noise

What is bad about median operation? It affects clean pixels as well Noticeable edge blurring after median

filtering

EE465: Introduction to Digital Image Processing 21

Idea of Improving Median Filtering Can we get rid of impulse noise

without affecting clean pixels? Yes, if we know where the clean pixels

areor equivalently where the noisy pixels

are How to detect noisy pixels?

They are black or white dots

EE465: Introduction to Digital Image Processing 22

Median Filtering with Noise Detection

Noisy image Y

x=medfilt2(y,[2*T+1,2*T+1]);Median filtering

Noise detectionC=(y==0)|(y==255);

xx=c.*x+(1-c).*y;Obtain filtering results

EE465: Introduction to Digital Image Processing 23

Image Example

clean noisy(p=0.2)

w/onoisedetection

withnoisedetection

EE465: Introduction to Digital Image Processing 24

Image Denoising Introduction Impulse noise removal

Median filtering Additive white Gaussian noise

removal 2D convolution and DFT

Periodic noise removal Band-rejection and Notch filter

EE465: Introduction to Digital Image Processing 25

Additive White Gaussian Noise

DefinitionEach pixel in an image is disturbed by a Gaussian random variableWith zero mean and variance 2

WjHiNjiN

jiNjiXjiY

1,1),,0(~),(

),,(),(),(2

X: noise-free image, Y: noisy image

Note: unlike impulse noise situation, every pixel in the image contaminatedby AWGN is noisy

EE465: Introduction to Digital Image Processing 26

Numerical Example2 =1

X Y

128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128

128 128 129 127 129 126 126 128 126 128 128 129 129 128 128 127 128 128 128 129 129 127 127 128 128 129 127 126 129 129 129 128 127 127 128 127 129 127 129 128 129 130 127 129 127 129 130 128 129 128 129 128 128 128 129 129 128 128 130 129 128 127 127 126

EE465: Introduction to Digital Image Processing 27

MATLAB Command

>Y = IMNOISE(X,’gaussian',m,v)

>Y = X+m+randn(size(X))*v;or

Note:rand() generates random numbers uniformly distributed over [0,1]randn() generates random numbers observing Gaussian distributionN(0,1)

EE465: Introduction to Digital Image Processing 28

Image Denoising

Noisy image Y

filteringalgorithm

Question: Why not use median filtering?Hint: the noise type has changed.

X̂denoisedimage

EE465: Introduction to Digital Image Processing 29

f(n) h(n) g(n)

)()()()()()()( nhnfnfnhknfkhngk

- Linearity

- Time-invariant property

)()()()( 22112211 nganganfanfa

)()( 00 nngnnf

Linear convolution

1D Linear Filtering

See review section

EE465: Introduction to Digital Image Processing 30

jwnenfwF )()(

dwewFnf jwn)(

21)(

forward

inverse

Note that the input signal is a discrete sequence while its FT is a continuous function

)()( nhnf time-domain convolution

)()( wHwFfrequency-domain multiplication

Fourier Series

EE465: Introduction to Digital Image Processing 31

Filter Examples

0 0.5 1 1.5 2 2.5 3 3.50

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

LP HP

w

|H(w)|Low-pass (LP)

h(n)=[1,1]

|h(w)|=2cos(w/2)

High-pass (LP)

h(n)=[1,-1]

|h(w)|=2sin(w/2)

EE465: Introduction to Digital Image Processing 32

forward transform inverse transform

• Properties- periodic

- conjugate symmetric

)()( kyNky

)()( * kykNy

1D Discrete Fourier Transform

Proof:

1

0

)()(N

n

knNWnxky

1

0

)()(N

n

knNWkynx

1

0

1

0

)( )()(N

n

knNn

N

n

nNkNn kyWxWxNky

1

0

*1

0

)( )()(N

n

knNn

N

n

nkNNn kyWxWxkNyProof:

}2exp{NjWN

EE465: Introduction to Digital Image Processing 33

Matrix Representation of 1D DFT

FA

NNN

N

NN

aa

aa

..............................

......

1

111

1,

,1

2

NN

Nj

N

klNkl

WeW

WN

a

Re

Im

NWklN

N

llk Wx

Ny

1

1

N

llklk xay

1

DFT:

EE465: Introduction to Digital Image Processing 34

Fast Fourier Transform (FFT)*

Invented by Tukey and Cooley in 1965 Basic idea: divide-and-conquer

Reduce the complexity of N-point DFT from O(N2) to O(Nlog2N)

122/12

22/2

12

)12(12

2

22

1

0

mn

kmNm

k

mn

kmNm

mn

mkNm

mn

mkNm

N

n

knNnk

WxWWx

WxWxWxy

N

N/2-point DFT N/2-point DFT

N-point DFT

EE465: Introduction to Digital Image Processing 35

Filtering in the Frequency Domain

)()()( nhnfng

convolution in the time domain is equivalent to multiplication in the frequency domain

)()()( kHkFkG

f(n) h(n) g(n) F(k) H(k) G(k)

DFT

EE465: Introduction to Digital Image Processing 36

2D Linear Filtering

f(m,n) h(m,n) g(m,n)

),(),(),(),(),(,

nmfnmhlnkmflkhnmglk

2D convolution

MATLAB function: C = CONV2(A, B)

EE465: Introduction to Digital Image Processing 37

2D Filtering=Two Sequential 1D Filtering

Just as we have observed with 2D transform, 2D (separable) filtering can be viewed as two sequential 1D filtering operations: one along row direction and the other along column direction

The order of filtering does not matter )()()()(),( 1111 mhnhnhmhnmh

h1 : 1D filter

EE465: Introduction to Digital Image Processing 38

Numerical Exampleh1(m)=[1,1], h1(n)=[1,-1]1D filter

1111

)()( 11 nhmh

1111

)()( 11 mhnh

MATLAB command: >h1=[1,1];h2=[1,-1];>conv2(h1,h2)>conv2(h2,h1)

EE465: Introduction to Digital Image Processing 39

Fourier Series (2D case)

m n

nwmwjenmfwwF )(21

21),(),(

Note that the input signal is discrete while its FT is a continuous function

),(),( nmhnmf spatial-domain convolution

),(),( 2121 wwHwwFfrequency-domain multiplication

EE465: Introduction to Digital Image Processing 40

Filter Examples Low-pass (LP)

h1(n)=[1,1]

|h1(w)|=2cos(w/2)1D

h(n)=[1,1;1,1]

|h(w1,w2)|=4cos(w1/2)cos(w2/2)2D

w1

w2

|h(w1,w2)|

EE465: Introduction to Digital Image Processing 41

Image DFT Example

Original ray image X choice 1: Y=fft2(X)

EE465: Introduction to Digital Image Processing 42

Image DFT Example (Con’t)

choice 1: Y=fft2(X) choice 2: Y=fftshift(fft2(X))Low-frequency at the centerLow-frequency at four corners

FFTSHIFT Shift zero-frequency component to center of spectrum.

EE465: Introduction to Digital Image Processing 43

Gaussian Filter

)2

exp(),( 2

22

21

21 wwwwH

)2

exp(),( 2

22

nm

nmh

FT

>h=fspecial(‘gaussian’, HSIZE,SIGMA);MATLAB code:

EE465: Introduction to Digital Image Processing 44

(=1)PSNR=24.4dB

Image Example

PSNR=20.2dB

noisy

(=25)

denoised denoised

(=1.5)PSNR=22.8dB

Matlab functions: imfilter, filter2

45

Gaussian Filter=Heat Diffusion

2

2

2

2 ),,(),,(),,(),,(y

tyxIx

tyxItyxIttyxI

Linear Heat Flow Equation:

)()0,,(),,( tGyxItyxI

scale A Gaussian filterwith zero mean and variance of t

Isotropic diffusion:

46

Basic Idea of Nonlinear Diffusion*

x

y I(x,y)

image I

image I viewed as a 3D surface (x,y,I(x,y))Diffusion should be anisotropicinstead of isotropic

Experimental Results

EE465: Introduction to Digital Image Processing 47

(Gaussian filtering)PSNR=24.4dBPSNR=20.2dB

noisy

(=25)

linear diffusion

(TV filtering)PSNR=27.5dB

nonlinear diffusion

Hammer-Nail Analogy

EE465: Introduction to Digital Image Processing 48

Gaussian filter

median filter

salt-pepper/impulse noise

Gaussian noise

periodic noise???

EE465: Introduction to Digital Image Processing 49

Image Denoising Introduction Impulse noise removal

Median filtering Additive white Gaussian noise

removal 2D convolution and DFT

Periodic noise removal Band-rejection and Notch filter

EE465: Introduction to Digital Image Processing 50

Periodic Noise Source: electrical or electromechanical

interference during image acquistion Characteristics

Spatially dependent Periodic – easy to observe in frequency

domain Processing method

Suppressing noise component in frequency domain

EE465: Introduction to Digital Image Processing 51

Image Example

spatial

Frequency (note the four pairs of bright dots)

EE465: Introduction to Digital Image Processing 52

Band Rejection Filter

otherwise

WDwwWDwwH1

220),(

22

21

21

w1

w2

EE465: Introduction to Digital Image Processing 53

Image Example

Before filtering After filtering

Advanced Denoising Techniques*

EE465: Introduction to Digital Image Processing 54

IN

Is

IEIW

jijiN III ,,1

jijiS III ,,1

jijiE III ,1,

jijiW III ,1,

][,1

, IcIcIcIcII WWEESSNNtji

tji

WESNdIgc dd ,,,||),(||

Basic idea: from linear diffusion (equivalent to Gaussian filtering)to nonlinear diffusion (with implicit edge-stopping criterion)