Point Processing Filtering

7/24/2019 Point Processing Filtering

1/110

Point Processing & Filtering

CS194: Image Manipulation & Computational Photography

Alexei Efros, UC Berkeley, Fall 2015


2/110

Image Formation

f(x,y) = reflectance(x,y) * illumination(x,y)

Reflectance in [0,1], illumination in [0,inf]


3/110

Problem: Dynamic Range

1500

1

25,000

400,000

2,000,000,000

The real world is

High dynamic range


4/110

Long Exposure

10-6 106

10-6 106

Real world

Picture

0 to 255

High dynamic range


5/110

Short Exposure

10-6 106

10-6 106

Real world

Picture

0 to 255

High dynamic range


6/110


7/110

Simple Point Processing: Enhancement


8/110

Power-law transformations


9/110

Basic Point Processing


10/110

Negative


11/110

Log


12/110

Contrast Stretching


13/110

Image Histograms

Cumulative Histograms

s = T(r)


14/110

Histogram Equalization


15/110

Color Transfer [Reinhard, et al, 2001]

Erik Reinhard, Michael Ashikhmin, Bruce Gooch, Peter Shirley, ColorTransferbetweenImages. IEEE Computer Graphics and Applications, 21(5), pp. 3441. September 2001.
http://www.cs.bris.ac.uk/Publications/pub_master.jsp?id=2000476http://www.cs.bris.ac.uk/Publications/pub_master.jsp?id=2000476http://www.cs.bris.ac.uk/Publications/pub_master.jsp?id=2000476http://www.cs.bris.ac.uk/Publications/pub_master.jsp?id=2000476http://www.cs.bris.ac.uk/Publications/pub_master.jsp?id=2000476http://www.cs.bris.ac.uk/Publications/pub_master.jsp?id=2000476http://www.cs.bris.ac.uk/Publications/pub_master.jsp?id=2000476


16/110

Limitations of Point Processing

Q: What happens if I reshuffle all pixels within

the image?

A: Its histogram wont change. No point

processing will be affected


17/110

What is an image?

We can think of an image as a function, f, from R2to

R: f(x, y ) gives the intensityat position (x, y )

Realistically, we expect the image only to be defined over a

rectangle, with a finite range:

f: [a,b]x[c,d][0,1]

A color image is just three functions pasted together.

We can write this as a vector-valued function:

( , )

( , ) ( , )

( , )

r x y

f x y g x y

b x y

=


18/110

Images as functions


19/110

Sampling and Reconstruction


20/110

2006 Steve Marschner 20

Sampled representations

How to store and compute with continuous functions? Common scheme for representation: samples

write down the functions values at many points

[FvDFHfig.1

4.1

4b/Wolberg]


21/110


Reconstruction

Making samples back into a continuous function for output (need realizable method)

for analysis or processing (need mathematical method)

amounts to guessing what the function did in between

[FvDFHfig.1

4.1

4b/Wolberg]


22/110

1D Example: Audio

low high

frequencies


23/110


Sampling in digital audio

Recording: sound to analog to samples to disc Playback: disc to samples to analog to sound again

how can we be sure we are filling in the gaps correctly?


24/110


Sampling and Reconstruction

Simple example: a sign wave


25/110


Undersampling

What if we missed things between the samples? Simple example: undersampling a sine wave

unsurprising result: information is lost


26/110


Undersampling



surprising result: indistinguishable from lower frequency


27/110


Undersampling



surprising result: indistinguishable from lower frequency

also, was always indistinguishable from higher frequencies

aliasing: signals traveling in disguise as other frequencies


28/110

Aliasing in video

Slide by Steve Seitz


29/110

Aliasing in images


30/110

Whats happening?

Input signal:

x = 0:.05:5; imagesc(sin((2.^x).*x))

Plot as image:

Alias!

Not enough samples


31/110

Antialiasing

What can we do about aliasing?

Sample more often

Join the Mega-Pixel craze of the photo industry

But this cant go on forever

Make the signal less wiggly

Get rid of some high frequencies

Will loose information

But its better than aliasing


32/110


Preventing aliasing

Introduce lowpass filters: remove high frequencies leaving only safe, low frequencies

choose lowest frequency in reconstruction (disambiguate)


33/110


Linear filtering: a key idea

Transformations on signals; e.g.: bass/treble controls on stereo

blurring/sharpening operations in image editing

smoothing/noise reduction in tracking

Key properties linearity: filter(f + g) = filter(f) + filter(g)

shift invariance: behavior invariant to shifting the input

delaying an audio signal

sliding an image around

Can be modeled mathematically by convolution


34/110


Moving Average

basic idea: define a new function by averaging over asliding window

a simple example to start off: smoothing


35/110


Moving Average

Can add weights to our moving average Weights [, 0, 1, 1, 1, 1, 1, 0, ] / 5


36/110

Cross-correlation

This is called a cross-correlationoperation:

Let be the image, be the kernel (ofsize 2k+1 x 2k+1), and be the output

image

Can think of as a dot product between

local neighborhood and kernel for each pixel

In 2D: box filter


37/110

111

111

111

Slide credit: David Lowe (UBC)

],[ h

In 2D: box filter

Image filtering


38/110

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Credit: S. Seitz

],[],[],[,

lnkmflkhnmglk

++=

[.,.]g[.,.]f

Image filtering

111

111

111

],[ h

Image filtering


39/110

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]g[.,.]f

Image filtering

111

111

111

],[ h

Credit: S. Seitz

],[],[],[,

lnkmflkhnmglk

++=

Image filtering


40/110

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]g[.,.]f

Image filtering

111

111

111

],[ h

Credit: S. Seitz

Image filtering


41/110

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]g[.,.]f

Image filtering

111

111

111

],[ h

Credit: S. Seitz

Image filtering


42/110

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]g[.,.]f

Image filtering

111

111

111

],[ h

Credit: S. Seitz

Image filtering


43/110

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]g[.,.]f

Image filtering

111

111

111

],[ h

Credit: S. Seitz

?

Image filtering


44/110

0 10 20 30 30

50

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]g[.,.]f

Image filtering

111

111

111

],[ h

Credit: S. Seitz

?

Image filtering 111][h


45/110

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

[.,.]g[.,.]f

Image filtering111

111

111],[ h

Credit: S. Seitz

],[],[],[,

lnkmflkhnmglk

++=

Box Filter


46/110

What does it do?

Replaces each pixel with

an average of its

neighborhood

Achieve smoothing effect

(remove sharp features)

111

111

111

Slide credit: David Lowe (UBC)

],[ h

Box Filter


47/110

Linear filters: examples

Original

111

111111

Blur (with a meanfilter)

Source: D. Lowe

=


48/110

Practice with l inear filters

000

010000

Original

?

Source: D. Lowe


49/110


000

010000

Original Filtered(no change)

Source: D. Lowe


50/110


000

001000

Original

?

Source: D. Lowe


51/110


000

100000

Original Shifted leftBy 1 pixel

Source: D. Lowe


52/110

Other filters

-101

-202

-101

Vertical Edge

(absolute value)

Sobel

Q?


53/110

Other filters

-1-2-1

000

121

Horizontal Edge

(absolute value)

Sobel

Back to the box filter


54/110

Back to the box filter


55/110


Moving Average

Can add weights to our moving average Weights [, 0, 1, 1, 1, 1, 1, 0, ] / 5


56/110


Weighted Moving Average

bell curve (gaussian-like) weights [, 1, 4, 6, 4, 1, ]


57/110


Moving Average In 2D

What are the weights H?0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0


Gaussian filtering


58/110

60

Gaussian filtering

A Gaussian kernel gives less weight to pixels further from the center

of the window

This kernel is an approximation of a Gaussian function:

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

1 2 1

2 4 2

1 2 1


Mean vs Gaussian filtering


59/110

61

Mean vs. Gaussian fil tering


Important filter: Gaussian


60/110

62

Weight contributions of neighboring pixels by nearness

0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.022

0.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003

5 x 5, = 1

Slide credit: Christopher Rasmussen

po a e Gauss a

Gaussian Kernel


61/110

63

Gaussian Kernel

Standard deviation : determines extent of smoothing

Source: K. Grauman

= 2 with 30 x 30kernel

= 5 with 30 x 30kernel


62/110

Gaussian filters

= 30 pixels= 1 pixel = 5 pixels = 10 pixels

Choosing kernel width


63/110

65

Choosing kernel width

The Gaussian function has infinite support, but discrete filters

use finite kernels

Source: K. Grauman

Practical matters


64/110

66

How big should the filter be?

Values at edges should be near zero

Rule of thumb for Gaussian: set filter half-width to about 3

Practical matters

Side by Derek Hoiem

Cross-correlation vs Convolution


65/110

Cross-correlation vs. Convolution

cross-correlation :

A convolutionoperation is a cross-correlation where the filter is

flipped both horizontally and vertically before being applied to

the image:

It is written:

Convolution is commutativeand associative



66/110

Convolution

Adapted from F. Durand


67/110


Convolution is nice!

Notation: Convolution is a multiplication-like operation

commutative

associative

distributes over addition

scalars factor out

identity: unit impulse e= [, 0, 0, 1, 0, 0, ]

Conceptually no distinction between filter and signal

Usefulness of associativity often apply several filters one after another: (((a* b1) * b2) * b3)

this is equivalent to applying one filter: a * (b1* b2* b3)


68/110

Gaussian and convolution

Removes high-frequency components fromthe image (low-pass filter)

Convolution with self is another Gaussian

Convolving twice with Gaussian kernel of width

= convolving once with kernel of width

Source: K. Grauman

* =

Image half-sizing


69/110

Image half sizing

This image is too big to

fit on the screen. How

can we reduce it?

How to generate a half-sized version?

Image sub-sampling


70/110

Image sub sampling

Throw away every other row and

column to create a 1/2size image- calledimage sub-sampling

1/4

1/8


Image sub-sampling


71/110

Image sub sampling

1/4 (2x zoom) 1/8 (4x zoom)

Aliasing! What do we do?

1/2


Sampling an image


72/110

Sampling an image

Examples of GOOD sampling

Undersampling


73/110

Undersampling

Examples of BAD sampling -> Aliasing

Gaussian (lowpass) pre-filtering


74/110

Gaussian (lowpass) pre filtering

G 1/4

G 1/8

Gaussian 1/2

Solution: filter the image, thensubsample Filter size should double for each size reduction. Why?


Subsampling with Gaussian pre-filtering


75/110

Subsampling with Gaussian pre filtering

G 1/4 G 1/8Gaussian 1/2


Compare with...


76/110

Compare with...

1/4 (2x zoom) 1/8 (4x zoom)1/2


Gaussian (lowpass) pre-filtering


77/110

Gaussian (lowpass) pre filtering

G 1/4

G 1/8

Gaussian 1/2

Solution: filter the image, thensubsample Filter size should double for each size reduction. Why?

How can we speed this up? Slide by Steve Seitz

Image Pyramids


78/110

Image Pyramids

Known as a Gaussian Pyramid [Burt and Adelson, 1983]

In computer graphics, a mip map [Williams, 1983]

A precursor to wavelet transform



79/110

A bar in the

big images is ahair on the

zebras nose;

in smaller

images, a

stripe; in the

smallest, theanimals nose

Figure from David Forsyth

Gaussian pyramid construction


80/110

py

filter mask

Repeat Filter

SubsampleUntil minimum resolution reached

can specify desired number of levels (e.g., 3-level pyramid)

The whole pyramid is only 4/3 the size of the original image!Slide by Steve Seitz

What are they good for?


81/110

y g

Improve Search Search over translations

Classic coarse-to-fine strategy

Search over scale

Template matching

E.g. find a face at different scales


82/110

Taking derivative by convolution

Partial derivatives with convolution


83/110

For 2D function f(x,y), the partial derivative is:

For discrete data, we can approximate using finitedifferences:

To implement above as convolution, what would be the

associated filter?

),(),(lim

),(

0

yxfyxf

x

yxf +=

1

),(),1(),( yxfyxf

x

yxf +

Source: K. Grauman

Partial derivatives of an image


84/110

g

Which shows changes with respect to x?

-11

1-1

or-1 1

x

yxf

),(

y

yxf

),(

Finite difference filters


85/110

Other approximations of derivative filters exist:

Source: K. Grauman

Image gradient


86/110

The gradient points in the direction of most rapid increase

in intensity

g g

The gradient of an image:

The gradient direction is given by

Source: Steve Seitz

The edge strengthis given by the gradient magnitude

How does this direction relate to the direction of the edge?

Image Gradient


87/110

g

x

yxf

),(

y

yxf

),(

Effects of noise


88/110

Consider a single row or column of the image Plotting intensity as a function of position gives a signal

Where is the edge?Source: S. Seitz

Solution: smooth first


89/110

To find edges, look for peaks in )( gfdx

d

f

g

f * g

)( gfdxd

Source: S. Seitz

Derivative theorem of convolution


90/110

This saves us one operation:

Thisimagecannotcurrentlybedisplayed.

Derivative of Gaussian filter


91/110

* [1 -1] =

Derivative of Gaussian filter


92/110

Which one finds horizontal/vertical edges?

x-direction y-direction

Example


93/110

input image (Lena)

Compute Gradients (DoG)


94/110

X-Derivative ofGaussian

Y-Derivative ofGaussian

Gradient Magnitude

Get Orientation at Each Pixel


95/110

Threshold at minimum level

Get orientation

theta = atan2(-gy, gx)

MATLAB demo


96/110

i m = i m2doubl e( i mr ead( f i l emane) ) ;g = f speci al ( ' gaussi an' , 15, 2) ;i magesc( g) ;sur f l ( g) ;

gi m = conv2( i m, g, ' same' ) ;i magesc( conv2( i m, [ - 1 1] , ' same' ) ) ;i magesc( conv2( gi m, [ - 1 1] , ' same' ) ) ;dx = conv2( g, [ - 1 1] , ' same' ) ;

Sur f l ( dx) ;i magesc( conv2( i m, dx, ' same' ) ) ;

Practical matters


97/110

What is the size of the output?

MATLAB: filter2(g, f, shape) or conv2(g,f,shape) shape= full: output size is sum of sizes of f and g

shape= same: output size is same as f

shape= valid: output size is difference of sizes of f and g

f

gg

gg

f

gg

gg

f

gg

gg

full same valid

Source: S. Lazebnik

Practical matters


98/110

What about near the edge?

the filter window falls off the edge of the image need to extrapolate

methods:

clip filter (black)

wrap around copy edge

reflect across edge

Source: S. Marschner

Practical mattersQ?


99/110

methods (MATLAB):

clip filter (black): imfilter(f, g, 0)

wrap around: imfilter(f, g, circular) copy edge: imfilter(f, g, replicate)

reflect across edge: imfilter(f, g, symmetric)

Source: S. Marschner

Review: Smoothing vs. derivative filters


100/110

Smoothing filters

Gaussian: remove high-frequency components;low-pass filter

Can the values of a smoothing filter be negative?

What should the values sum to?

One:constant regions are not affected by the filter

Derivative filters Derivatives of Gaussian

Can the values of a derivative filter be negative? What should the values sum to?

Zero:no response in constant regions

High absolute value at points of high contrast

Template matching


101/110

Goal: find in image

Main challenge: What is agood similarity ordistance measure

between two patches? Correlation

Zero-mean correlation

Sum Square Difference

Normalized Cross Correlation

Side by Derek Hoiem

Matching with filters


102/110

Goal: find in image

Method 0: filter the image with eye patch

Input Filtered Image

],[],[],[,

lnkmflkgnmhlk

++=

What went wrong?

f = image

g = filter

Side by Derek Hoiem



103/110

Goal: find in image

Method 1: filter the image with zero-mean eye

Input Filtered Image (scaled) Thresholded Image

)],[()],[(],[,

lnkmgflkfnmhlk

++=

True detections

Falsedetections

mean of f



104/110

Goal: find in image

Method 2: SSD

Input 1- sqrt(SSD) Thresholded Image

2

,

)],[],[(],[ lnkmflkgnmhlk

++=

True detections



105/110

Can SSD be implemented with linear filters?2

,


++=

Side by Derek Hoiem



106/110

Goal: find in image

Method 2: SSD

Input 1- sqrt(SSD)

2

,


++=

Whats the potential

downside of SSD?

Side by Derek Hoiem



107/110

Goal: find in image

Method 3: Normalized cross-correlation

5.0

,

2

,

,

2

,

,

)],[()],[(

)],[)(],[(

],[

++

++=

lk

nm

lk

nm

lk

flnkmfglkg

flnkmfglkg

nmh

mean image patchmean template

Side by Derek Hoiem



108/110

Goal: find in image


Input Normalized X-Correlation Thresholded Image

True detections



109/110

Goal: find in image


Input Normalized X-Correlation Thresholded Image

True detections

Q: What is the best method to use?


110/110

A: DependsZero-mean filter: fastest but not a great

matcher

SSD: next fastest, sensitive to overallintensity

Normalized cross-correlation: slowest,

invariant to local average intensity andcontrast

Point Processing Filtering

Documents