Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University
Edge Preserving Filtering
Median Filter
Bilateral Filter
Shai Avidan
Tel-Aviv University
Slide Credits� (partial list)
• Rick Szeliski
• Steve Seitz
• Alyosha Efros
• Yacov Hel-Or
• Marc Levoy
• Bill Freeman
• Fredo Durand
• Sylvain Paris
A Gentle Introduction
to Bilateral Filtering
and its Applications
“Fixing the Gaussian Blur”:
the Bilateral Filter
Sylvain Paris – MIT CSAIL
Box Average
average
input
square neighborhood
output
normalized
box function
sum over
all pixels q
intensity at
pixel qresult at
pixel p
Equation of Box Average
��
��S
IBIBAq
qp qp )(][ �
0
Square Box Generates Defects
• Axis-aligned streaks
• Blocky results
input
output
unrelated
pixels
unrelated
pixels
related
pixels
Box Profile
pixel
position
pixel
weight
Strategy to Solve these
Problems
• Use an isotropic (i.e. circular) window.
• Use a window with a smooth falloff.
box window Gaussian window
Gaussian Blur
average
input
per-pixel multiplication
output*
input
box average
Gaussian blur
normalized
Gaussian function
Equation of Gaussian Blur
� ���
��S
IGIGBq
qp qp ||||][ �
Same idea: weighted average of pixels.
0
1
unrelated
pixels
unrelated
pixels
uncertain
pixels
uncertain
pixels
related
pixels
Gaussian Profile
pixel
position
pixel
weight��
���
��
2
2
2exp
2
1)(
����
xxG
size of the window
Spatial Parameter
� ���
��S
IGIGBq
qp qp ||||][ �
small � large �
input
limited smoothing strong smoothing
Blur Comes from
Averaging across Edges
*
*
*
input output
Same Gaussian kernel everywhere.
Bilateral Filter
No Averaging across Edges
*
*
*
input output
The kernel shape depends on the image content.
[Aurich 95, Smith 97, Tomasi 98]
space weight
not new
range weight
I
new
normalization
factor
new
Bilateral Filter Definition:
an Additional Edge Term
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
Same idea: weighted average of pixels.
Illustration a 1D Image
• 1D image = line of pixels
• Better visualized as a plot
pixel
intensity
pixel position
space
Gaussian Blur and Bilateral Filter
space rangenormalization
Gaussian blur
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
Bilateral filter[Aurich 95, Smith 97, Tomasi 98]
space
space
range
p
p
q
q
� ���
��S
IGIGBq
qp qp ||||][ �
Space and Range Parameters
• space �s : spatial extent of the kernel, size of
the considered neighborhood.
• range �r : “minimum” amplitude of an edge
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
How to Set the Parameters
Depends on the application. For instance:
• space parameter: proportional to image size
– e.g., 2% of image diagonal
• range parameter: proportional to edge
amplitude
– e.g., mean or median of image gradients
• independent of resolution and exposure
A Few
More Advanced
Remarks
Bilateral Filter Crosses Thin Lines• Bilateral filter averages across
features thinner than ~2�s
• Desirable for smoothing: more pixels = more robust
• Different from diffusion that stops at thin lines
close-up kernel
Iterating the Bilateral Filter
• Generate more piecewise-flat images
• Often not needed in computational
photo.
][ )()1( nn IBFI ��
input
1 iteration
2 iterations
4 iterations
Bilateral Filtering Color Images
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
� � � ���
���S
GGW
IBFq
qqp
p
p CCCqp ||||||||1
][rs ��
For gray-level images
For color images
intensity difference
color difference
The bilateral filter isThe bilateral filter is
extremely easy to adapt to your need.extremely easy to adapt to your need.
scalar
3D vector
(RGB, Lab)
input
output
Applications
• Image denoising
• HDR Compression
– Key idea:
– Break image into base and detail layers
– Compress base
– Recompose image
Fast Implementation: 3D Kernel
• Idea: represent image data such that the weights
depend only on the distance between points
[Paris and Durand 06]
pixel
intensity
pixel position
1D image
Plot
I = f ( x )
far in range
close in space
1st Step: Re-arranging
Symbols� � � �
� � � ��
�
�
�
���
���
S
S
IIGGW
IIIGGW
IBF
q
qpp
q
qqp
p
p
qp
qp
||||||
||||||1
][
rs
rs
��
��
� � � �
� � � � 1||||||
||||||][
rs
rs
�
�
�
�
���
���
S
S
IIGGW
IIIGGIBFW
q
qpp
q
qqppp
qp
qp
��
��
Multiply first equation by Wp
1st Step: Summary
• Similar equations
• No normalization factor anymore
• Don’t forget to divide at the end
� � � �
� � � � 1||||||
||||||][
rs
rs
�
�
�
�
���
���
S
S
IIGGW
IIIGGIBFW
q
qpp
q
qqppp
qp
qp
��
��
2nd Step: Higher-dimensional Space
pp
space
range
• “Product of two Gaussians” = higher dim.
Gaussian
2nd Step: Higher-dimensional Space
pp
space
range
• 0 almost everywhere, I at “plot location”
2nd Step: Higher-dimensional Space
pp
• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
2nd Step: Higher-dimensional Space
pp
• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
• Result is at “plot location”
�������������� �������
������� ��
������
����
������
�������
���������
������
������
�������
���������
������
New num. scheme:
• simple operations
• complex space
�������������� �������
���������� ����
������
����
�����������������
������������
����� �
������ ��
����� �
������ ��
Strategy:
downsampled
convolution
Conceptual view,
not exactly
the actual algorithm
The Algorithm
NEW IDEA : ‘Joint’ or ‘Cross’
Bilateral’ Petschnigg(2004) and
Eisemann(2004)Bilateral � two kinds of weights
NEW : get them from two kinds of images.
• Smooth image A pixels locally, but
• Limit to ‘similar regions’ of image B
Why do this? To get ‘best of both
images’
Ordinary Bilateral Filter
Bilateral � two kinds of weights, one image A :
� � � ���
���S
AAAGGW
ABFq
qqp
p
p qp ||||||1
][rs ��
cc
ss
Image A:
������������
��������
f(x)f(x)xx
‘Joint’ or ‘Cross’ Bilateral Filter
NEW: two kinds of weights, two images
� � � ���
���S
ABBGGW
ABFq
qqp
p
p qp ||||||1
][rs ��
cc
ss
A: Noisy, dim(ambient image)
cc
ss
B: Clean,strong
(Flash image)
Image A: Warm, shadows, but too Noisy(too dim for a good quick photo)(too dim for a good quick photo)
No-flash
Image B: Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)(flash: simple light, ALMOST no shadows)
MERGE BEST OF BOTH: apply
‘Cross Bilateral’ or ‘Joint Bilateral’
(it really is much better!)
Video Enhancement Using
Per Pixel Exposures (Bennett, 06)
From this video:
ASTA: Adaptive
SSpatio-
TTemporal
Accumulation Filter
ASTA
Replace pixel difference with a general dissimilarity measure
D(x,x)=0, D(x,y)=D(y,x)
Shot noise
A new dissimilarity measure
Instead of comparing pixel intensities, look at their local
spatial neighborhood
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
The Process for One Frame
The Process for One Frame
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
The Process for One Frame
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
(color: # avg’ pixels)
The Process for One Frame
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
Bilateral Filter Variant: Mostly Temporal
• FIFO for Histogram-stretched video
– Carry gain estimate for each pixel;
– Use future as well as previous values;
• Expanded Bilateral Filter Methods:
– Static scene? Temporal-only avg. works well
– Motion? Bilateral rejects outliers: no ghosts!
• Generalize: ‘Dissimilarity’ (not just || Ip – Iq ||2)
• Voting: spatial filter de-noises motion
Multispectral Bilateral Video Fusion
(Bennett,07)
• Result:
– Produces watchable result from unwatchable input
–– VERYVERY robust; accepts almost any dark video;
– Exploits temporal coherence to emulate
Low-light HDR video, without special equipment