Top Banner
Lecture 2: Image filtering CS6670: Computer Vision Noah Snavely Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
64

CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Aug 07, 2019

Download

Documents

phungtram
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Lecture 2: Image filtering

CS6670: Computer VisionNoah Snavely

Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm

Page 2: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Lecture 2: Image filtering

CS6670: Computer VisionNoah Snavely

Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm

Page 3: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Lecture 2: Image filtering

CS6670: Computer VisionNoah Snavely

Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm

Page 4: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

CS6670: Computer VisionNoah Snavely

Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm

Lecture 2: Image filtering

Page 5: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Reading

• Szeliski, Chapter 3.1-3.2

Page 6: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

What is an image?

Page 7: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

What is an image?

Digital Camera

The EyeSource: A. Efros

We’ll focus on these in this class

(More on this process later)

Page 8: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

What is an image?

• A grid (matrix) of intensity values

(common to use one byte per value: 0 = black, 255 = white)

=

255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 20 0 255 255 255 255 255 255 255

255 255 255 75 75 75 255 255 255 255 255 255

255 255 75 95 95 75 255 255 255 255 255 255

255 255 96 127 145 175 255 255 255 255 255 255

255 255 127 145 175 175 175 255 255 255 255 255

255 255 127 145 200 200 175 175 95 255 255 255

255 255 127 145 200 200 175 175 95 47 255 255

255 255 127 145 145 175 127 127 95 47 255 255

255 255 74 127 127 127 95 95 95 47 255 255

255 255 255 74 74 74 74 74 74 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

Page 9: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

• We can think of a (grayscale) image as a function, f, from R2 to R (or a 2D signal):– f (x,y) gives the intensity at position (x,y)

– A digital image is a discrete (sampled, quantized) version of this function

What is an image?

x

y

f (x, y)

snoop

3D view

Page 10: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Image transformations

• As with any function, we can apply operators to an image

• We’ll talk about a special kind of operator, convolution (linear filtering)

g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)

Page 11: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Question: Noise reduction• Given a camera and a still scene, how can

you reduce noise?

Take lots of images and average them!

What’s the next best thing?Source: S. Seitz

Page 12: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Image filtering

• Modify the pixels in an image based on some function of a local neighborhood of each pixel

5 14

1 71

5 310

Local image data

7

Modified image data

Some function

Source: L. Zhang

Page 13: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Linear filtering

• One simple version: linear filtering (cross-correlation, convolution)– Replace each pixel by a linear combination of its

neighbors

• The prescription for the linear combination is called the “kernel” (or “mask”, “filter”)

0.5

0.5 00

10

0 00

kernel

8

Modified image data

Source: L. Zhang

Local image data

6 14

1 81

5 310

Page 14: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Cross-correlation

This is called a cross-correlation operation:

Let be the image, be the kernel (of size 2k+1 x 2k+1), and be the output image

Page 15: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Convolution

• Same as cross-correlation, except that the kernel is “flipped” (horizontally and vertically)

• Convolution is commutative and associative

This is called a convolution operation:

Page 16: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Convolution

Adapted from F. Durand

Page 17: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Mean filtering

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

1 1 1

1 1 1

1 1 1 * =

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

Page 18: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Linear filters: examples

000

010

000

Original Identical image

Source: D. Lowe

* =

Page 19: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Linear filters: examples

000

001

000

Original Shifted left

By 1 pixel

Source: D. Lowe

* =

Page 20: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Linear filters: examples

Original

111

111

111

Blur (with a mean filter)

Source: D. Lowe

* =

Page 21: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Linear filters: examples

Original

111

111

111

000

020

000

-

Sharpening filter (accentuates edges)

Source: D. Lowe

=*

Page 22: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Sharpening

Source: D. Lowe

Page 23: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Smoothing with box filter revisited

Source: D. Forsyth

Page 24: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Gaussian Kernel

Source: C. Rasmussen

Page 25: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Gaussian filters

= 30 pixels= 1 pixel = 5 pixels = 10 pixels

Page 26: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Gaussian filter

• Removes “high-frequency” components from the image (low-pass filter)

• Convolution with self is another Gaussian

– Convolving two times with Gaussian kernel of width = convolving once with kernel of width

Source: K. Grauman

* =

Page 27: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Sharpening

Source: D. Lowe

Page 28: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Sharpening revisited• What does blurring take away?

original smoothed (5x5)

detail

=

sharpened

=

Let’s add it back:

original detail

+ α

Source: S. Lazebnik

Page 29: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Sharpen filter

Gaussianscaled impulseLaplacian of Gaussian

imageblurredimage unit impulse

(identity)

Page 30: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Sharpen filter

unfiltered

filtered

Page 31: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Convolution in the real world

Source: http://lullaby.homepage.dk/diy-camera/bokeh.html

Bokeh: Blur in out-of-focus regions of an image.

Camera shake

*=Source: Fergus, et al. “Removing Camera Shake from a Single Photograph”, SIGGRAPH 2006

Page 32: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Questions?

Page 33: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Edge detection

• Convert a 2D image into a set of curves

– Extracts salient features of the scene

– More compact than pixels

Page 34: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Origin of Edges

• Edges are caused by a variety of factors

depth discontinuity

surface color discontinuity

illumination discontinuity

surface normal discontinuity

Page 35: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Characterizing edges

• An edge is a place of rapid change in the image intensity function

imageintensity function

(along horizontal scanline) first derivative

edges correspond toextrema of derivativeSource: L. Lazebnik

Page 36: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

• How can we differentiate a digital image F[x,y]?

– Option 1: reconstruct a continuous image, f, then compute the derivative

– Option 2: take discrete derivative (finite difference)

1 -1

How would you implement this as a linear filter?

Image derivatives

-1

1

: :

Source: S. Seitz

Page 37: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

The gradient points in the direction of most rapid increase in intensity

Image gradient

• The gradient of an image:

The edge strength is given by the gradient magnitude:

The gradient direction is given by:

• how does this relate to the direction of the edge?Source: Steve Seitz

Page 38: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Image gradient

Page 39: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Effects of noise

Where is the edge?Source: S. Seitz

Noisy input image

Page 40: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Solution: smooth first

f

h

f * h

Source: S. Seitz

To find edges, look for peaks in

Page 41: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

• Differentiation is convolution, and convolution is associative:

• This saves us one operation:

Associative property of convolution

f

Source: S. Seitz

Page 42: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

2D edge detection filters

Gaussianderivative of Gaussian (x)

Page 43: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Derivative of Gaussian filter

x-direction y-direction

Page 44: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Side note: How would you compute a directional derivative?

= ?

+ =

(From vector calculus)

Directional deriv. is a linear combination of

partial derivatives

Page 45: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Derivative of Gaussian filter

x-direction y-direction

+ =

Page 46: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

The Sobel operator

• Common approximation of derivative of Gaussian

-1 0 1

-2 0 2

-1 0 1

1 2 1

0 0 0

-1 -2 -1

• The standard defn. of the Sobel operator omits the 1/8 term– doesn’t make a difference for edge detection

– the 1/8 term is needed to get the right gradient value

Page 47: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Sobel operator: example

Source: Wikipedia

Page 48: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Example

• original image (Lena)

Page 49: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Finding edges

gradient magnitude

Page 50: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

thresholding

Finding edges

where is the edge?

Page 51: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

• Check if pixel is local maximum along gradient direction

– requires interpolating pixels p and r

Non-maximum supression

Page 52: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

thresholding

Finding edges

Page 53: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

thinning(non-maximum suppression)

Finding edges

Page 54: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Canny edge detector

1. Filter image with derivative of Gaussian

2. Find magnitude and orientation of gradient

3. Non-maximum suppression

4. Linking and thresholding (hysteresis):– Define two thresholds: low and high

– Use the high threshold to start edge curves and the low threshold to continue them

Source: D. Lowe, L. Fei-Fei

MATLAB: edge(image,‘canny’)

Page 55: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Canny edge detector

• Still one of the most widely used edge detectors in computer vision

• Depends on several parameters:

J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 8:679-714, 1986.

: width of the Gaussian blur

high thresholdlow threshold

Page 56: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Canny edge detector

Canny with Canny with original

• The choice of depends on desired behavior

– large detects “large-scale” edges

– small detects fine edges

Source: S. Seitz

Page 57: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Scale space (Witkin 83)

• Properties of scale space (w/ Gaussian smoothing)

– edge position may shift with increasing scale ()

– two edges may merge with increasing scale

– an edge may not split into two with increasing scale

larger

Gaussian filtered signal

first derivative peaks

Page 58: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Questions?

• 3-minute break

Page 59: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Images as vectors

• Very important idea!

0

1

2D image Scanline (1D signal)

Vector

(A 2D, n x m image can be represented by a vector of length nm formed by concatenating the rows)

Page 60: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Multiplying row and column vectors

= ?

Page 61: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Filtering as matrix multiplication

What kind of filter is this?

Page 62: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Filtering as matrix multiplication

=

Page 63: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Another matrix transformation

1D Discrete cosine transform (DCT) basis

2D DCT basis

Page 64: CS6670: Computer Vision - cs.cornell.edu · • Properties of scale space (w/ Gaussian smoothing) –edge position may shift with increasing scale ( ) –two edges may merge with

Another matrix transformation

1D Discrete cosine transform (DCT) basis

2D DCT basis