Edges

Edges• Edges = jumps in brightness/color

Brightness jumps marked in white


• Important!– Give object outlines and shapes



• Important!– Give object outlines and shapes– Relatively robust not so affected by changes in lighting, camera, viewpoint…


Edges not so affected by lighting

Edges not so affected by lighting

Face outline persists, despite very different lighting

What do “edges” look like?B

right

ness

Row number

Brig

htne

ss

Column number

“Jumps” can be gradual or jagged

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

Brightness jumps are not always

obvious when you look at the brightness

profile

Detecting edges

• Want to detect jumps

brightness profiles at a jump (ideally)

Detecting edges



Of course should detect jumps inany orientation!

Detecting edges



Off course should detect jumps inany orientation!

100 200 300 400 500 600 700 800

50

100

150

200

250

300

350

400

450

500

550

Detecting edges


• Filters detect themselves


Eye detection:Convolve with eye filter

Detecting edges


• Filters detect themselves

To detect jumps, convolve image with jumps


Eye detection:Convolve with eye filter

• To detect jumps, convolve with jumps

Detecting edges


• If image has no jump, convolving should give zero (convolving a constant image should give 0)

Filter weights should sum to 0

In equations:

Detecting edges

,,

0 i j consti j

w I ,,

0 i ji j

w


• Mask weights should sum to 0

• To detect all jumps (dark-to-bright and bright-to-dark)…

Detecting edges


• Mask weights should sum to 0

• To detect all jumps (dark-to-bright and bright-to-dark)…

Use magnitude (absolute value) of convolution result

Detecting edges

Edge detection• To detect jumps along x

Convolve with [1, -1]• To detect jumps along y

Convolve with 11

Vertical jumpsHorizontal jumpsAfter taking absolute value of convolution result, only jumps stand out as bright spots.

Smooth image regions, where the brightness changes slowly, are dark.

, with [1, 1] *x xI I I 1

, with *1y yI I I

Edge Detection• To “detect edges”, decide how big a jump

must be to qualify– Choose threshold T– If magnitude of convolution is >T at some

pixel, mark it as edge pixel (“edgel”)

Marked (white) pixels have T is about 15% of

xI THorizontal edges

max xI

Edge detection

• How to detect edges in any orientation?

• Answer: compute

remember:

2 2 ,x y x yI I I I

1*

1yI I

[1, 1] *xI I

xI yI

2 2x yI I

2 2x yI I

Edgels: T = 12% of 2 2max x yI I Edgels: T = 25% of 2 2max x yI I

Edgels: T = 6% of 2 2max x yI I

2 2x yI I

Edgels: T = 12% of 2 2max x yI I Edgels: T = 25% of 2 2max x yI I

Edgels: T = 6% of 2 2max x yI I

Higher threshold: cleaner but more fragmented

Edge detection

• Many questions– We’re convolving with [-1 1]

Edge detection


(Overall sign isn’t important…)

Edge detection

• Many questions– We’re convolving with [-1 1] = [... 0 0 0 -1 1 0 0 …]

Edge detection

• Many questions– We’re convolving with [-1 1]…

Should we use [-1 -1 -1 -1 1 1 1 1 ] instead?

(This would detect jumps between large bright areas next to large dark areas)

Edge detection

• Many questions– We’re convolving with [-1 1]…

Should we use [-1 -1 -1 -1 1 1 1 1 ] instead?

(This would detect jumps between large bright areas next to large dark areas)

But not so good for

Edge detection


– Should we look for smoother, more gradual change?

[-0.02 -0.08 -0.06 0.06 0.08 0.02]

Edge detection


– Should we look for smoother, more gradual change?

[-0.02 -0.08 -0.06 0.06 0.08 0.02]Recall smooth brightness profile

for car image edge

Edge detection


– To detect long boundary, should we average along it?

Edge detection



0

0 -1 11-101-1

000

13

Detects jumps along x directionaverages over y direction

Edge detection



0

0 -1 11-101-1

000

13

Detects jumps along x directionaverages over y direction

Good for long low contrast boundaries,accurate orientation measurement

Note: edge detection and derivative

Original image:Blurry brightness transition

First derivativederivative has maximumat point of steepest rise




• To detect fast change in image, compute derivative.




• To detect fast change, compute derivative. same as convolving with [1 -1] !





1[1, 1] * j jjI I I

1 1 2... ...j j j jI I I I

1 1





1[1, 1] * j jjI I I

1 1 2... ...j j j jI I I I

1 1 1" "

" " 1j jI IdI

dj j j





1[1, 1] * j jjI I I

1 1 2... ...j j j jI I I I

1 1 1" "

" " 1j jI IdI

dj j j

Remember the flip in convolution

Convolution and Derivatives

• Convolving I with [1, -1] gives

• Convolving I with gives

xI

11

yI

Approximatederivativesin x and y




• Define gradient of I

xI

11

yI





• Define gradient of I– From calculus:

xI

11

yI


, , ,x x yf ff x y f f

y




• Define gradient of I– From calculus:

– Approximate image gradient

xI

11

yI


, , ,x x yf ff x y f f

y

, ,x yI i j I I

What is the gradient?

, ( ,0)I I kx y

Change


, (0, )I I kx y

Change


, ( 1, 2)I I k kx y

More Change

Less Change


, ( 1, 2)I I k kx y

More Change

Less Change

Gradient direction is perpendicular to edge


, ( 1, 2)I I k kx y

More Change

Less Change

Gradient direction is perpendicular to edge

Gradient Magnitude gives size of brightness jump

Gradient• Brightness change in direction

ˆ t = cos ,sin

x

yt

I I dx I dxt x dt y dt

=Chain rule:


ˆ t = cos ,sin

x

yt


=Chain rule: cos sinI I

x y

=


ˆ t = cos ,sin

x

yt



x y

=

ˆ,I Ix y

= t ˆ = I t


ˆ t = cos ,sin

x

yt



x y

=

ˆ,I Ix y

= t ˆ = I t

Can compute derivative in any direction from !


ˆ t = cos ,sin

x

yt

It

Chain rule:

• Dot product largest for parallel vectors…

ˆ = I t


ˆ t = cos ,sin

x

yt

It

Chain rule:

• Dot product largest for parallel vectors, so largest change

ˆ = I t

(largest ) for movement along directionIt

ˆ / t I I


ˆ t = cos ,sin

x

yt

It

Chain rule:

• Dot product largest for parallel vectors, so largest change

ˆ = I t

(largest ) for movement along directionIt

ˆ / t I I

I I/ I ISize of this largest change:

Edge detection and gradient

Conclusions

• Detect edges (big brightness jumps) by computing and thresholding

• Image lines perpendicular to

I

I I

But image data is messy!• Brightness profile

• Can we detect jump?

But image data is messy!• Brightness profile

• Can we detect jump?

• Derivative (convolution with [1 -1] )

• Where’s the jump???

• Problem: Brightness fluctuates a lot– Fluctuations are fast (moderate jumps from one pixel to next)

– Often transition at edge is smooth So on edge only moderate brightness jumps between pixels

Edge doesn’t stand out

• Derivative (convolution with [1 -1] )• Brightness profile

• Problem– Fluctuations are fast– Transition at edge is smooth

• Solution– First smooth image to reduce fast fluctuations– Then take derivative

• Brightness profile

Smoothed image


• Brightness profile

Derivative of smoothed image

Edge now stands out.


Smoothed edge detection

• Recommended strategy– Smooth first– Then convolve with “derivative” filters

' * *I D G I

Smoothing filter (e.g., Gaussian)

“derivative” filter (e.g., [1 -1])

image

Smoothed edge detection

• To make more efficient , use associativity of convolution

' * *I D G I

* *D G I“Derivative” of gaussian

D and G are small, convolving them is fast. Also, can just compute once, then apply to many images.

Smoothing and Differentiation• D * G

Derivative of Gaussian filters

Horizontal smoothed derivative Vertical smoothed derivative1

*1G

1, 1 *G

Smoothing and Differentiation• D * G

Derivative of Gaussian filters

Horizontal smoothed derivative Vertical smoothed derivative1

*1G

1, 1 *G Gx

“ ” Gy

“ ”

Other Smoothed Derivative Kernels

– Sobel kernels (more old fashioned)

(These already incorporate smoothing)

– Discrete directional derivatives

-1 0 1

-2 0 2-1 0 1

1 2 1

0 0 0-1 -2 -1

y

x

1 00 1

0 11 0

0 1 2-1 0 1 -2 -1 0

Smoothed gradient

As before, “smoothed gradient” gives “smoothed derivative" in any direction

' ' ' '* , *G GI Ix y

Conclusions

• First smooth, then take derivative

• To get gradient, do this in x & y directions

• To detect edges, threshold magnitude of smoothed gradient

Conclusions

• First smooth, then take derivative

• To get gradient, do this in x & y directions

• To detect edges, threshold magnitude of smoothed gradient

But how much smoothing should you do??Depends, no single answer…

• Amount of smoothing affects semantics of edges

• More smoothing gives: more significant edges, smoother curves, less noise, better detection… but less detail, worse edge localization

1 pixel 3 pixels 7 pixelsDerivatives after smoothing at three different scales

Smoothing + localization

• More smoothing makes it harder to tell exactly where edges are located.

But median filter?

Threshold

• What should the threshold be?

Threshold

• What should the threshold be?– No single answer– Also depends on the amount of smoothing

Threshold


• Smoothing reduces amount of change

Original image (1D)

Smoothed image

Threshold


• Smoothing reduces amount of change

Original image (1D)

Smoothed image

Note: transition blurs,Slope decreases

Threshold

• More smoothing smaller derivative sizes Gradient magnitude shrinks

Original image I (1D)0 10 20 30 40 50 60

0.006

0.007

0.008

0.009

0.01

0.011

0.012

0.013

0.014

Amount of smoothing(standard deviation of Gaussian)

max *xG Ix

Threshold

• More smoothing smaller derivative sizes Gradient magnitude shrinks

Original image I (1D)0 10 20 30 40 50 60

0.006

0.007

0.008

0.009

0.01

0.011

0.012

0.013

0.014

Amount of smoothing(standard deviation of Gaussian)

max *xG Ix

Gradient size measured on the edge

Threshold

• Conclusion– For more smoothing, use lower threshold

fine scalehigh threshold

coarse scale,high threshold

Using same threshold as before causes loss of detail

coarsescalelowthreshold

Origin of Edges

• Edges are caused by a variety of factors

depth discontinuity

surface color discontinuity

illumination discontinuity

surface normal discontinuity

Surface Boundaries

Sharp surface angles(normal discontinuity)

Change in material properties

Boundaries of lighting

shadow

Edge Detection Issues:

1) What smoothing scale (ie, width of Gaussian) + threshold on ?

2) Identify true edge points? Usually > threshold along fat sausage containing true edge line

3) We are really interested in finding curves (that could outline an object) How to link detected edge pixels into curves?

| I|

| I | at two different scales

| I |

Edge detection so far

1. Filter out noise Smooth with 2D Gaussian

2. Take derivative

change.fastest ofdirection theis

changes imagefast how tells

Gradient theis ,),(

22

JJ

JJJ

yJ

xJJJJ

yx

yx

IGJ

Edge detection so far

3. Find edges by thresholding J

Finding the Peak above threshold on fat “sausage,” not a line

Want to thin “sausage” of edge pixels to single curve

Technique used is called Non Maximum Suppression

I

We wish to mark a curve within the sausage where the magnitude is biggest. We can do this by looking for a maximum along a slice normal to the curve (non-maximum suppression). These points should form a curve.

Non-maximumsuppression

At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.

Edge Following

3) Link edge points into curves?

Hysteresis

• Drop-outs? – High threshold to start curves– Low threshold to continue.

longer edge curves!

Canny in MATLAB

E = edge(Image,’canny’); or

E = edge(Image,’canny’,Threshold, );

We still have to choose the smoothing scale and the threshold. By default, MATLAB takes .If you don’t provide the threshold, it tries to estimate a reasonable one.

1

Effect of (for Gaussian smoothing)

Canny with Canny with original

• The choice of depends on desired behavior– large detects large scale edges– small detects fine features

The Canny edge detector

• original image (Lena)


• norm of the gradient


• thresholding


• thinning• (non-maximum suppression)

Derivative filters: technicalities

• We use “derivative filters” to detect edges

• But images are really continuous functions

• Derivative filter should give good approximation to continuous derivative of original continuous image

Discrete Derivatives (1D)• Let = pixel size.


• Relate to continuous derivative by Taylor expansion

I x I x 1,1

2 2

2 ...2

dI d II x I xdx dx

I x I x dI Odx

Discrete Derivatives (1D)• Better to use odd size kernel!

• For size 3, need

Solve for derivative by subtracting…

, ,I x I x I x

2

...2x xxI x I x I I

2

...2x xxI x I x I I

xI

2

2 x

I x I xI O




, ,I x I x I x

2

...2x xxI x I x I I

2

...2x xxI x I x I I

xI

2

2 x

I x I xI O

Derivative filter [-1/2 0 1/2]




, ,I x I x I x

2

...2x xxI x I x I I

2

...2x xxI x I x I I

xI

2

2 x

I x I xI O

Derivative filter [-1/2 0 1/2]

Now no O( ).Better approximationto derivative

These even terms cancelled in the expression for xI

xI

Message

• Use odd size filters (length 3, 5, …)

Second Derivative (1D) (size 3, need )

2

...2x xxI x I x I I

2

...2x xxI x I x I I

, ,I x I x I x


2

2

...2x xxI x I x I I

22

2xx

I x I x I xI O

, ,I x I x I x

Add, divide through by

2

...2x xxI x I x I I


2nd derivative filter [1 -2 1]

No O( )since sumcancels odd powers of

2

...2x xxI x I x I I

2

...2x xxI x I x I I

22xx

I x I x I xI O

, ,I x I x I x

Discrete Derivative (1D)

Convolution filters (remember: flipped from correlation)

• First derivative: [1/2 0 -1/2]• Second derivative: [1 -2 1]

Note: [1 -2 1] = [1 -1] * [1 -1]

first derivative filter




Note: [1 -2 1] = [1 -1] * [1 -1]

As usual, can compute “second derivative” of an image by taking two first “derivatives”…




Note: [1 -2 1] = [1 -1] * [1 -1] First “derivative” filters, eg [-1 1]

As usual, can compute “second derivative” of an image by taking two first “derivatives”…

Second “derivative” filter

2

2

` ' ` '* *d I d d Idx dx dx

` ' ` '* *d d Idx dx

Discrete Derivative (1D)(size 5)

• For bigger filters, taylor expand to higher order

First derivative filter (size 5): 1 2 2 1012 3 3 12

Second derivative filters


• is the Laplacian

Gaussian Derivative of Gaussian

Example of 2nd derivative filter:

Laplacian of Gaussian

2 2

2 2

1, exp2 2

x yG x y

Gx

2G


• is the Laplacian12 *

1f

1 2 1 * f




2 2

2 2

1, exp2 2

x yG x y

Gx

2G

Approximate 2nd derivs



1f

1 2 1 * f 0 1 01 4 1 *0 1 0

f




2 2

2 2

1, exp2 2

x yG x y

Gx

2G



1f

1 2 1 * f 0 1 01 4 1 *0 1 0

f




Can approximate Laplacian of an image(or a Gaussian) by convolving with this filter

2 2

2 2

1, exp2 2

x yG x y

Gx

2G

2nd Derivative Filters

• Note: can also use these to detect edges

Image

First derivative

2nd derivative



Image

First derivative

2nd derivative

Edge is inflection point:2nd derivative = 0



• Can detect edges from zeros of 2nd derivative

Image

First derivative

2nd derivative

Edge is inflection point:2nd derivative = 0

Using 2nd Derivative Filters

• For 2D images, can use Laplacian-of-Gaussian to detect edges


2G

Laplacian

• at a pixel pixel darker than average of its neighbors

0 1 01 4 1 *0 1 0

I

2I2 0I

Laplacian


0 1 01 4 1 *0 1 0

I

2I2 0I

Image I

Laplacian


0 1 01 4 1 *0 1 0

I

2I2 0I

I

y

xEdge boundary between bright anddark regions

Bright region

Dark region

surface plot of IImage I

Laplacian


0 1 01 4 1 *0 1 0

I

2I

I

y


Bright region

Dark region


Pixels brighterthan neighbors

2 0I

2 0I

Laplacian


0 1 01 4 1 *0 1 0

I

2I

I

y


Bright region

Dark region


Pixels darkerthan neighbors

2 0I

2 0I

Laplacian


0 1 01 4 1 *0 1 0

I

2I

I

y


Bright region

Dark region


On edge

2 0I

2 0I


• For 2D images, can find edges by:

– Convolve image with a Laplacian-of-Gaussian (ie, compute Laplacian of smoothed image)

– Mark zero pixels (in practice, mark pixels on boundary between ‘+’ and ‘–’ regions)

• Since region boundaries are continuous curves, this always marks continuous curves!


• For 2D images, can find edges by:

– Convolve image with a Laplacian-of-Gaussian (ie, compute Laplacian of smoothed image)

– Mark zero pixels (in practice, mark pixels on boundary between ‘+’ and ‘–’ regions)

• Since region boundaries are continuous curves, this always marks continuous curves!Not so

good in practice

Laplacian


• Edge detection: strategy 2

0 1 01 4 1 *0 1 0

I

2I2 0I

Laplacian


• Edge detection: strategy 2– When image is smooth, pixels have brightness similar to their

neighbors and is small.

– Near edges, I changes rapidly and can be big.

0 1 01 4 1 *0 1 0

I

2I2 0I

2I

2I

Laplacian





So look for pixels where is large.

0 1 01 4 1 *0 1 0

I

2I2 0I

2I

2I

2I

Laplacian





So look for pixels where is large.

(But this doesn’t get the position of the edge right…)

0 1 01 4 1 *0 1 0

I

2I2 0I

2I

2I

2I

2I

Laplacian


0 1 01 4 1 *0 1 0

I

2I2 0I

I

y


Bright region

Dark region


Largest values forwhere surface most curved



Gaussian Smaller Gaussian


• Why?


• Why?

2

0 1 01 4 1 *0 1 0

G G


• Why?

2

0 1 0 0 1/ 4 01 4 1 * 4 1/ 4 0 1/ 4 *0 1 0 0 1/ 4 0

G G G G


• Why?

2

0 1 0 0 1/ 4 01 4 1 * 4 1/ 4 0 1/ 4 *0 1 0 0 1/ 4 0

G G G G

Convolution of an averaging filter with a Gaussian(which is another averaging filter)


• Why?

2

0 1 0 0 1/ 4 01 4 1 * 4 1/ 4 0 1/ 4 *0 1 0 0 1/ 4 0

G G G G

Convolution of an averaging filter with a Gaussian(which is another averaging filter)

Averaging twice is still averaging….So roughly the same as convolving with a single, bigger Gaussian

Caution:Derivatives are noisy

Higher derivatives are noisier

First derivative: black2nd derivative: red



Why?



Why?

Original signalI

fluctuations



Why?

Original signal

Correlate with [-1 1]

I

I’

fluctuations

Fluctuationsdouble in size



Why?

Original signal

Correlate with [-1 1]

Correlate again with [-1 1](2nd derivative)

I

I’

fluctuations

Fluctuationsdouble in size

I’’ Fluctuationsquadruple

Oriented Gaussian smoothing

Edges

Documents

jumps dark

jumps filters

jumps brightness profiles

viewpoint brightness

shapes brightness jumps

jagged6 brightness jumps

jumps mask weights

brightness changes