Dr. J. Shanbehzadeh Shanbehzadeh@gmail M.HosseinKord

Dr. J. [email protected]

M.HosseinKord

Science and Research Branch of Islamic Azad University

Lecture 07 – Pyramids

Machine Vision

1/49 slides

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Table of Contents2

7-1) Gaussian Pyramids

7-2) Laplacian Pyramids

7-3) Applications of Laplacian pyramids

7-5) Optical flow using Pyramids

7-1-1) Reduce 7-1-2) Expand

7-3-1) Image compression 7-3-2) Image composting

7-4) Interpolation

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

3

Original ImgHighest Resolution

Lowest Resolution

.

.

.

7-1)Gaussian Pyramids

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

7-1-1) Reduce4

Level l

Level l-1

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

7-1-1) Reduce- Convolution

5

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

7-1-1) Reduce (1D):Example

6

i = 2

Convolution Mask:[w(-2), w(-1), w(0), w(1), w(2)]

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

7-1-1) Reduce (1D)

7

gl = REDUCE (gl-

1)

Convolution Mask:[w(-2), w(-1), w(0), w(1), w(2)][ c , b , a , b , c ]

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

7-1-2) Expand8

n=1n=2

Notice:

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

7-1-2) Expand(1D)9

i = 4

[w(-2), w(-1), w(0), w(1), w(2)][ c , b , a , b , c ]

Involved weights [c , a , c]

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

7-1-2) Expand(1D)10

i = 3

[w(-2), w(-1), w(0), w(1), w(2)][ c , b , a , b , c ]

Involved weights [b , b]

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Expand11

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Convolution Mask12

•Separable

•Symmetric

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Convolution Mask13

• The sum of mask should be 1.

•All nodes at a given level must contribute the same total weight to the nodes at the next higher level.

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Convolution Mask14

a + 2b + 2c = 1

a + 2c = 2b

b= ¼

cacb b

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Convolution Mask 15

a= 0.5 TRINGULAR

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

a= 0.4 GAUSSIAN

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Gaussian Mask16

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Gaussian Pyramid17

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Gaussian Pyramid18

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramids20

• Similar to edge detected images.• Most pixels are zero.• Can be used for image compression.

L1 = g1 – EXPAND[g2]L2 = g2 – EXPAND[g3]L3 = g3 – EXPAND[g4]L4 = g4

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramids21

Lower in size and resolution

Gaussian Pyramid

Laplacian Pyramid

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

L1 = g1 – EXPAND[g2]L2 = g2 – EXPAND[g3]L3 = g3 – EXPAND[g4]L4 = g4

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Applications of Laplacian pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Image compression23

•Compute Gaussian pyramid

•Compute Laplacian pyramid

•Code Laplacian pyramid

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

24

• Decode Laplacian pyramid.• Compute Gaussian pyramid from Laplacian pyramid.

• g1 is reconstructed image.

Image compression

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Image Compression (Entropy)

25

7.6

4.4

5.0

5.6

6.2

0.77

1.9

3.3

4.2

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Image Compression26

1.58

0.73

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Combining Apple & Orange27

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Algorithm28

• Generate Laplacian pyramid Lo of orange image.

• Generate Laplacian pyramid La of appleimage.

• Generate Laplacian pyramid Lc by– copying left half of nodes at each level from apple

and– right half of nodes from orange pyramids.• Reconstruct combined image from Lc.

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Interpolation

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Interpolation30

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

1‐D Interpolation31

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

1=< x =<2

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

2‐D Interpolation32

Bilinear

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Bi‐linear Interpolation33

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Optical flow using Laplacian Pyramid

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Why Lucas Kanade with Pyramids?35

• Horn-Schunck and Lucas-Kanade optical method works only for small motion.

• If object moves faster, the brightness changes rapidly, 2x2 or 3x3 masks fail to estimate spatiotemporal derivatives.

• Pyramids can be used to compute large optical flow vectors.

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Lucas Kanade with Pyramids

36

LK for highest level of Laplacian pyramid ui , viDo the interpolation u*i-1 , v*i-1Multiply by 2 u*i-1 , v*i-1Calculate ft according to displacement of u*i-1 , v*i-1 LK for level l-1 of Laplacian pyramid u’i-1 , v’i-1Accurate value of Optical flow is ui-1 = u*i-1 + u’i-1

vi-1 = v*i-1 + v’i-1

Lucas Kanade

Lucas Kanade

Interpolation

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramid37

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramid38

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramid39

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramid40

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids

R.

C.

Gon

zale

z, a

nd R

. E

. W

oods

, D

igita

l Im

age

Pro

cess

ing,

New

Jer

sey:

Pre

ntic

e H

all,

3rd e

ditio

n, 2

008.

Laplacian Pyramid41

7-1) Gaussian Pyramids 7-1-1) Reduce 7-1-2) Expand 7-1-3) Convolution Mask

7-2) Laplacian Pyramids 7-3) Applications of Laplacian pyramids 7-3-1) Image compression 7-3-2) Image composting

7-4) interpolation

7-5) Optical flow using Pyramids