4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line.

TECHNIONIsrael Institute of Technology

Department of Electrical EngineeringThe Vision Research and Image Science Laboratory

By

Elad Aharoni Guy Abramovich

Supervised by

Dr. Michael Elad

Summer 1998 - Winter 1999

Supported by the Ollendorff Research Center Fund

4/99 Super-Resolution 2

Super-resolution

Given a collection of measured images of thesame scene, it is possible to fuse them to a onehigher resolution image, called the "SuperResolution Image"

Definition

Project Goal

1. To reconstruct a super-resolution image, whilepreserving its edges

2. Develop fast algorithm for the pure translationmotion case

4/99 Super-Resolution 3

Out Line

Part One

1. Simulate the creation of measured images

2. The super-resolution as a quadratic problem

3. The basic Steepest-Descent algorithm

4. Improvements of the SD algorithm

5. The Normalized-Steepest-Descent algorithm

6. The Conjugate-Gradient algorithm

7. Regularization

8. Simulations and Results

Part Two

1. Motion estimation algorithm

2. Getting input images

3. Estimating the super-resolution image

4/99 Super-Resolution 4

Simulate Creation Process of

Target Images

Warp1F

1

Blur1

1H

Decimate1D

1

noise 1

1 measuredlow resolutionimage

st

1Y

Warp NF

N

Blur N

NH

Decimate ND

N

noise N

N measuredlow resolutionimage

th

NY

sourcehigh - resolutionimage X

Creation of Yk images

Images Yk1k

Nmr supposed to be taken by CCD

camera

We assume Yk kNmr1are different representations of a

single high resolution image X of size LL

4/99 Super-Resolution 5

Simulate Creation Process of Target Images (Cont.)

M o d e l i n g o f c r e a t i o n p r o c e s s : Yk

Dk

Hk

Fk

X Ek

Fk

L L 2 2 g e o m e t r i c w a r p i n g

Hk

L L 2 2 l i n e a r - s p a c e i n v a r i a n t b l u r

Dk

M L 2 2 d e c i m a t i o n o p e r a t o r

Ek

M 2 1 a d d i t i v e z e r o m e a n G a u s s i a n n o i s e

Yk

M 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e m e a s u r e d

i m a g e s

X L 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e

s u p e r - r e s o l u t i o n i m a g e

All matrices assumed to be known in advance

4/99 Super-Resolution 6

Simulate Creation Process of Target Images (Cont.)

Assumptions: Hk = H

Dk = D

Fk – Pure displacement.

G r o u p i n g N e q u a t i o n s i n t o o n eY

YN

D H F

DN

HN

FN

X

E

E N

C

CN

X E

1 1 1 1 1 1

L

N

MMMMM

O

Q

PPPPP

L

N

MMMMM

O

Q

PPPPP

L

N

MMMMM

O

Q

PPPPP

L

N

MMMMM

O

Q

PPPPP

w h e r e C D H Fk k k k

Final result YCXE

4/99 Super-Resolution 7

Creation of Yk Images from Source

Image

Source Image Yk Images

4/99 Super-Resolution 8

The Super-resolution Problem as Quadratic Problem

YCXE Classic model of restoration problem

Given a solution X the Error function:

E Y C Xrr

2 2

D if f e r e n t i a t i n g w i th r e s p e c t t o X a n d e q u a t i n g t o z e r o :

2 0 C Y C XT a fC Y C C XT T c h

t h e n :

X C C C YT T c h1

P C YT - ו R C CT

R X P

Well known classic pseudoinverse result XRP1

Practically impossible to invert

Requires alternative solutions – iterative algorithms

R - Size of L L2 2 2 2400 400 c h

4/99 Super-Resolution 9

Steepest-Descent Algorithm

Described by the following equations:

X X R X Pj j 1

R F H D D H F

P F H D Y

kT

k

NT T

k

kT T

k

NT

k

1

1

F1 H D DT HT FT1

XK

H D DT HTFN FNT

Y1

XK1

XK

YN

Xk

Xk

FjT

j

NHT DT D H F

jX

kY

j IKFH1

1

The basic Steepest-Descent algorithm

Error function (quality factor):

2 2

1

Y DHF Xk k

k

N

Relatively slow (H implemented as convolution)

constant chosen manually: =0.1 – slowconvergence

=0.5 – fast convergence

4/99 Super-Resolution 10

Restoration Results ofSteepest-Descent Algorithm

4/99 Super-Resolution 11

Enhanced Steepest-Descent Algorithm

Basic algorithm is very slow

Blurring operator is equal for all images

F i r s t e n h a n c e m e n t : F k a n d H a r e c o m m u t a t i v e

T h e n :

2 2

1 1

1

LNM OQP LNM OQP

Y D H F X R H F D D F H

P H F D Y

k kk

NT

kT T

kk

N

Tk

T

k

NT

k

A n d t h e n e w e q u a t i o n :

X X H F D D F H X Yk kT

kT T

j

N

j k j

11

a f

4/99 Super-Resolution 12

Enhanced Steepest-Descent Algorithm (Cont.)

Blurring operator performed twice instead of 2N in

every iteration

Save of 30% in run-time while retaining result

F1 D DT

HT

FT1

XK

D DTFN FNT

Y1

XK1

XK

YN

H

Commutative warping and blurring operators

4/99 Super-Resolution 13

Enhanced Steepest-Descent Algorithm (Cont.)

S e c o n d e n h a n c e m e n t :

R H F D D F H H R HTk

T Tk

k

NT LNM OQP

10

R F D D FkT T

kk

N

01

P H F D Y H PTk

T Tk

k

NT LNM OQP

10

P F D YkT T

kk

N

01

T h e n e w e q u a t i o n : X X H R H X Pj jT

j 1 0 0

Matrices P0 and R0 calculated once only for the

whole algorithm

If warping is based on Nearest-Neighbor then:

R0 is diagonal

R0 implemented as mask

4/99 Super-Resolution 14

Enhanced Steepest-Descent Algorithm (Cont.)

XK H R0

P0

HT XK1

The enhanced Steepest-Descent algorithm

Save of 80% in run-time while retaining result,compared to the original SD algorithm

4/99 Super-Resolution 15

The Normalized SD algorithm

SD algorithm - constant chosen by user

Normalized - SD: calculates optimal for every

iteration

X X Ej j j j 1

E R X Pj j

and jjT

j

jT

j

E E

E R E

1

Calculation of optimal adds 20% to run-time

4/99 Super-Resolution 16

Restoration Results of NSD Algorithm

4/99 Super-Resolution 17

Conjugate Gradient Algorithm

Complicated calculations

Comparison to NSD algorithm:

On synthetic square problem

On our current simulation

T h e C G a l g o r i t h m : G i v e n R , P , X k a n d V k - 1

1a f E R X Pk k

2 11

1 1

a f kkT

k

kT

k

E R VV R V

3 1 1a f V E Vk k k k

4a f kkT

k

kT

k

E VV R V

T h e f i n a l e q u a t i o n : X X Vk k k k 1

4/99 Super-Resolution 18

Conjugate Gradient Algorithm (Cont.)

E KK

K 1 V K

1

MUL

2 3

4

K KV XK1

XK

P

R

-

+

VK 1

The Conjugate-Gradient algorithm

CG iteration run-time is 50% longer than in the NSD

4/99 Super-Resolution 19

Comparison of CG vs. NSD

Comparison parameters: Error function 2 xafNorma X X true 2

Both algorithms converge to same result

The norma factor getting worse because of ringing effect

Xtrue calculation is explained later

4/99 Super-Resolution 20

Comparison of CG vs. NSD (Cont.)

Concentrating on the first few iterations

CG vs NSD in the first few interations

4/99 Super-Resolution 21

Comparison of CG vs. NSD (Cont.)

CG and NSD after 7 iterations

CG converges faster - damaged by stronger ringing effect

4/99 Super-Resolution 22

Regularization

Ringing effect damages images as we converge

Possible reasons: True R matrix is not inversible

Error function doesn't reflect realminimization

We're interested in - min X Xrestore true 2

Regularization component in the error function

2 2

1

2

Y D H F X D Xk k

k

N ~

Differentiating with respect to X and equating to zeroyields:

R F H D D H F D D

P F H D Y

kT T T

kk

NT

kT T

kT

k

N

1

1

~ ~

~~DDT- Laplasian operator

- determines the regularization scale. Constant chosen

manually

4/99 Super-Resolution 23

Regular Regularization

XK H R0

P0

HT XK1

~ ~D DXT

K

Regular regularization implementation in NSD

What is the optimal ?

Run for various values of :

Convergence factor for various values of

Optimal result for =0.6

4/99 Super-Resolution 24

Regularization Results for Various Values of

4/99 Super-Resolution 25

Comparison Between CG and NSD Including Regularization

Conclusion: CG and NSD converge to same resultCG converges faster in the first fewiterations

4/99 Super-Resolution 26

Adaptive Regularization

Is it possible to implement regularization while

preserving edges?

W e ig h te d re g u la riza tio n c o m p o n e n ts in th e e rro rfu n c tio n :

2 2

1

2

Y C X D Xk k

k

N

W

~

D iffe re n tia tin g w ith re sp ec t to X an d e q u a tin g to z e roy ie ld s :

R F H D D H F D W D

P F H D Y

kT T T

kk

NT

kT T

kT

k

N

1

1

~ ~

Implementation of adaptive regularization in NSDalgorithm:

XK H R0

P0

HT XK1

~ ~D WDXT

K

4/99 Super-Resolution 27

W Matrix

Prevents edges from being regularized

Detects edges by calculating derivatives

Built according to derivatives values

Implemented as mask

a

W

b ערךרת גז הנ

1

a

W

ערךרת גז הנ

1

a

W

b ערךרת גז הנ

1

Different threshold functions

Image of W mask

4/99 Super-Resolution 28

W Matrix (Cont.)

Low threshold High threshold

Adaptive regularization Regular

4/99 Super-Resolution 29

Motion Estimation

Relative translation is not known

M o tio nE stim atio n ן בי ת סי ח י זה ו תז

ת ונו מ ת שתי ה

I (x, y)1

I (x, y)2

(dx, dy)

Algorithm based on Taylor series

Accurate for small warping (0-2 pixels)

We deal with large warping (0-10 pixels)

Implement iterative algorithm

4/99 Super-Resolution 30

Motion Estimation Implementation as Iterative Algorithm

M o ti o nE stim ati o n

I (x, y)1 I (x, y)2

I nv erseW arp ing

M o ti o nE stim ati o n

I nv erseW arp ing

M o ti o nE stim ati o n

Tota lW arp ing(dx , dy)

(dx1 , dy1)

(dx2 , dy2)

(dxn , dyn)

4/99 Super-Resolution 31

Input Images

Take 60 images from video sequence

Create difference images

Choose best 15 with minimal rotation and scaling

Difference images including rotation and scaling

Difference images including minimal rotation and scaling

4/99 Super-Resolution 32

Final Results

Implemented in NSD algorithm

Specific blurring kernel of CCD camera -

Regularization parameter -

Optimal result for =1.8

4/99 Super-Resolution 33

Final Results (Cont.)

Optimal Result for =1.8 and =2

4/99 Super-Resolution 34

Final Results (Cont.)

Zoom on the final results

Good obtained results

Insufficient due to technical problems:

Strong anti-aliasing effect

Rotation and scaling

4/99 Super-Resolution 35

Aliasing Effect

The Original Image Sampled Image The Restored Image

The Original Image Sampled Image The Restored Image

The original images decimated in order to "add"aliasing effect

Very strong anti-aliasing effect caused by the camera

4/99 Super-Resolution 36

Aliasing Effect (Cont.)

No resolution improvement in the result image

compared to the original

Quality improvement in the result image compared to

the original

The Original Image Sampled Image The Restored Image

4/99 Super-Resolution 37

Summing

We developed a fast version algorithm forSuper-Resolution Reconstruction, assuming puredisplacement between the images

Simulations on synthetic data verified that the takenshortcuts are valid and produce 80% reduction nrun-time, while preserving output quality

Results on true video images show slightimprovement, due to too strong anti-aliasing effect