TEC H N IO N IsraelInstitute ofTechnology D epartm entofElectricalEngineering T he Vision Research and Image Science L aboratory By Elad A haroni G uy A bram ovich Supervised by D r. M ichaelElad Sum m er1998 -W inter1999 Supported by the O llendorffR esearch C enter Fund
37
Embed
4/99Super-Resolution2 Super-resolution 4/99Super-Resolution3 Out Line.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
TECHNIONIsrael Institute of Technology
Department of Electrical EngineeringThe Vision Research and Image Science Laboratory
By
Elad Aharoni Guy Abramovich
Supervised by
Dr. Michael Elad
Summer 1998 - Winter 1999
Supported by the Ollendorff Research Center Fund
4/99 Super-Resolution 2
Super-resolution
Given a collection of measured images of thesame scene, it is possible to fuse them to a onehigher resolution image, called the "SuperResolution Image"
Definition
Project Goal
1. To reconstruct a super-resolution image, whilepreserving its edges
2. Develop fast algorithm for the pure translationmotion case
4/99 Super-Resolution 3
Out Line
Part One
1. Simulate the creation of measured images
2. The super-resolution as a quadratic problem
3. The basic Steepest-Descent algorithm
4. Improvements of the SD algorithm
5. The Normalized-Steepest-Descent algorithm
6. The Conjugate-Gradient algorithm
7. Regularization
8. Simulations and Results
Part Two
1. Motion estimation algorithm
2. Getting input images
3. Estimating the super-resolution image
4/99 Super-Resolution 4
Simulate Creation Process of
Target Images
Warp1F
1
Blur1
1H
Decimate1D
1
noise 1
1 measuredlow resolutionimage
st
1Y
Warp NF
N
Blur N
NH
Decimate ND
N
noise N
N measuredlow resolutionimage
th
NY
sourcehigh - resolutionimage X
Creation of Yk images
Images Yk1k
Nmr supposed to be taken by CCD
camera
We assume Yk kNmr1are different representations of a
single high resolution image X of size LL
4/99 Super-Resolution 5
Simulate Creation Process of Target Images (Cont.)
M o d e l i n g o f c r e a t i o n p r o c e s s : Yk
Dk
Hk
Fk
X Ek
Fk
L L 2 2 g e o m e t r i c w a r p i n g
Hk
L L 2 2 l i n e a r - s p a c e i n v a r i a n t b l u r
Dk
M L 2 2 d e c i m a t i o n o p e r a t o r
Ek
M 2 1 a d d i t i v e z e r o m e a n G a u s s i a n n o i s e
Yk
M 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e m e a s u r e d
i m a g e s
X L 2 1 v e c t o r , C S r e p r e s e n t a t i o n o f t h e
s u p e r - r e s o l u t i o n i m a g e
All matrices assumed to be known in advance
4/99 Super-Resolution 6
Simulate Creation Process of Target Images (Cont.)
Assumptions: Hk = H
Dk = D
Fk – Pure displacement.
G r o u p i n g N e q u a t i o n s i n t o o n eY
YN
D H F
DN
HN
FN
X
E
E N
C
CN
X E
1 1 1 1 1 1
L
N
MMMMM
O
Q
PPPPP
L
N
MMMMM
O
Q
PPPPP
L
N
MMMMM
O
Q
PPPPP
L
N
MMMMM
O
Q
PPPPP
w h e r e C D H Fk k k k
Final result YCXE
4/99 Super-Resolution 7
Creation of Yk Images from Source
Image
Source Image Yk Images
4/99 Super-Resolution 8
The Super-resolution Problem as Quadratic Problem
YCXE Classic model of restoration problem
Given a solution X the Error function:
E Y C Xrr
2 2
D if f e r e n t i a t i n g w i th r e s p e c t t o X a n d e q u a t i n g t o z e r o :
2 0 C Y C XT a fC Y C C XT T c h
t h e n :
X C C C YT T c h1
P C YT - ו R C CT
R X P
Well known classic pseudoinverse result XRP1
Practically impossible to invert
Requires alternative solutions – iterative algorithms
R - Size of L L2 2 2 2400 400 c h
4/99 Super-Resolution 9
Steepest-Descent Algorithm
Described by the following equations:
X X R X Pj j 1
R F H D D H F
P F H D Y
kT
k
NT T
k
kT T
k
NT
k
1
1
F1 H D DT HT FT1
XK
H D DT HTFN FNT
Y1
XK1
XK
YN
Xk
Xk
FjT
j
NHT DT D H F
jX
kY
j IKFH1
1
The basic Steepest-Descent algorithm
Error function (quality factor):
2 2
1
Y DHF Xk k
k
N
Relatively slow (H implemented as convolution)
constant chosen manually: =0.1 – slowconvergence
=0.5 – fast convergence
4/99 Super-Resolution 10
Restoration Results ofSteepest-Descent Algorithm
4/99 Super-Resolution 11
Enhanced Steepest-Descent Algorithm
Basic algorithm is very slow
Blurring operator is equal for all images
F i r s t e n h a n c e m e n t : F k a n d H a r e c o m m u t a t i v e
T h e n :
2 2
1 1
1
LNM OQP LNM OQP
Y D H F X R H F D D F H
P H F D Y
k kk
NT
kT T
kk
N
Tk
T
k
NT
k
A n d t h e n e w e q u a t i o n :
X X H F D D F H X Yk kT
kT T
j
N
j k j
11
a f
4/99 Super-Resolution 12
Enhanced Steepest-Descent Algorithm (Cont.)
Blurring operator performed twice instead of 2N in
every iteration
Save of 30% in run-time while retaining result
F1 D DT
HT
FT1
XK
D DTFN FNT
Y1
XK1
XK
YN
H
Commutative warping and blurring operators
4/99 Super-Resolution 13
Enhanced Steepest-Descent Algorithm (Cont.)
S e c o n d e n h a n c e m e n t :
R H F D D F H H R HTk
T Tk
k
NT LNM OQP
10
R F D D FkT T
kk
N
01
P H F D Y H PTk
T Tk
k
NT LNM OQP
10
P F D YkT T
kk
N
01
T h e n e w e q u a t i o n : X X H R H X Pj jT
j 1 0 0
Matrices P0 and R0 calculated once only for the
whole algorithm
If warping is based on Nearest-Neighbor then:
R0 is diagonal
R0 implemented as mask
4/99 Super-Resolution 14
Enhanced Steepest-Descent Algorithm (Cont.)
XK H R0
P0
HT XK1
The enhanced Steepest-Descent algorithm
Save of 80% in run-time while retaining result,compared to the original SD algorithm
4/99 Super-Resolution 15
The Normalized SD algorithm
SD algorithm - constant chosen by user
Normalized - SD: calculates optimal for every
iteration
X X Ej j j j 1
E R X Pj j
and jjT
j
jT
j
E E
E R E
1
Calculation of optimal adds 20% to run-time
4/99 Super-Resolution 16
Restoration Results of NSD Algorithm
4/99 Super-Resolution 17
Conjugate Gradient Algorithm
Complicated calculations
Comparison to NSD algorithm:
On synthetic square problem
On our current simulation
T h e C G a l g o r i t h m : G i v e n R , P , X k a n d V k - 1
1a f E R X Pk k
2 11
1 1
a f kkT
k
kT
k
E R VV R V
3 1 1a f V E Vk k k k
4a f kkT
k
kT
k
E VV R V
T h e f i n a l e q u a t i o n : X X Vk k k k 1
4/99 Super-Resolution 18
Conjugate Gradient Algorithm (Cont.)
E KK
K 1 V K
1
MUL
2 3
4
K KV XK1
XK
P
R
-
+
VK 1
The Conjugate-Gradient algorithm
CG iteration run-time is 50% longer than in the NSD
4/99 Super-Resolution 19
Comparison of CG vs. NSD
Comparison parameters: Error function 2 xafNorma X X true 2
Both algorithms converge to same result
The norma factor getting worse because of ringing effect
Xtrue calculation is explained later
4/99 Super-Resolution 20
Comparison of CG vs. NSD (Cont.)
Concentrating on the first few iterations
CG vs NSD in the first few interations
4/99 Super-Resolution 21
Comparison of CG vs. NSD (Cont.)
CG and NSD after 7 iterations
CG converges faster - damaged by stronger ringing effect
4/99 Super-Resolution 22
Regularization
Ringing effect damages images as we converge
Possible reasons: True R matrix is not inversible
Error function doesn't reflect realminimization
We're interested in - min X Xrestore true 2
Regularization component in the error function
2 2
1
2
Y D H F X D Xk k
k
N ~
Differentiating with respect to X and equating to zeroyields:
R F H D D H F D D
P F H D Y
kT T T
kk
NT
kT T
kT
k
N
1
1
~ ~
~~DDT- Laplasian operator
- determines the regularization scale. Constant chosen
manually
4/99 Super-Resolution 23
Regular Regularization
XK H R0
P0
HT XK1
~ ~D DXT
K
Regular regularization implementation in NSD
What is the optimal ?
Run for various values of :
Convergence factor for various values of
Optimal result for =0.6
4/99 Super-Resolution 24
Regularization Results for Various Values of
4/99 Super-Resolution 25
Comparison Between CG and NSD Including Regularization
Conclusion: CG and NSD converge to same resultCG converges faster in the first fewiterations
4/99 Super-Resolution 26
Adaptive Regularization
Is it possible to implement regularization while
preserving edges?
W e ig h te d re g u la riza tio n c o m p o n e n ts in th e e rro rfu n c tio n :
2 2
1
2
Y C X D Xk k
k
N
W
~
D iffe re n tia tin g w ith re sp ec t to X an d e q u a tin g to z e roy ie ld s :
R F H D D H F D W D
P F H D Y
kT T T
kk
NT
kT T
kT
k
N
1
1
~ ~
Implementation of adaptive regularization in NSDalgorithm:
XK H R0
P0
HT XK1
~ ~D WDXT
K
4/99 Super-Resolution 27
W Matrix
Prevents edges from being regularized
Detects edges by calculating derivatives
Built according to derivatives values
Implemented as mask
a
W
b ערךרת גז הנ
1
a
W
ערךרת גז הנ
1
a
W
b ערךרת גז הנ
1
Different threshold functions
Image of W mask
4/99 Super-Resolution 28
W Matrix (Cont.)
Low threshold High threshold
Adaptive regularization Regular
4/99 Super-Resolution 29
Motion Estimation
Relative translation is not known
M o tio nE stim atio n ן בי ת סי ח י זה ו תז
ת ונו מ ת שתי ה
I (x, y)1
I (x, y)2
(dx, dy)
Algorithm based on Taylor series
Accurate for small warping (0-2 pixels)
We deal with large warping (0-10 pixels)
Implement iterative algorithm
4/99 Super-Resolution 30
Motion Estimation Implementation as Iterative Algorithm
M o ti o nE stim ati o n
I (x, y)1 I (x, y)2
I nv erseW arp ing
M o ti o nE stim ati o n
I nv erseW arp ing
M o ti o nE stim ati o n
Tota lW arp ing(dx , dy)
(dx1 , dy1)
(dx2 , dy2)
(dxn , dyn)
4/99 Super-Resolution 31
Input Images
Take 60 images from video sequence
Create difference images
Choose best 15 with minimal rotation and scaling
Difference images including rotation and scaling
Difference images including minimal rotation and scaling
4/99 Super-Resolution 32
Final Results
Implemented in NSD algorithm
Specific blurring kernel of CCD camera -
Regularization parameter -
Optimal result for =1.8
4/99 Super-Resolution 33
Final Results (Cont.)
Optimal Result for =1.8 and =2
4/99 Super-Resolution 34
Final Results (Cont.)
Zoom on the final results
Good obtained results
Insufficient due to technical problems:
Strong anti-aliasing effect
Rotation and scaling
4/99 Super-Resolution 35
Aliasing Effect
The Original Image Sampled Image The Restored Image
The Original Image Sampled Image The Restored Image
The original images decimated in order to "add"aliasing effect
Very strong anti-aliasing effect caused by the camera
4/99 Super-Resolution 36
Aliasing Effect (Cont.)
No resolution improvement in the result image
compared to the original
Quality improvement in the result image compared to
the original
The Original Image Sampled Image The Restored Image
4/99 Super-Resolution 37
Summing
We developed a fast version algorithm forSuper-Resolution Reconstruction, assuming puredisplacement between the images
Simulations on synthetic data verified that the takenshortcuts are valid and produce 80% reduction nrun-time, while preserving output quality
Results on true video images show slightimprovement, due to too strong anti-aliasing effect