ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.

ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH

ACCURACY SUPER-RESOLUTION

Michalis Vrigkas, Christophoros Nikou, Lisimachos P. KondiUniversity of Ioannina

Department of Computer ScienceIoannina, Greece

• Objective– Reconstruct a high-resolution image from a

sequence of low-resolution images.• Improve spatial resolution.

• Constraints on low-resolution images– Motion– Rotation– Blurring– Subsampling– Additive noise

MOTIVATION

• MAP scheme for image super-resolution.

• Registration in two parts– At first, the LR images are registered by

establishing correspondences between robust SIFT (Scale-Invariant Feature Transform) features.

– In the second step, the estimation of the registration parameters is fine tuned along with the estimation of the HR image.• Mutual Information Criterion: maximize the mutual

information between HR image and each of the upscaled LR images.

APPROACH

• Let the high-resolution image

where

• The set of LR images is described as

We consider p LR images each of size

FORMULATION MODEL

1 2 [ , , , ]TNz z zz 1 1 2 2N L N L N

1 2[ , , , ]T T T Tpy y yy

1 2M N N

• Observation model:– All images are ordered lexicographically

– represents zero-mean additive Gaussian noise,

– is the degradation matrix, performing the operations of:• motion• blur• down-sampling

FORMULATION MODEL (cont.)

y = Wz +n

nW

( )k k kW = DB M s

• The Gaussian prior for the HR image is:

– is the Laplacian of the image z– controls the precision and the shape of the

distribution

• The likelihood of the LR images is Gaussian:

MAP ESTIMATOR

/2

/2

( | |) 1( ) exp ( ) ( )

(2 ) 2

T NT

Np

Q Qz Qz Qz

Qz

22

1 ( ) ( )( | ) exp

2(2 )

T

pMpM

p

y Wz y Wzy z

• MAP approach– Maximize – Which leads to a MAP functional to be minimized with

respect to HR image z and the transformation parameters s:

• Use gradient descent method– The update equation is given by:

where εn is the step size at the n-th iteration.

MAP ESTIMATOR (cont.)

( | ) ( | ) ( )p p pz y y z z

2 2 2

1

( , ) | | ( ) || || where =||p

k k kk

L

z s y W s z Qz

1( , ) | n

n nn L

z z z

z z z s

• Objective: independently detect corresponding keypoints in scaled versions of the same image.

• Idea: Given a keypoint in two images, determine if the surrounding neighborhoods contain the same structure up to scale.

• SIFT features are invariant to:– Image scale and rotation– Affine transformations– Changes in illumination and noise

[D. G. Lowe. "Distinctive image features from scale invariant

keypoints.”International Journal of Computer Vision 60 (2), pp. 91-110, 2004.]

SCALE INVARIANT FEATUTE TRANSFORM - SIFT

• Basics: the mutual information is maximized when the two images are correctly registered.

• The mutual information between two images A and B is:

– H(A) and H(B) are the marginal entropies of the random variables A and B.

– H(A,B) is the joint entropy.

MUTUAL INFORMATION CRITERION

( , ) ( ) ( ) ( , )

( , ) ( , ) log

( )· ( )AB

ABa b A B

I A B H A H B H A B

p a bp a b

p a p b

• Normalized Mutual Information:– Robust measure in order to provide invariance to the

overlapping areas between the two images.

• Problem: – If mutual information is not initialized close to the

optimal solution it is trapped by local maxima.• Good initialization is important.

• Solution:– Initialization using SIFT descriptors.

MUTUAL INFORMATION CRITERION (cont.)

( ) ( )( , )

( , )

H A H BNMI A B

H A B

• Estimation of registration parameters in two steps.– First step, LR images are registered by

employing SIFT features.• Minimization of mean square error between the

locations of features between the reference image and the LR images.

• Provides good initialization.

IMAGE REGISTRATION

– Second step, the estimation of the registration parameters is fine-tuned along with the estimation of the HR image, by maximization of mutual information criterion.• Iterative scheme.

• Contribution:– The registration accuracy is improved at each

iteration step.– Refinement of the mutual information

registration.

IMAGE REGISTRATION (cont.)

• Synthetic data sets.

• LR images were created by rotating, translating, blurring, down-sampling and degrading by noise.– Translation: uniformly selected in [-3, 3] (in pixels)– Rotation: uniformly selected in [-5, 5] (in degrees)– Down-sampling factor: 2 (4 pixels to 1)– Blurring: 5x5 Gaussian kernel, standard deviation

of 1– Additive noise: AWGN to obtain SNR of 30 dB and

20 dB

EXPERIMENTAL PARAMETERS

• First estimate of the HR image– Bicubic interpolation

• Total number of realizations for each case: 10

• Convergence: or 70 iterations reached.

• Quantitative evaluation: peak signal to noise ratio

EXPERIMENTAL PARAMETERS (cont.)

15/ 10

n n n z z z‖ ‖ ‖ ‖

2

10 2

255PSNR 10log

ˆ|| ||z z

COMPARE METHODS

• Books (PSNR = 26.06 dB)

• 4 LR images used

EXPERIMETAL RESULTS

LR image

Reconstructed HR image

• Front page (PSNR = 26.14 dB)

• 6 LR images used

EXPERIMETAL RESULTS (cont.)

LR image

Reconstructed HR image

• Car (PSNR = 28.13 dB)

• 5 LR images used

EXPERIMETAL RESULTS (cont.)

LR image

Reconstructed HR image

• Eye chart (PSNR = 27.33 dB)

• 4 LR images used

EXPERIMETAL RESULTS (cont.)

LR image

Reconstructed HR image

EXPERIMETAL RESULTS (cont.)

• Statistics for the compared SR methods

+1.5 dB on average better results than SIFT.

• Hybrid registration approach– SIFT-based image registration combined with

the maximization of mutual information.– High precision registration

• High accuracy super-resolved image.– Improvement is 1.5 dB on average higher for

both 30 dB and 20 dB.

• Proposed algorithm converges faster than the standard solution.

CONCLUTIONS

QUESTIONS?

?

THANK YOU ALL FOR YOUR PARTICIPATION AND

PATIENCE!