Byung Cheol Song Shin- Cheol Jeong Yanglim Choi

1

Byung Cheol SongShin-Cheol JeongYangl im Choi

Video Super-Resolution Algorithm Using Bi-directional

Overlapped Block Motion Compensation

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 3, MARCH 2011

2

IntroductionMotion-compensated SR

Basic concept Hierarchical motion estimation Bi-directional OBMC Result

Learning-based SRHybrid super-resolutionExperimental Results

OUTLINE

3

For the last few decades, many image interpolation algorithms have been developed to display high quality scaled images on cutting-edge digital consumer application such as HDTV,DSC.

Traditional interpolation methods usually suffer from several types of visual degradation.

To overcome the problem, many algorithms are proposed. But there are many problem.(Ex. structurally weak against textures, computational complexity)

INTRODUCTION

4

Brandi et al. presented an interesting SR approach. They defined the so-called key frames (KF) that sparsely exist in a video

sequence and have HR resolution. The remaining frames in the video sequence, i.e., non-key frames(NKF) had LR resolution

Brandi et al. took advantage of the fact that few KF (encoded at HR) may provide enough HF information to the up-scale NKF (encoded at LR)

However, it rarely found true motion information because it made use of the conventional full-search motion estimation.

This paper presents a hybrid SR algorithm where each LR patch is adaptively selected between a temporally super-resolved patch and a spatially super-resolved patch using adjacent HR KFs. Temporally super-resolved patch MSR Spatially super-resolved patch LSR

INTRODUCTION

5

A target frame in LR video sequence is interpolated by using forward and backward HR key-frame(KF).

MOTION-COMPENSATED SR Basic Concept

• The KF interval N can be constant or variable.• These paper employ cubic convolution for Up-Scaling.

UnHR U

nHRUHR0U

NHR

6

First, the MV for overlapping M×M matching block is searched by using a rate-constrained ME.

The conventional rate-constrained ME find the best v for each matching block by minimizing the rate as well as distortion

In order to maximize the ME performance and to concurrently reduce the computational burden, we adopt a rate-constrained fast full search algorithm presented in [15].

MOTION-COMPENSATED SR Hierarchica l Motion Estimation

λ : Lagrange multipier

7

Assume that MVs between up-scaled LR frames are statistically very similar to those between corresponding HR frame.

Replace the unknown MVs for HR frames with the MVs obtains from the up-scaled LR frames.

Employ BOBMC based on bi-directional MVs .the BOBMC is performed on a 4×4 block basis.

Each MV may be a forward or backward MV. The direction is determined according to SAD

MOTION-COMPENSATED SR B i -d irectional OBMC

8

MOTION-COMPENSATED SR Result

original MSRFan’s [12] Brandi’s [14]

9

It cannot often provide acceptable visual quality due to non-translational motion, occlusion, inaccurate motion estimation and limited motion search range.

When such temporal motion compensation does not work well, we employ a learning-based SR in order to avoid the degradation of visual quality.

MOTION-COMPENSATED SR Drawback

10

On-the-fly learning stage Construct a trained dictionary using adjacent LR KFs and HR KFs.

Inference stage The best-matched patch for the is exhaustively searched from the

dictionary.(the patch having minimum SAD is chosen as best-match) Produce a spatially super-resolved HR patch from

LEARNING-BASED SR

Produce Laplacian patch

K-means clustering

Seek optimal

LMS algorithm

Optimal dictionary

11

COMPARE MSR & LSR

MSR LSR

Background is well- motion-compensated

Seldom be well- motion-compensated

Better Quality

12

HYBRID SUPER-RESOLUTION

• The boundary between the temporally super-resolved patches and spatially super-resolved patches, we can observe blocking artifact because they are derived from different frames.

• We apply a simple smoothing filter to the boundary pixels

13

PSNR performance of the MSR according to N

EXPERIMENTAL RESULTS

Without/with little global motion N seldom affects

With large global motion

14

EXPERIMEN

TAL R

ESULTS

N = 30M =16L = 4Search range =±64

MSR Hybrid SR

[1] X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Trans. Image Process., vol. 10, no. 10, pp. 1521–1527, Oct. 2001.[12] W. Fan and D. Y. Yeung, “Image hallucination using neighbor embedding over visual primitive manifolds,” in Proc. CVPR, 2007, pp. 1–7.[14] F. Brandi, R. Queiroz, and D. Mukherjee, “Super-resolution of video using key frames and motion estimation,” in Proc. IEEE ICIP, Oct.2008, pp. 321–324.

Fan’s[12]Barsiu’s

Brandi’s[14]

NEDI[1]

BLI Bi-cubic

15

EXPERIMEN

TAL R

ESULTS

N = 30M =16L = 4Search range =±64

16

EXPERIMEN

TAL R

ESULTS

N = 30M =16L = 4Search range =±64