Interframe Coding

Interframe Coding

Heejune AHNEmbedded Communications Laboratory

Seoul National Univ. of TechnologyFall 2008

Last updated 2008. 10. 12

Heejune AHN: Image and Video Compression p. 2

Agenda

Interframe Coding Concept Block Matching Algorithm Fast Block Matching Algorithms Block Matching Algorithm Variations Enhanced Motion Models Implementation Cases


1. Interframe Coding

Motivation Video has High Temporal Correlation between frames. Var[ X(t+1) – X(t) ] << Var[ X(t+1) ]

Two successive video frames DFD(displaced Frame Difference)


Motion Estimation and compensation Motion estimation

• Find the best parameters of current frame from reference frames Motion compensation

• Subtracts (Add) the predicted values from current frame (to DFD frame)

Reference frames

Current frame

MC

ME

Recon.

Motion parameters

EncodeResidual

Texture Info MC

Reference frames

Recon.


Performance Criteria Coding performance

• Residual signal has low energy (variance measure) Complexity

• Computational and implementation complexity Storage and Delay

• Number of required frames Side Information

• Size and complexity of motion parameters Error resilience

• When data is partially lost. Some factors are trade off

Coding perf. against complexity, storage, side info, error resilience.


2D Motion

x

y

stationarybackground

movingobject

shiftedobject

time t

t

previous frame

current frame

d x

dy

Prediction for the luminance signal S(x,y,t) within the moving object:

ˆ S (x, y, t) S(x dx , y dy , t t)

„Displacement vector“


2. Block Matching Algorithm

BMA(Block matching algorithm) Segment frame into same rectangular Blocks 2-D linear motion (mvx, mvy) per each block

Real Motion

MV

X(t) X(t+1)


Difference Measure MSE

MAE and SAE

CCF (Cross Correlation Function)

21

0

1

02

)),( -),((1

yxcmvymvxrN

MSEN

yyx

N

x

|),( -),(|1 1

0

1

02

yxcmvymvxrN

MAEN

yyx

N

x

|),( -),(|1

0

1

0

yxcmvymvxrSAEN

yyx

N

x

)),()(m-),((1

r

1

0

1

02 c

N

yyx

N

x

myxcmvxmvxrN

CCF


Full Search Algorithm “Full Search” does Not means the whole frame, but whole

position in limited Search Window Method

• Raster order or Spiral order (Figure. 6.6)

x x+6x-6

y-6

y+6

y

x x+6x-6

y-6

y+6

y


Full Search Complexity (2w+1) x (2w+1) points (for search window [-w, w]) NxN size Block computation

int SAE(uchar *f, uchar *g, int mvx, int mvy){

for ( x=0; x< N; x++){

for ( y=0; y< N; y++){

sae += ABS(*(f + (y+mvy)*width +(x+mvx), *(g + y*width+x)); }

mvx_min = mvy_min = 0;min = SAE(f, g, 0, 0); for(mvy=-w, mvy<=w, mvy++) for(mvx=-w, mvx<=w, mvx++){

sae = SAE(pre, cur, mv, mv) if(min >sae) mvx_min = mvx, mvy_min = mvy, min = sae;

}


3. Fast BMAs

Complexity Reduction Approaches Reduce test points

• Monotonic variation assumption – The closer to the optimal point, the smaller difference

Change the test-point order (more like first)• Binary Search than Linear Search

• Benefit from Early Stop of block difference calculation Reduce the computation at one point

• Sub-sampled value Note

Trade-off!

0-w +w


TSS (3-Step Search) Step 0: Search center (0,0), n = w Step 1: n = floor[ n / 2 ] Step 2: Search 8 points and find the

min values Step 3: if n == 1 stop, o.w. Go to

Step 1 Properties

Logarithmic/Binary search (only 3 step when p = 8)

Search decreasing distance • w/2 => w/4 => w/8 . . . . until 1

Complexity : O(log2w)

x x+6x-6

y-6

y+6

y

1 1

11

1

2

2

3

2

1

1

11

2

2 2 2

2

3

3

3

3

3

3


2D Logarithmic Search Step 0: Search center (0,0) Step 1: Search 4 points with s step

size Step 2: find min, if center S = S/2,

ow. move center to the min locaiton Step 3: if S = 1, go to step 4, else

go to Step 1 Step 4: search the 8 neighbors, and

decide min. Properties

Similar to TSS, but more accurate Complexity ~ O(log2w) but not fixed

loop count

1

1 1

1

2

2

2

2

3

43

4555

55 5 5

5


Examples TSS (Tree Step Search) Logarithmic Search Cross Search One-at-a-time Search Nearest Neighbors Search

From Other Source. TSS (Three Step search) TDL (Two Dim. Logarithmic) CDS (Conjugate Direction Search) CSA (Cross Search Algorithm) OSA (Orthogonal Search Algorithm)


Fast BMA Performance

Complexity

Algorithm Maximum number of search points

w

4 8 16

FSM (2w + 1)2 81 289 1089

TDL 2 + 7 log2 w 16 23 30

TSS 1 + 8 log2 w 17 25 33

MMEA 1 + 6 log2 w 13 19 25

CDS 3 + 2w 11 19 35

OSA 1 + 4 log2 w 9 13 17

CSA 5 + 4 log2 w 13 17 21


Estimation Performance

Algorithm Split screen Trevor white

entropy (bits/pel)

standard deviation

entropy (bits/pel)

standard deviation

FSM 4.57 7.39 4.41 6.07

TDL 4.74 8.23 4.60 6.92

TSS 4.74 8.19 4.58 6.86

MMEA 4.81 8.56 4.69 7.46

CDS 4.84 8.86 4.74 7.54

OSA 4.85 8.81 4.72 7.51

CSA 4.82 8.65 4.68 7.42


Issues in Fast MC Algorithm

Local Minimum Error Fast MC calculates only few of positions Many cases are not “monotonic” curves, single hill. Possibly can conclude with local minimum. See Figure 6.15

1 1 1

22

3

3


Hierarchical MC Reduced image

• Sub-sampled, filtered

• N levels with half resolution Search top (N) level fully

• reduced search window range (w/2N-1) Search lower N-1 level

• only 9(8?) neighbor positions only


Benefits of hierarchical search Escape Local minimum Complexity Reduction

• e.g) Window = 16

full search (2 × 32 + 1)2 = 4225 operations

HBMA with N =4, (2 × 4 + 1)^2 + 3 × 9 = 108 operations

Sub-sampled signal

Original signal


4. Variations of BMA: Multi-frame MC

Multiple Frame MC “Forward pred” starts from H.261 “backward, bidirectional” starts from MPEG-1 “multiple reference (each MB takes its own ref picture) starts from

H.264

forward

forward

bidirectional: average

backward


4. Variations of BMA: Multi-frame MC

Multiple Frame distance Search Range = frame difference x window

• Since displacement = velocity x time eg) w = 8, 64 points (1 frame diff), 256 points (2 frame diff)

Practice• search only [-w, w] of (mvx1, mvy1) for (mvx2, mvy2)

-w

+2w+w

-2w

tt -1t -2

mvx1,mvy1

mvx2,mvy2


MV at Boundary

Restriction on MV range Should inside of reference pictures In H.261/MPEG-1, MPEG-2, MPEG-4

Unrestricted MV Extrapolates (extends with same boundary pixel value) In H263 Annex D,H.264

t

-w

+w

t -1

-w

+w

Extrapolated t -1


Sub-pixel Motion Estimation

Note Object cannot happens to move integer pixels We have only integer pixel samples

Sub-pixel estimation Get the fractional pel values in reference frame Normally using linear interpolation Half-pel/quarter-pel


5. Enhanced Motion Models

More Motion Estimation Model Rigid 2D Translation (BMA)

• + Transformation Global Motion

• + Illumination variation

• + zoom-in/out Object Model

• + overlapping of objects

• + 3D Rotation

• + Non rigid objects (deformation)

Some are from computer vision area But at present most tools are too complex for application to video

coding area Some are included in MPEG-4 Part 2’s Object Oriented Coding


Examples Region based motion

compensation • How to get/describe shape

and motion

Global motion (picture warping)

• Called Camera motion

Mesh-based Deformation


6. Implementation

Video Encoder and Decoder Complexity Profiling


SW Optimization

Algorithm level optimization : independent of CPU Data structure design (most modern CPU, RISC)

Memory Cache optimization Current blocks into cache Loop unrolling (See Fig. 6.21)

• Reduce the pointer operation and jump prediction (pipelining)

CPU-specifics Optimization SIMD (Single Instruction with Multiple Data)

• Packed Instruction (See Fig. 6.22)• TI DSP, Intel MMX etc

MIMD (MuParalell Processing Core)• VLIW (Very Long Instruction Word) of TI DSP

GPU DMA utilization Coprocessor Utilization

• DCT, ME, Post/Pre Processing


HW Optimization

Criteria Performance, cycle count, gate-count, data flow

Example #1: Full Search Parallelization

• M function block, then M Speed up

Search WindowMemory

(DRAM/SRAM)

Current MB(SRAM)

SAE SAE SAE SAE

Comparator


Example #2: Fast Search TSS and Hierachical search (has fixed clock property) Pipelining blocks for speed up

Search WindowMemory

(DRAM/SRAM)

Current MB(SRAM)

STEP1

(+/-4

Step 2

(+/-2)

Step3

(+/-1)

Step 4

(+/-1/2)

t=4 block4 block 3 block 2 block 1

t= 3 block 3 block 2 block 1

t=2 block 2 block 1t =1 block 1

Interframe Coding

Documents

video compressionp

search center

resolution search

time search

w 111935osa1

w of mvx1

mvy2heejune ahn

log2 w 172533mmea1