FEC and RDO in SVC Thomas Wiegand 1. Outline Introduction SVC Bit-Stream Raptor Codes Layer-Aware FEC Simulation Results Linear Signal Model Description.

FEC and RDO in SVC

Thomas Wiegand

1

Outline

• Introduction• SVC Bit-Stream• Raptor Codes• Layer-Aware FEC• Simulation Results• Linear Signal Model• Description of the Algorithm• Experimental Results

2

Introduction• C. Hellge, T. Schierl, and T. Wiegand, “RECEIVER DRIVEN

LAYERED MULTICAST WITH LAYER-AWARE FORWARD ERROR CORRECTION,” ICIP 2008.

• C. Hellge, T. Schierl, and T. Wiegand, “MOBILE TV USING SCALABLE VIDEO CODING AND LAYER-AWARE FORWARD ERROR CORRECTION,” ICME 2008.

• C. Hellge, T. Schierl, and T. Wiegand, “Multidimensional Layered Forward Error Correction using Rateless Codes,” ICC 2008.

• M. Winken, H. Schwarz, and T. Wiegand, “JOING RATE-DISTORTION OPTIMIZATION OF TRANSFORM COEFFICIENTS FOR SPATIAL SCALABLE VIDEO CODING USING SVC,” ICIP 2008.

3

SVC Bit-Stream

• Spatial-temporal-quality cube of SVC

4http://vc.cs.nthu.edu.tw/home/paper/codfiles/kcyang/200710100050/Overview_of_the_Scalable_Video_Coding_Extension_of_the_H.264.ppt

RECEIVER DRIVEN LAYERED MULTICAST WITH LAYER-AWARE FORWARD ERROR CORRECTION

C. Hellge, T. Schierl, and T. Wiegand

ICIP 2008

5

C. Hellge, T. Schierl, and T. Wiegand, “MOBILE TV USING SCALABLE VIDEO CODING AND LAYER-AWARE FORWARD ERROR CORRECTION,” ICME 2008.C. Hellge, T. Schierl, and T. Wiegand, “Multidimensional Layered Forward Error Correction using Rateless Codes,” ICC 2008.

SVC Bit-Stream

• Equal FEC

6

Raptor Codes (1/2)

• Non-systematic Raptor codes

0

1

2

3

0

1

2

3

4

5

Gp

0

1

2

3

4

5

GLT

0

1

2

3

4

5

6

7

= =

7

precoding process LT coding process

SSs PSs PSs ESs

Raptor Codes (2/2)

• Systematic Raptor codes

• Construction of pre-code symbols– GLT , Gp, and SSs.

– GpSys =

– Solving

0 0

k

n-1

Gp GLT = =

0 0

GpSys =0

p-1

0

k-1

0

…

…

k-1 p-1 p-1 k-1

…

… …

……

GLT’

GLT’’

k 0

p-1

…

p

s

k

0

k-1

…

?

=

8

unknown

unknown

Gp I

GLT’

s

k

k s

p

Layer-Aware FEC (1/5)

• Example 1

• Example 2

9

Layer-Aware FEC (2/5)

• Encoding process– Example 3

10

Layer-Aware FEC (3/5)

• Decoding process– Example 4

11

Layer-Aware FEC (4/5)

• GLayeredLT(m) = [G*LT0 | G*LT1 | … | GLTm]

12

GLayeredLT(m) =

PSs0

PSs1

PSs2

…PSsm

ESs0

ESs1

ESs2

…ESsm

Layer-Aware FEC (5/5)

• GpSysLayered(m)

13

0

SS0

0

SS1

0

GpSysLayered(m) =

PSs0

PSs1

PSs2

…PSsm

0

ESs0 0

ESs1…

0

ESsm

Simulation Results (1/2)

• QVGA (BL) and VGA (EL) resolution using SVC over a DVB-H channel.– JSVM 8.8– GOP size = 16

• Size of a transmission block = 186 bytes• Mean error burst length = 100 TBs

14

Simulation Results (2/2)

15

Joint Rate-Distortion Optimization of Transform

Coefficients For Spatial Scalable Video Coding Using SVC

M. Winken, H. Schwarz, and T. Wiegand

ICIP 200816

Hybrid Video Decoding

17

1 2

3 4

5 6

7 8

s5

s6

s7

s8

s2

s3

½ (s2+s3)sx

u5

u6

u7

u8

s1

s2

s3

s4

s5

s6

s7

s8

0000c5

c6

c7

c8

s1

s2

s3

s4

000sx

0 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 1 0 0 0 0 0 00 0 ½ ½ 0 0 0 0 0 1 0 0 0 0 0 00 0 0 0 0 0 0 0

s1

s2

s3

s4

s5

s6

s7

s8

0 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 ? ? ? ?0 0 0 0 ? ? ? ? 0 0 0 0 ? ? ? ?0 0 0 0 ? ? ? ?

= + +

= +

Motion compensatedvalues

Dequantized residual

Decoded pixel values

Motion vectors DeQuntized andiDCT parameters

Motion compensation iQ and iDCT Exception

Linear Signal Model (1/6)

• Linear signal model for K inter frames– s = Ms + Tc + p

• s: A (KWH)1 vector of decoded signal• M: A (KWH)(KWH) matrix of motion parameters• T: A (KWH)(KWH) matrix of inverse quantization

and DCT parameters• c: A (KWH)1 vector of received transform coefficients• p: A (KWH)1 intra signal or motion parameters outside

s s11

…s1

WH

…sK

1

…sK

WH

s11

…s1

WH

…sK

1

…sK

WH

c11

…c1

WH

…cK

1

…cK

WH

p11

…p1

WH

…pK

1

…pK

WH

= + +1 K K+1

18

WH

W

H

WH

W

H

Linear Signal Model (2/6)

• Optimal transform coefficients selection– Decoder receives MVs (M) and quantized

transform coefficients (c).– fixed motion parameters (M), quantization

parameters (T), and intra predictions (p).• Rate and distortion are mainly controlled by c.

– c’ = argminc{D(c) + R(c)}

subject to s = Ms + Tc + p• D(c) = ||x - s||2

2, R(c) = ||c||1

19

Linear Signal Model (3/6)

• Optimal transform coefficients selection– Problem: MVs cannot be determined before the

transform coefficients are selected (trade-off)– Solution:

20

s11

…s1

WH

s21

…s2

WH

s31

…s3

WH

…sK

1

…sK

WH

c11

…c1

WH

c21

…c2

WH

c31

…c3

WH

…cK

1

…cK

WH

p11

…p1

WH

p21

…p2

WH

p31

…p3

WH

…pK

1

…pK

WH

= + +

s11

…s1

WH

s21

…s2

WH

s31

…s3

WH

…sK

1

…sK

WH

fixed

fixed

initial

initial

initial

initial

Linear Signal Model (4/6)

• Optimal transform coefficients– Problem size: K W H

• Sliding window approach (Reduce problem size)– s = M s + T c + p

21

window size

step size

Linear Signal Model (5/6)

• Extension for spatial scalability– s0 = M0s0 + T0c0 + p0

– s1 = M1s1 + T1c1 + p1 + Bs0 + RT0c0

H.264/AVC MCP & Intra-prediction

Hierarchical MCP & Intra-prediction

Base layer coding

Base layer coding

texture

motion

texture

motion

Inter-layer prediction•Intra•Motion•Residual

Spatial decimation

Multiplex Scalable bit-stream

H.264/AVC compatible coder

H.264/AVC compatible base layer bit-stream

22

Inter-layer motion prediction

Inter-layer residual prediction

Linear Signal Model (6/6)

• Optimal transform coefficients in spatial scalability– c0’ D0(c0) + 0R(c0)

c1’ D1(c0,c1) + 1(R(c0)+R(c1))

subject to s0 = M0s0 + T0c0 + p0

s1 = M1s1 + T1c1 + p1 + Bs0 + RT0c0

c0’ (1-w)(D0(c0) + 0R(c0)) +

c1’ w(D1(c0,c1) + 1(R(c0)+R(c1)))

where = (W1H1)/(W0H0)

= argminc0’c1’

23

= argminc0’c1’

Description of the Algorithm

• Determine M0, T0, M1, T1, B, p0, R, and p1 by encoding the first K pictures using SVC reference encoder model.

• Solve optimization to determine c0 of the base layer.

• Based on new c0, determine B and R again.

• Solve optimization problem for only the enhancement layer.

24

Experimental Results (1/2)

• JSVM 9.9– IPPP– QCIF (base layer) and CIF (enhancement layer)– CABAC– QP difference: 3– Sliding windows size: 55 for base layer and

1010 for enhancement layer

25

Experimental Results (2/2)

26