Entropy Coding Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2013 Last updated 2013. 10. 28.

Entropy Coding

Heejune AHNEmbedded Communications Laboratory

Seoul National Univ. of TechnologyFall 2013

Last updated 2013. 10. 28

Heejune AHN: Image and Video Compression p. 2

Agenda

[Midterm exam solution] Entropy Coding in Video Codec Basic Information Theory on Entropy coding Huffman Coding and Implementation Arithmetic Coding and Implementation


1. Entropy Coding in Video Coding

Video Encoder Components A source model + an Entropy Coder

Design Goals of entropy coder (lossless) Compression efficiency : smaller output bit count Computational efficiency: Implementation suitability Error robustness and access efficiency: easy to synchronize

Entropy coderSource ModelTexture Info

Block Motion

Transform

Quantizer

Symbolmapper

Entropy coders

Motion Info

Control/Mode info


Data symbol Mapping

Symbols Input to (General) Entropy Coder Well-defined Information units from real-raw data (numbers) e.g.) Coefficients in blocks, Motion Vectors for MBs, Coding Modes

for Video Coding

(DCT) Coefficient symbol mapping Properties

• Sparse

• non-zeros Near DC (0, 0)

• vertical and horizontal : even or not Solution

• Zig-zag scanning

• (Run, Length) symbol mapping

102 -33 - 4 - 2 - 1- 3

21 -2 - 1- 3

- 3 - 1

2

1 1

- 2


Zigzag Scanning Frame picture Field Picture (horizontal-first scan)

Run-Length (run, length) : 2-D symbol mapping in H.261 and MPEG-1/2 EOB (end of block) : extra symbol for all remaining zeros (run, length, last) : 3-D in H.263 and MPEG-4 Intra DC is differential coded form previous block Do the text example (p 166)

X

Y


Motion Vector Symbols Differential coded

• Other H.261

Quaitzation Parameters Differential coded : DQ = Qcurrent -Qprev No abrupt quant step change (-> abrupt quality change)

MVMV1

MV

MV3MV2

MV1

pi = Median (mv1i, mv2i, mv3i)mvdi = mvi -pi


Coded Mode Coding Mode

• Motion compensated or not, Coded or not etc block has data or not

• No texture info

Synchronization Headers High-Level Header

Global parameters (coding options): resolution, pixel aspect ratio, quantization options, etc

Change MQUANT

Change MQUANT

Change MQUANT

Change MQUANT

No Change to MQUANT

No Change to MQUANT

No Change to MQUANT

No Change to MQUANT

P picture

Perform motion compensation

Set motion vector to zero

Inter

Intra

Coded

Not Coded

Coded

Not Coded (Skipped)

Inter

Intra

Y

3 4

1 2

5 6

16

8

16 8

Cr Cb

휘도신호 색차신호


2. Entropy Coding Background

Definition of Entropy (by. C. Shannon)

Input symbols space & occurrence prob. :

Information Uncertainty = information (when revealed) length, amount of info for a symbol Rare Events (p = 0.0000001, e.g. ..) => big news ! => more info

Interpretation (the implication) The amount of Information of S (Qualitatively)

• The less predictable, the more information Shannon, Information theory (Quantitatively)

• H(S) is the theoretical limit (upper-bound) bits for representing S.

H S I S( ) [ ( )]

)( )1

(log)( 2 bitsp

psIpmm

mmm

m

{ , , , }a a aM0 1 1 )( mrm asPp

I spmm

( ) log( )1


Binary Signal Example (0,1) Entropy

Entropy is maximum at p = ½ , ie. equi-probable.

H p p p p p2 1 1( ) log ( ) log( )

p

1.0

0.0 0.5 1.0

0.0

0.5

H p2 ( )

1 2 H p( )


Entropy coding

Concept Assign shorter length code for more frequent symbol, Average

code-length is reduced So, VLC (variable-length-coding) Lossless coding algorithm: used in File compression

An example Let code a file with “squeeze” {s, q, u, e, z} with prob. { 1/7, 1/7, 1/7, 3/7, 1/7} Fixed length coding

• 3 bits (5 symbol)* 7 symbol = 21 bits VLC coding

• {u, s, u, q, z} => {‘1’, ‘000’, ‘001’, ‘010’, ‘011’}

• 1bit* 3symbols + 3bits* 4symbols = 3 + 12 = 15 bits ( << 21 bits)


Coding algorithm

How to code? Still “Open” Problem Welcome you if you dare

Huffman Coding Integer length Code Book Used in H.261, H.263, MPEG-2 Pioneered David Huffman (in 1952)

Arithmetic Coding Factional length Coding file compression (zip) Used in H.263, MPEG-4, H.264 Pioneered by Jorma Rissanen

David Huffman

Jorma Rissanen


Prefix-free coding

Prefix-free coding A code is not same as the prefix part of any other code So, uniquely decodable by scanning from the start Huffman Code is a kind of prefix code Compare two codes

symbol codeword

a1 a2 a3 a4

0101010101

Symbol codeword

a1 a2 a3 a4

0101101110


Huffman Coding

Mapping a symbol to a variable length code variable & integer length code (an optimal integer length code) uniquely decodable (prefix-free)

Huffman Code Generation Step 1 : Order the symbols in increasing order of probability Step 2 : Take two smallest, assign ‘1’ to the larger probability symbol and ‘0’ to the

smaller. Step 3 : Combine the two smallest-probability symbols into a node (group), and

assign the combined probability to the node Step 4 : IF only one node remains, STOP, ELSE reorder the resultant symbols

(considering the combined node as one symbol)


Huffman Encoding & Decoding

Encoding Table lookup Symbol => code (code value, length)

Decoding Comparing codes bit by bit Bit stream => boundary (code value and length) => symbol

0 0001

1 11

2 10

3 000001

4 00001

… 01

N-3 001

N-1 000000

01 …

11 1

10 2

001 N-3

0001 0

00001 4

000001 3

000000 N-1

Encode Table Decode Table


An Example

Motion Vector Code book Generation (See Fig 8.5) Encoding example

• 3 motion values, 1, 0, 0.5 => 01011/1/011 Decoding example

• 01011/1/011 => 1, 0, 0.5

mvx Prob. log2(1/P)

-1.5 0.014 6.16

-1 0.024 5.38

-0.5 0.117 3.10

0 0.646 0.63

0.5 0.101 3.31

1 0.027 5.21

1.5 0.016 5.97

mvx Code bit length

(Huffman)

Bit length

(Ideal)

0 1 1 0.63

-0.5 00 2 3.10

+0.5 011 3 3.31

-1.5 01000 5 5.21

1.5 01001 5 5.38

-1 01010 5 5.97

1 01011 5 6.16


Code Table Design

Ideal Huffman Code Input statistics adaptive; performance Disadvantages

• Statistics calculation burden (big data)

• Transmission overhead from an encoder to decoders

• Implementation difficulty

Generic Huffman code based on large set data (various video sequences) Disadvantages

• Lower performance possible

Video coding standards Provides generic code with optional extension


Code table design example

H.263/MPEG-4 TCOEF 3 Dimensional (last, run, level)

• Ideal table size = 2 x 63 x 128 x 2 (sign). Huge!! Only 102 most frequent values has specific code (upto 13 bits)

• Others : ESCAPE Code (0000011) + 13 bit (last, run, level)

• Note : Fixed-length code is better for very small probability codes No successive 8 0’s allowed

• Used for synchronization start code


26 partial list ‘s’ : sign bit Last Run LEVEL Code

0 0 1 10s

0 1 1 110s

0 2 1 1110s

0 0 2 1111s

1 0 1 0111s

0 3 1 01101s

0 4 1 01100s

0 5 1 01011s

0 0 3 010101s

0 1 2 010100s

0 6 1 010011s

0 7 1 010010s

0 8 1 010001s

0 9 1 010000s

Last Run LEVEL Code

1 1 1 001111s

1 2 1 001110s

1 3 1 001101s

1 4 1 001100s

0 0 4 0010111s

0 10 1 0010110s

0 11 1 0010101s

0 12 1 0010100s

1 5 1 0010011s

1 6 1 0010010s

1 7 1 0010001s

1 8 1 0010000s

ESCAPE 0000011s

.etc


H.263/MPEG-4 MVD (motion vector difference) dmvx and dmvy See table 8.6 and Figure 8.8 MVD = 0 is 1 bit (‘1’)

H.264 Universal VLC (UVLC) Systemic (Universal) table generation (just need ordering) 1 1-bit code, 2 3-bit codes, 4 5-bits codes…. Easy to implementation and statistics adaptive

1 => most frequent 1 symbol

0 x0 1 => the next 2 symbols

0 x1 0 x0 1 => the next 4 symbols

0 x2 0 x1 0 x0 1 => …

0 X3 0 x2 0 x0 1

…


Example

Textbook Coeff matrix

(0,0,4), (0,0,-1), (0,1,2), (0,0,-3), (0,5, -1), (1,3,1) Final bit stream

• 00101110/101/0101000/010111/0011010

4 -1 1

-3

2

-1


Implementation Issues

Encoder Length required (because of Variable length)

Decoder

Packing Register(Shifter)

Lookup Or Calculate VLC

Code

Length

Symbol

Table select

Output bitstream

Packing Register(Shifter)

Lookup Or Calculate VLC

bitstream

Length

Symbol

Error

Input bitstream


Difficulties Table Size is huge

• num of last X num of run X num level

• H.263/MPEG-4 TCOEF => 16002 entries

• over 64 MB TLB for Encoder

• Over 16K Comparator for Decoder

Complexity Reduction in Encoder

If(|level| < 13 && run < 39){ output TLB[last][run][level];}else{ output ESCAPE CODE; output last; output run; output level;}

Switch(last){ case .. switch(run){ case .. switch(level){ } …….

Table Based Method

Code Based Method

If((last, run, level) in most frequent range){ output TLB1[last][run][level];}else if((last, run, level) in 2nd frequent range){

} else ….


Complexity reduction in decoder

do{Bi = take first Li bits from stream;symbol = TLB1[Bi];

}while( symbol != continue);

01 …

11 1

10 2

001 NA

001 N-3

0001 0

00001 4

000001 3

000000 N-1

01 …

11 1

10 2

001 N-3

0001 0

00001 4

000001 3

000000 N-1


Error Handling Cause of error

• Transmission error, storage error, system errors

• packet loss, bit-reversal, bit-erosion Error detection

• Undefined code (in syntax), visual corruption (in semantics) Error recovery

• Even after Forward error correction/Feedback Recovery

• Synchronization Code (slice or picture level)

• Reversible VLC code (in MPEG-4)

Real Error

Error detection

Sync. code


Arithmetic Coding

Motivating example Table 8.1: 0.5 (p = 0.101) ideally 3.31 bits but 5 bits assigned Why ideal and why 5 bit assigned ?

Arithmetic coding Assign code a range not a value (fractional bit code)

Better compression than Integer Coding • Especially one big symbol

Patent• IBM: Q-Coder

))( ),1([),[)( iiiii aCDFaCDFHLaI

)( ir aP

(-2) (-1) (0) (+1) (+2)0.0 0.1 0.3 0.7 0.9 1.0


)()( 11 nonnn xIRLxI ),[ nn HL

)( ofend low

1nxI

)( of (length) range 1nxI

)( updatedor initial by the

))( ),([ i.e., ,),[ interval symbol 1

ir

iiii

aP

aCDFaCDFHL

nnn LHR 1 ; range

Arithmetic Encoding Algorithm

Step 1: Step 2: Calculate Range for n-th symbol

Repeat remaining symbols or Stop Send any value in the range

Normally the shortest representation (e.g. binary)

1,0,101 00 HLRo


Example: Encoding and Decoding

Input symbols 0, -1, 0, 2

(0)0.0 0.1 0.3 0.7 0.9 1.0

(-1)0.3 0.34 0.42 0.58 0.66 0.7

(0)0.34 0.348 0.364 0.396 0.412 0.42

(2)0.364 0.3672 0.3736 0.3864 0.3928 0.396

0.394531 => check !!


More Issues on Arithmetic Coding

Performance 0.394531 =0.011001010b => 9 bits Theoretic min = 0.828 bits

Termination method Why I stopped decoding with 4 symbols? I knew that 4 symbols required (predefined length) Extra “End of Sequence” code

Fixed point arithmetic no symbolic computing CPU yet. All floating point is not exact. H.263/4 14 bits accuracy, i.e. smallest probability is 1/16384 Shift the determined MSB bits and contains only LSB bits in buffer

01011111010100101010010101Unchangeable after new symbol range calculation


Conclusion

Entropy coding Symbol to bit stream mapping and reverse Data compression using statistical redundancy approaching to the

theoretical limit (entropy) Integer length code like Huffman code Fractional length codes like Arithmetic code

Entropy coding in video coding Texture (DCT coefficients) Motion Vectors Mode and side information Huffman code is traditional tool for entropy coding Arithmetic code is new stronger tool but not widely used yet.

Entropy Coding Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2013 Last updated 2013. 10. 28.

Documents

video compressionp

entropy codinghuffman

coding modes

xyheejune ahn

implementationheejune

entropy codingconcept

length symbol mapping102

frequent symbol