Digital Video Compression Fundamentals and Standards Web Technology.

Digital Video Compression Digital Video Compression Fundamentals and StandardsFundamentals and Standards

Web TechnologyWeb Technology

2008/12/262008/12/26 Digital Video Compression Fundamentals anDigital Video Compression Fundamentals and Standardsd Standards

22

OutlineOutline

IntroductionIntroduction

Video Compression StandardsVideo Compression Standards

Simulation Reference SoftwareSimulation Reference Software

Future Work and ConclusionsFuture Work and Conclusions


33

OutlineOutline






44

Introduction (1/2)Introduction (1/2)

Why video compression technique is Why video compression technique is important ?important ?

One movie video without compressionOne movie video without compression720 x 480 pixels per frame720 x 480 pixels per frame

30 frames per second30 frames per second

Total 90 minutesTotal 90 minutes

Full colorFull color

The total quantity of data = 167.96 G Bytes !!


55

Introduction (2/2)Introduction (2/2)

What is the difference between video compresWhat is the difference between video compression and image compression?sion and image compression?

Temporal RedundancyTemporal Redundancy

Coding method to remove redundancyCoding method to remove redundancyIntraframe CodingIntraframe Coding

Remove spatial redundancyRemove spatial redundancy

Interframe CodingInterframe CodingRemove temporal redundancyRemove temporal redundancy


66

The most intuitive method to remove The most intuitive method to remove Spatiotemporal redundancySpatiotemporal redundancy

3-Dimensional DCT3-Dimensional DCTRemove spatiotemporal correlationRemove spatiotemporal correlation

Good for low motion videoGood for low motion video

Bad for high motion videoBad for high motion video

1 1 1

30 0 0

8 (2 1) (2 1) (2 1)( , , ) ( ) ( ) ( ) ( , , ) cos cos cos

2 2 2

N N N

t x y

x u y v t wF x y t C u C v C w x y t

N N N N

for 0,..., 1 , 0,..., 1 and 0,..., 1

1/ 2 for 0where 8 and ( )

1 otherwise

u N v N w N

kN C k


77

The most popular method to remove The most popular method to remove temporal redundancytemporal redundancy

The Block-Matching AlgorithmThe Block-Matching Algorithm


88

Matching FunctionMatching Function

The dissimilarity between two blocks andThe dissimilarity between two blocks and

The matching criteriaThe matching criteriaMean square error (MSE)Mean square error (MSE)

High precision is neededHigh precision is needed

Mean absolute difference (MAD)Mean absolute difference (MAD)

Low precision is enoughLow precision is enough

2( , ) ( - )M u v u v

( , ) | - |M u v u v

1

1 , 1

( , ) [ ( , ), ( , )]p q

n n x y

Vy Vx

D s t M x y x V y V

n 1n( , )D s t


99

The Exhaustive Block-Matching Algorithm

Reference Frame Current Frame

Search Range

MotionVector

11 22 33 44 55 66 77 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 88

12 22 33 44 55 66 77 8812 22 33 44 55 66 77 8812 22 33 44 55 66 77 8812 22 33 44 55 66 77 8811 23 34 44 55 66 77 8811 23 34 44 55 66 77 8811 23 34 44 55 66 77 8811 23 34 44 55 66 77 88

1 0 0 0 0 0 0 01 0 0 0 0 0 0 01 0 0 0 0 0 0 01 0 0 0 0 0 0 00 1 1 0 0 0 0 00 1 1 0 0 0 0 00 1 1 0 0 0 0 00 1 1 0 0 0 0 0

|A|=12


1010

Fast Block-Matching AlgorithmsFast Block-Matching Algorithms

EBMA needs Intensive computationEBMA needs Intensive computation

Fast Algorithm is neededFast Algorithm is neededFind the possible local optimalFind the possible local optimal

720 480128 128 16 16 30 170 G instructions/sec

16 16

Reference Frame Current Frame


1111

Fast Block-Matching Algorithms Fast Block-Matching Algorithms

The characteristics of fast algorithmThe characteristics of fast algorithmNot accurate as EBMANot accurate as EBMA

Save large computationSave large computation

Two famous fast algorithmTwo famous fast algorithm2-D logarithm Search Method2-D logarithm Search Method

Three Steps Search MethodThree Steps Search Method


1212

2-D logarithm Search Method2-D logarithm Search Method

1 1

1

1

1 2

2

2

3

3

4

44

4 4

1

2 3

4


1313

Three Step Search MethodThree Step Search Method

11 1

1

1

2 2

2

2

11

11

2

2

2

2

3 3 3

3 3

3 3 3

1 2

3


1414

Multiresolution Motion Estimation (1/3)Multiresolution Motion Estimation (1/3)

The number of levels is The number of levels is LL

ll-th level images of the target frames-th level images of the target frames

where is set of pixels at level where is set of pixels at level LL

At the At the ll-th level, the MV is -th level, the MV is

At the At the ll-th level, the estimated MV is-th level, the estimated MV is

Determine update such that Determine update such that errorerror is minimized is minimized

The new motion vector isThe new motion vector is

, ( ), , 1, 2,...t l l t x x

l

1( ) ( ( ))l lU d x d x

( )xd

2, 1,| ( ( ) ( )) ( )) |

pll l l

x l

error

x d x q x x

( )lq x

( ) ( ) ( )ll l d x d x q x


1515


Variable block size methodVariable block size method


1616



1717

OutlineOutline






1818

The Development of Video The Development of Video Compression StandardsCompression Standards

…………

5-point 1D DCT

6-point 1D DCT

…………

5-point 1D DCT

6-point 1D DCT…………

5-point 1D DCT

6-point 1D DCT


1919

The MPEG-1 StandardThe MPEG-1 Standard

Group of Pictures

Motion EstimationMotion Estimation

Motion CompensationMotion Compensation

Differential CodingDifferential Coding

DCTDCT

QuantizationQuantization

Entropy CodingEntropy Coding


2020

Group of Pictures (1/2)Group of Pictures (1/2)

I-frame (Intracoded Frame)I-frame (Intracoded Frame)Coded in one frame such as DCT. Coded in one frame such as DCT. This type of frame do not need previous frame This type of frame do not need previous frame

P-frame (Predictive Frame)P-frame (Predictive Frame)One directional motion prediction from a previous frameOne directional motion prediction from a previous frame

The reference can be either I-frame or P-frameThe reference can be either I-frame or P-frameGenerally referred to as inter-frameGenerally referred to as inter-frame

B-frame (Bi-directional predictive frame)B-frame (Bi-directional predictive frame)Bi-directional motion prediction from a previous or future frameBi-directional motion prediction from a previous or future frame

The reference can be either I-frame or P-frameThe reference can be either I-frame or P-frameGenerally referred to as inter-frameGenerally referred to as inter-frame


2121

Group of Pictures (2/2)Group of Pictures (2/2)

The distance between two nearest P-frame or P-frame and I-frameThe distance between two nearest P-frame or P-frame and I-frame

denoted by denoted by MM

The distance between two nearest I-framesThe distance between two nearest I-frames

denoted by denoted by NN

I B B P B B P B B I

GOPGOP

Bidirectional Motion Compensation

Forward Motion Compensation

N=9

M=3


2222

The MPEG-1 Encoder (1/4)The MPEG-1 Encoder (1/4)

DCT QEntropyCoding

Q-1

IDCT

MotionCompensation

FrameMemory

MotionEstimation

DCT QEntropyCoding

DCT Q

Q-1

IDCT

MotionCompensation

FrameMemory

MotionEstimation

Intra-frame

EntropyCoding

Residue

Motion Vector

Inter-frame


2323


Differential CodingDifferential Coding

is the input image is the input image

is the predictive imageis the predictive image

DCT DCT

^

( ) ( ) ( )D t t t ( )t

^

( )t

1 1

0 0

2 (2 1) (2 1)( , ) ( ) ( ) ( , ) cos cos

2 2

for 0,..., 1 and 0,..., 1

1/ 2 for 0where 8 and ( )

1 otherwise

N N

x y

x u y vF u v C u C v f x y

N N N

u N v N

kN C k


2424


QuantizationQuantizationIntra quantization matrixIntra quantization matrix

Inter quantization matrixInter quantization matrix

intra

8 16 19 22 26 27 29 34

16 16 22 24 27 29 34 37

19 22 26 27 29 34 34 38

22 22 26 27 29 34 37 40

22 26 27 29 32 35 40 48

26 27 29 32 35 40 48 58

26 27 29 34 38 46 56 69

27 29 35 38 46 56 69 83

Q

intra

16 17 18 19 20 21 22 23

17 18 19 20 21 22 23 24

18 19 20 21 22 23 24 25

19 20 21 22 23 24 26 27

20 21 22 23 25 26 27 28

21 22 23 24 26 27 28 30

22 23 24 26 27 28 30 31

23 24 35 27 28 30 31 33

Q


2525

The MPEG-1 Encoder (4/4)The MPEG-1 Encoder (4/4)Motion CompensationMotion Compensation

Exploit motion vector and the previous reconstructed frame to generate Exploit motion vector and the previous reconstructed frame to generate the predictive framethe predictive frame

is the compensated image is the compensated image

is the previous imageis the previous image

is the motion vectoris the motion vector

( , ) ( ( , ), ( , )), ( , ) ( , )n n x yx y x v p q y v p q x y MB p q ( , )n x y

( , )n x y

( ( , ), ( , ))x yv v p q v p q

Reference frameReference frame Target frameTarget frame


2626

The MPEG-2 StandardThe MPEG-2 Standard

Field/Frame DCT CodingField/Frame DCT Coding

Field/Frame Prediction Mode SelectionField/Frame Prediction Mode Selection

Alternative Scan Order Alternative Scan Order

Various Picture Sampling FormatsVarious Picture Sampling Formats

User Defined Quantization MatrixUser Defined Quantization Matrix


2727

Progressive Scan and Interlaced Progressive Scan and Interlaced ScanScan

Progressive Scan

Interlaced Scan


2828

Field/Frame DCT CodingField/Frame DCT Coding

The field type DCT The field type DCT Fast motion videoFast motion video

The frame type DCT The frame type DCT Slow motion videoSlow motion video

Field DCT Coding Frame DCT CodingLuminance MB


2929

Alternative Scan OrderAlternative Scan Order

Zigzag scan orderZigzag scan orderFrame DCTFrame DCT

Alternative scan orderAlternative scan orderField DCTField DCT

Zig-zag scan Alternate scan

1 1 1 1 1 1 1 10 0 0 0 0 0 0 01 1 1 1 1 1 1 10 0 0 0 0 0 0 01 1 1 1 1 1 1 10 0 0 0 0 0 0 01 1 1 1 1 1 1 10 0 0 0 0 0 0 0

4 0 0 0 0 0 0 00.72 0 0 0 0 0 0 00 0 0 0 0 0 0 00.85 0 0 0 0 0 0 00 0 0 0 0 0 0 01.27 0 0 0 0 0 0 00 0 0 0 0 0 0 03.62 0 0 0 0 0 0 0

2D DCT2D DCT


3030

The MPEG-2 Encoder (2/2)The MPEG-2 Encoder (2/2)QuantizationQuantization

User can change the quantization if necessaryUser can change the quantization if necessary

Intra quantization matrixIntra quantization matrix

Inter quantization matrixInter quantization matrix

Various picture sampling formatsVarious picture sampling formats

4:4:44:4:4

4:2:2 4:2:2

4:2:0 4:2:0

intra

8 16 19 22 26 27 29 34

16 16 22 24 27 29 34 37

19 22 26 27 29 34 34 38

22 22 26 27 29 34 37 40

22 26 27 29 32 35 40 48

26 27 29 32 35 40 48 58

26 27 29 34 38 46 56 69

27 29 35 38 46 56 69 83

Q

inter

16 16 16 16 16 16 16 16

16 16 16 16 16 16 16 16

16 16 16 16 16 16 16 16

16 16 16 16 16 16 16 16

16 16 16 16 16 16 16 16

16 16 16 16 16 16 16 16

16 16 16 16 16 16 16 16

16 16 16 16 16 16 16 16

Q


3131


Base LayerBase LayerBasic quality requirementBasic quality requirement

For SDTVFor SDTV

Enhanced LayerEnhanced LayerHigh quality serviceHigh quality service

For HDTVFor HDTV

DCT QEntropyCoding

Q-1

IDCT

MotionCompensation

FrameMemory

MotionEstimation

QEntropyCoding

Q-1

+

+

++

+

Q-1

Bits Enhance

SNR Enhanced Layer

+

Base Layer

Bits Base


3232

H.264/AVCH.264/AVC

Variable Block SizeVariable Block Size

Multiple Reference FramesMultiple Reference Frames

Integer TransformInteger Transform

Intra PredictionIntra Prediction

In-loop Deblocking FilteringIn-loop Deblocking Filtering

1/4-pel Resolution Motion Estimation 1/4-pel Resolution Motion Estimation

CAVLCCAVLC


3333

Variable Block SizeVariable Block Size

The fixed block size may not be suitable for all The fixed block size may not be suitable for all motion objectsmotion objects

Improve the flexibility of comparisonImprove the flexibility of comparison

Reduce the error of comparisonReduce the error of comparison

7 types of blocks for selection7 types of blocks for selection

00

10

2 3

0 11

16 x 16 16 x 8 8 x 16 8 x 8

00

10

2 3

0 11

8 x 8 8 x 4 4 x 8 4 x 4


3434

Multiple Reference FramesMultiple Reference Frames

The neighboring frames are not the most similar in some casesThe neighboring frames are not the most similar in some cases

The B-frame can be reference frameThe B-frame can be reference frameB-frame is close to the target frame in many situations B-frame is close to the target frame in many situations


3535

Integer Transform for Reducing The Integer Transform for Reducing The Spatial Redundancy (1/2)Spatial Redundancy (1/2)

The transform matrix The transform matrix CC

4×4 Block Size4×4 Block Size

Separable Integer TransformSeparable Integer Transform

The transform coefficients are The transform coefficients are CXCCXCTT

1 1 1 1

2 1 1 2

1 1 1 1

1 2 2 1

C


3636

Integer Transform for Reducing The Integer Transform for Reducing The Spatial Redundancy (2/2)Spatial Redundancy (2/2)

00 01 02 03

10 11 12 13

20 21 22 23

30 31 32 33

0.5 0.5 0.5 0.5 0.5 0.6325 0.5 0.3162

0.6325 0.3162 0.3162 0.6325 0.5 0.3162 0.5 0.6325

0.5 0.5 0.5 0.5 0.5 0

0.3162 0.6325 0.6325 0.3162

T

x x x x

x x x xY AXA

x x x x

x x x x

.3162 0.5 0.6325

0.5 0.6325 0.5 0.3162

00 01 02 03

10 11 12 13

20 21 22 23

30 31 32 33

1 1 1 1 0.5 0.6325 0.5 0.3162

2 1 1 2 0.5 0.3162 0.5 0.6325=

1 1 1 1 0.5 0.3162 0.5 0.6325

1 2 2 1 0.5 0.6325 0.5 0.3162

x x x x

x x x x

x x x x

x x x x

00 01 02 03

10 11 12 13

20 21 22 23

30 31 32 33

0.5 0.5 0.5 0.5

0.3162 0.3162 0.3162 0.3162

0.5 0.5 0.5 0.5

0.3162 0.3162 0.3162 0.3162

1 1 1 1 1 2 1 1

2 1 1 2 1 1 1 2=

1 1 1 1 1 1 1 2

1 2 2 1 1 2 1

x x x x

x x x x

x x x x

x x x x

0.25 0.1581 0.25 0.1581

0.1581 0.1 0.1581 0.1

0.25 0.1581 0.25 0.1581

1 0.1581 0.1 0.1581 0.1

CC XX CCTT EE


3737

Intra PredictionIntra PredictionPredict the similarity between the neighboring pixels in one Predict the similarity between the neighboring pixels in one frame in advance, and exploit differential coding transform frame in advance, and exploit differential coding transform coding to remove the redundancy. coding to remove the redundancy.

VerticalHorizontal + + + +

++++

Mean

DC Diagonal down-left

Horizontal up

Diagonal right

Vertical right Vertical leftHorizontal down

Intra Prediction

Transform/Quantization

EntropyCoding


3838

Remove Perceptual RedundancyRemove Perceptual Redundancy

In-loop deblocking filteringIn-loop deblocking filteringRemove blocking artifactRemove blocking artifact

Result from block based motion compensationResult from block based motion compensation

Result from block based transform codingResult from block based transform coding

p3 p2 p1 p0

q0 q1 q2 q3

QP


3939

1/4-pel Resolution Motion 1/4-pel Resolution Motion Estimation Estimation

Integer search positions

Best integer match

Half-pel search positions

Best half-pel match

Quarter-pel search positions

Best quarter-pel match


4040

The H.264/AVC EncoderThe H.264/AVC Encoder

Transform/Quantization

EntropyCoding

InverseTransform/

De-QuantizationMotion

Compensation

De-blockingFilter

MotionEstimation

Residue

Motion Vector

Intra-framePrediction

CoderController

Control Data


4141

OutlineOutline






4242

H.264 Reference SoftwareH.264 Reference Software

JMJMCurrent software version: JM 15Current software version: JM 15

BenchmarkBenchmark

http://iphome.hhi.de/suehring/tml/


4343

Reference Software DemoReference Software Demo


4444

OutlineOutline






4545

Future WorkFuture Work

Fast Mode Decision AlgorithmFast Mode Decision Algorithm

Interpolation Filter DesignInterpolation Filter Design

Deblocking Filter DesignDeblocking Filter Design

DCT-Based Motion EstimationDCT-Based Motion Estimation

Implementation Based on TI DSPImplementation Based on TI DSP


4646

Fast Mode Decision AlgorithmFast Mode Decision AlgorithmThe computational cost of H.264 is large

Variable block-size ME

Multiple reference frames

Fast mode decision is needed for reducing the computation time


4747

Interpolation Filter DesignInterpolation Filter DesignIn order to estimate and compensate the fractional-pel displacements

Adaptive Interpolation filter for replacing the fixed coefficient filter


4848

Deblocking Filter DesignDeblocking Filter DesignBlock based ME and Transform result in the annoying Block based ME and Transform result in the annoying blocking artifactblocking artifact

Reduce the blocking artifact can increase the quality of Reduce the blocking artifact can increase the quality of compressed videocompressed video


4949

DCT Based Motion EstimationDCT Based Motion Estimation

Robust even in noisy environmentRobust even in noisy environment

Complexity comparisonComplexity comparisonDCT Based MEDCT Based ME

O(MO(M22))

Block Based MEBlock Based MEO(NO(N22××MM22))

ConceptConceptPseudo PhasePseudo Phase

Similar toSimilar to { ( )} { ( )}jF x t e F x t


5050

Implementation Based on TI DSPImplementation Based on TI DSP

TMS320C6416TMS320C6416

TI DM642 TI DM642


5151

ConclusionsConclusions

Thank YouThank You

Q & AQ & A


5252

References (1/2)References (1/2)[1] Yun Q.Shi and Huifang Sun, “Image and Video Compression for Multimedia Engineering: Fundamentals, Algorithms, and Standards”, CRC press, 2000.[2] Yao Wand, Jorn Ostermann and Ya-Qin Zhang, “Video Processing and Communications”, Prentice Hall, 2007.[3] Richardson, Lain E. G., “Video Codec Design: Developing Image and Video Compression Systems”, John Wiley & Sons Inc, 2002.[4] Barry G, Haskell, Atul Puri and Arun N. Netravali, “Digital Video : An Introduction to MPEG-2”, Boston : Kluwer Academic, 1999.[5] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard”, IEEE Trans. on Circuits and systems for video Technology, vol. 13, no. 7, pp. 560-576, July 2003.[6] G. Sullivan and T. Wiegand, “Video Compression - From Concepts to the H.264/AVC Standard”, Proceedings of the IEEE, Special Issue on Advances in Video Coding and Delivery, December 2004.[7] 酒井善則、吉田俊之共著，白執善編譯，“影像壓縮技術”，全華，2004.


5353

References (2/2)References (2/2)[8] Thomas Wedi, “Adaptive Interpolation Filters and High-Resolution Displacements For Video Coding”, IEEE Trans. on Circuits and Systems For Video Technology, vol.[9] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhan Urhan, Sarp Ertürk and Tae-Gyu Chang, “Optimal Post-Process/In-Loop Filtering for Improved Video Compression Performance”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007.[10] Shu-Fa Lin, Meng-Ting Lu, and Homer Chen, ” Fast Multi-Frame Motion Estimation for H.264 and Its Applications to Complexity-Aware Streaming”, IEEE International Symposium on Circuits and Systems Circuits and Systems, 2005., 2005.[11] Kai-Ting Cheng and Hsueh-Ming Hang, “Acceleration and Implementation of H.264 Encoder and Scalable Extension of H.264 Decoder on TI DSP Platform”, master thesis, June 2007

Digital Video Compression Fundamentals and Standards Web Technology.

Documents

video compression technique

image compression

movie video

variable block size

low motion videobad

popular method

intuitive method

llth level images