Recent Developments in Video Compression … Introduction – The MPEG and ITU-T Standardization Bodies Recent Developments in Video Compression Standards: from Scalable to Multi-view

1

Recent Developments in Video Compression Standards: from Scalable to Multi-view Video CodingAuthors: Bart Masschelein(*), Jiangbo Lu(*†), and Iole Moccagatta(*),

* Multimedia Group, IMEC, Kapeldreef 75, B-3001, Leuven, Belgium† Department of Electrical Engineering, University of Leuven, Belgium

Presented by Bart Masschelein

2007 IEEE International Conference on Consumer Electronics (ICCE)January 11th, 2007

Recent Developments in Video Compression Standards: from Scalable to Multi-view Video Coding Tutorial presented at ICCE 2007 © imec 2006

2

Outline• Introduction - The MPEG and ITU-T Standardization Bodies [7]

– Overview and accomplishments (MPEG-* and H.26*)

– Joint Video Team

– The MPEG-4 video standards

• Chapter 1 - H.264/MPEG-4 AVC [24]– New (wrt. MPEG-4 Part 2) tools

– Profiles and levels

– Market acceptance and deployment

• Chapter 2 - H.264 Annex G/MPEG-4 Scalable Video Coding [56]– Why do we need scalable video

– MPEG-4 SVC requirements and targeted apps.

– Historical overview and standardization timeline

– Tools enabling scalability

– Media Aware Network Element (MANE)

– Profiles and levels

• Chapter 3 - MPEG-4 Multi-view Video Coding [28]– History, context, and motivations

– Typical multi-view system setups and application scenarios

– Standardization and requirements on MVC

– Basic concepts and principles of MVC

– Coding tools currently considered

• Annex [15]– What else is going on as far as video coding standards (MPEG, ITU-T, SMPTE)

– Bibliography and reference points

– Acronyms and abbreviations

2

Introduction – The MPEG and ITU-T Standardization Bodies


4

The MPEG and ITU-T Standardization Bodies -Outline

• Overview and accomplishments (MPEG-* and H.26*)

• Joint Video Team• The MPEG-4 video standards

3


5

Moving Picture Experts Group (MPEG):ISO-IEC/JTC1/SC29/WG11

WG 1:Coding of still pictures

JPEG

WG 1:Coding of still pictures

JPEG

WG11:Coding of moving pictures and audio

MPEG

WG11:Coding of moving pictures and audio

MPEG

Subcommittee (SC) 29:Coding of audio, picture,

multimedia and hypermedia information

Subcommittee (SC) 29:Coding of audio, picture,

multimedia and hypermedia information

ISOInternational Organization for

Standardization

ISOInternational Organization for

Standardization

IECInternational Electrotechnical

Commission

IECInternational Electrotechnical

Commission

ISO/IECJoint Technical Committee (JTC) 1:

Information technology

ISO/IECJoint Technical Committee (JTC) 1:

Information technology


6

MPEG Standards

• ISO/IEC 11172 MPEG–1 (1992)– Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s

– Storage and retrieval of audio and video on compact disc

– Efficient encoding of non-interlaced pictures with roughly VHS quality at 1,15 Mbit/s

• ISO/IEC 13818 MPEG-2 (1994)– Generic coding of moving pictures and associated audio information, interlace support

– Digital video broadcasting (2-6 Mbps for SD, 10-20 Mbps for HD), DVD video (6-8 Mbps)

• ISO/IEC 14496 MPEG-4 (1998)– Coding of audio-visual objects (from frame-based to object-based coding)

– Coding and integration of natural (aka, pixel based) still images and video sequences with synthetic content (aka, 2D and 3D graphic geometry, i.e. compression of wire grid parameters, synthetic text)

– Efficient coding ranging from very low bit-rate (ex: 5 kbit/s) to more than 1 Gbit/s

– Support progressive as well as interlaced video from sub-QCIF to 'Studio' resolutions (4k x 4k pixels)

– Wide range of targets: from video streaming to real-time video conference, from DTV to interactive graphics applications

• ISO/IEC 15938 MPEG-7 (2001)– Multimedia content description interface

– Different from the previous MPEG standards in the sense that what is represented is not the information itself, but the information about the information

• ISO/IEC 21000 MPEG-21 (200x)– Multimedia framework

– Support environment to deliver and use all content types by different categories of users in multiple application domains

– Define Users (anybody in the value network) and Digital Items (assembly of content) on which Users execute Actions that generate other Digital Items that can become object of Transactions.

4


7

Video Coding Expert Group (VCEG):ITU-T SG 16 Q.6

[Bernd Girod: EE398B Image Communication II Video Coding Standards H.261&H.263 no. 7]


8

ITU-T SG 16 Q.6 Standards

• ITU-T Rec. H.261 (1990)– ISDN picture phones and for video conferencing systems

– Image format: CIF (352 x 288 Y samples) or QCIF (176 * 144 Y samples), frame rate 7.5 to 30 fps

– Bitrate multiple of 64 kbps (= ISDN-channel), typically 128 kbps including audio.

– Stand-alone videoconferencing system or desk-top videoconferencing system, integrated with PC

• ITU-T Rec. H.262 = MPEG-2• ITU-T Rec. H.263 and Annexes (H.263+ and H.263++) (1995)

– International standard for picture phones over analog subscriber lines

– Image format usually CIF, QCIF or Sub-QCIF, frame rate usually below 10 fps

– Arbitrary bitrate (typically 20 kbps for PSTN)

– Software-only PC video phone or TV set-top box

– Compression engine for Internet video streaming

– MPEG-4 Part 2 decoder decodes H.263 Baseline (short header mode)

• ITU-T Rec. H.264 = MPEG-4 AVC– Annex G = MPEG-4 SVC

5


9

Joint Video Team (JVT)

ISO/IECSC29/WG11

ITU-TSG16/VCEG

MPEG-4

MPEG-7

MPEG-21H.261

H.262/MPEG-2

H.263

H.263+

……

JVT

H.264/AVC

SVC

MVC


10

The H.264/MPEG-4 Video Standards

• MPEG-4 Part 2– IS: 2004 (status: 3rd edition)

• ITU-T Rec. H.264/MPEG-4 Part 10 Advanced Video Coding (AVC)– IS: 2005 (status: 3rd edition)

• AVC Amd. 1: Support of colour spaces– FDIS: October 2006 (status: FPDAM)

• AVC Amd. 2: Advanced 4:4:4 profiles– FDIS: January 2007 (status: PDAM)

• ITU-T Rec. H.264 Ann. G/AVC Amd. 3: Scalable Video Coding (SVC)– FDIS: January 2007 (status: PDAM)

• AVC Amd. 4: Multi-view Video Coding (MVC)– FDIS: January 2008 (status: WD)

6

Chapter 1 - H.264/MPEG-4 AVC


12

H.264/MPEG-4 AVC - Outline

• Focus on new (wrt. MPEG-4 Part 2) tools• Video Coding Layer (VCL) and Network Adaptation

Layer (NAL)• Some coding tools in details:

– FMO

– Intra Prediction

– Motion Compensated Prediction

– Transform and Quantization

– In-loop De-blocking Filter

– Entropy Coding

• Profile and Levels• Market Acceptance and Deployment

7


13

H.264/MPEG-4 AVC Layer Structure

Motivation: network adaptation


14

MPEG terminology

• Macroblock– 16x16 luma, 2 8x8 chroma

– macroblocks within a slice depend on each other

– macroblock can be further partitioned

• Slice– a picture can be split into 1

or several slices

– slices are self-contained, (only de-blocking may be performed across slices)

– slices are a sequence of macroblock

– sequence may not be consecutive

8


15

• Transmit macroblocks in non raster scan order

• Slice group– pattern of macroblock defined by a

slice group map

– a slice group may contain from 1 to several slices

• Slice group map types– interleaved

– dispersed (checker board)

– explicitly assign a slice group to each macroblock

– one or more “foreground” slice group and a “leftover” slice group

Slice Group 0

Slice Group 1

Slice Group 2

Flexible Macroblock Ordering (FMO)

Motivation: error resilience and ROI


16

The H.264/MPEG-4 AVC Codec Structure

9


17

Intra Prediction (1/2)

• Spatial domain prediction for Intra macroblocks• 4x4 Intra Prediction (luma)

– 9 modes

– 13 available and reconstructed (no db) border samples of neighboring blocks

– modes are coded w.r.t. “most probable” one

• Example: Diagonal-Left


18

Intra Prediction (2/2)

• 16x16 Intra Prediction (luma)– 4 modes

– 16 vert. +16 horiz. available and reconstructed (no db) border samples of neighboring macroblock

– mode is transmitted

• 8x8 Intra Prediction (chroma)– 4 modes

– 8 vert. +8 horiz. available and reconstructed (no db) border samples of neighboring blocks

– mode is transmitted

• Can predict from Intra or non-Intra macroblock

10


19

H.264/MPEG-4 AVC: Motion Compensated Prediction


20

Quarter Sample Interpolation

• Luma– separable 6-tap filter for ½ pel

– averaging integer and ½ pelposition for ¼ pel

• bilinear interpolation for chroma (down to 1/8)

11


21

Additional Features of H.264/MPEG-4 AVC Mot. Comp.


22

Multiple Reference Frames

• Multiple picture buffer– Decoded Picture Buffer

– FIFO or sliding window

– adaptive memory control

– 16 pictures max (memory is constrained)

• per-8x8 reference control• Bi-predicted picture: 2

sets of motion vector per block

12


23

New Types of Temporal Referencing

• Known dependencies (MPEG-1, MPEG-2, etc.)

• New dependencies– referencing order and display order are decoupled

• IBBPBBP.. vs. IBBPBBBBPBP... – referencing type and picture type are decoupled

• B slices can be used as reference


24

Transform Coding (1/2)

13


25

Transform Coding (2/2)

• Separable 4x4 transform– exact match inverse transform (integer

arithmetic)

– 16-bit arithmetic

– easy implementation (adds and shifts, no multiplications)

• Hadamard transform– transform coeff. that covers the entire

macroblock

– 4x4 for Intra 16x16 MB (DCs only)

– 2x2 for Chroma DCs

– adds only


27

In-Loop De-blocking Filter

• filter edges of 4x4 block• highly content adaptive

– slice level: alpha and beta control global filter strength

– edge level: MB type, motion, and #coded coeff. control filtering strength

– sample level: quantizer dependent threshold control filtering for each sample

• filtering should be done after entire macroblock has been decoded• vertical edges first, then horizontal edges• filter both luma and chroma• may cross slice boundary

14


28

Entropy Encoding


29

Variable Length Coding

• Exp-Golomb code– almost all symbols except transform coefficients

• Context Adaptive VLC (CA-VLC)– Context are built dependent of transform coefficients

– Transform coefficients are scanned backward

– TotalCoeff (replace end-of-block) and TrailingOnes

• 4 VLC tables for luma and 1 VLC table for chroma• selection based on transform coeff. of 2 neighboring blocks (block above

and on the left)– Coefficients levels

• sign of TralingOnes• bits spent to represent one level is function of previously encoded levels

(max range is variable)– Run of zeros preceding each coefficient

15


30

There is more ....

• More coding tools– Motion vector prediction using motion vectors from 4 neighboring blocks

– Adaptive Weighted prediction (generalized B slices)

• each prediction sample can be weighted• an offset can be added

– Interlaced coding

• field or frame coding• macroblock adaptive frame/field coding

– Context-adaptive Binary Arithmetic Codec (CABAC)

• More error resilience and network adaptation tools– Parameter set structure

– Network Adaptation Layer (NAL) syntax structure

– Arbitrary Slice Ordering (ASO)

– Data Partitioning (DP)

– Redundant Slices

– SP/SI synchronization/switching slices

• More sideband information– Supplemental Enhancement Information (pan-scan, cropping, etc.)

– Video Usability Information (aspect ratio of luma sample, overscan, etc.)


31

Profile and Levels (1/2)

B slices

WeightedPrediction

Interlace at frame and MB level

CABAC

I slices

P slices

CAVLC

SP andSI slices

DataPartitioning

FMOand ASO

RedundantSlicesMain profile

Extended profile

Baseline profileProfiles and Levels are de-coupled• all profiles supports all levels• levels supports from QCIF@64Kb/s to 2kx4k@240Mb/s

16


32

Profile and Levels (2/2)

• 4 High profiles (formally FRExt)– High (8-bit/sample video for 4:2:0)

– High 10 (10-bit/sample for 4:2:0)

– High 4:2:2 (10-bit/sample for 4:2:2)

– High 4:4:4 (12-bit/sample for 4:4:4)

– 3 new features:

• adaptive block-size transform• encode specified perceptual-based quantization scaling matrices• lossless coding ROI

• Few level and/or profile-dependent constraints– max number of mot. vectors per two consecutive MBs for some levels (all profiles)

– max number of slices for Main Profile (22 for SD-TV)

– limit on mot. vectors “spreading” for Simple Profile

– etc.


33

H.264/MPEG-4 AVC Coding Efficiency Performance

Fig. 7: (a) – (e) Comparison of R-D curves for MPEG-2 (MP2), MPEG-4 Part 2 ASP (MP4 ASP) and H.264/AVC (MP4 AVC). I frames were inserted every 15 frames (N=15) and two non-reference B frames per reference I or P frame were used (M=3)

[Sullivan, SPIE, Aug. 2004]

17


34

H.264/MPEG-4 AVC Market Acceptance (1/2)

• Disclaimers– not intended to be comprehensive

– public information, out-of-date at any point in time

– major sources: Gary Sullivan and Thomas Wiegand

• 3GPP– Baseline profile @ level 1b

– recommended codec for PSS, MMS, and PSC in Rel. 6 (3GPP)

– optional codec for streaming service (3GPP2)

• Mobile TV– Baseline profile @ levels 1b – 2.0

– DVB: mandatory codec – Trial in Berlin

– DAB: mandatory/single codec in T-DMB

– ARIB: mandatory codec for mobile segment broadcast over ISDB-T

– 3GPP: recommended codec for MBMS in rel.6


35

H.264/MPEG-4 AVC Market Acceptance (2/2)

• SDTV and HDTV– Main/High profiles @ levels 3.0 – 4.0

– DVB: additional codec to MPEG-2 in services over MPEG-2 TS, single codec in services over IP

– ISMA: mandatory codec in version 2

• DVD Forum: mandatory video decoding for HD-DVD• IPR Licensing: MPEG-LA and Via Licensing Pool• Support in players: Apple QuickTime (internet streaming)

18


36

H.264/MPEG-4 AVC Deployment

• Videoconferencing Systems: baseline profile @ levels 1b –4.0 by 8 companies (Polycom, Tandberg, Sony, France Telecom, etc.)

• Consumer Electronics: Sony (PSP)• ASIC

– Mobilygen (wireless video)

– Broadcom, Conexant, DG2L, Harmonic, Cradle Technologies, Modulus Video, Motorola, etc. (broadcast encoding and transmission)

• PC and DSP S/W Solutions: Intel, Envivio, Equator, FastVDO, Hantro, etc.

• Test Equipment: Tektronix and Vqual

AVC/H.264 gaining, but MPEG-2 still strong!

Chapter 2 –H.264 Annex G/MPEG-4 Scalable Video Coding

19


38

H.264 Annex G/MPEG-4 Scalable Video Coding (SVC) – Outline

• Why do we need scalable video• MPEG-4 SVC requirements and targeted apps.• Historical overview and standardization timeline• Tools enabling scalability• Media Aware Network Element (MANE)• Profiles and levels


39

Traditional Video Coding

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

20


40

Increasing Demands


41

Simulcast: Encoding

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

21


42

Simulcast: Decoding

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001


43

Dramatic Increase of Heterogeneous Devices

22


44

Solution: from Simulcast to Scalable Video Coding

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001

00101101101001011010110101101100010110110010110111101010001011011000111011110011100110111001


45

MPEG-4 SVC: An Example Application

• Digital Video Surveillance– live monitoring in a multi-point surveillance system low delay, pyramid

scalability, temporal random access

– storage multipath scalability, temporal random access

– interactive surveillance random spatial access

– mosaic display, image analysis, robust video streaming, etc.

– Bosch, GE, Panasonic

Scalable video coding in digital video security [GE (formally VisioWave), 2005]

23


46

Requirement: (Extended) Spatial scalability

001011011010010110101101011011000101101100101101111010100010110110001110111100

11100110111001


47

Extended Spatial Scalability

non-dyadic scaling

cropping

combination of cropping and non-dyadic

scaling

24


48

Extended Spatial Scalability: Usage Scenario

16:9 HDTV

http://products.sel.sony.com/hdtv/

4:3 SDTV

croppingand

scaling


49

Requirement: Temporal Scalability

001011011010010110101101011011000101101100101101111010100010110110001110111100

11100110111001

25


50

Requirement: Quality Scalability

001011011010010110101101011011000101101100101101111010100010110110001110111100

11100110111001


52

Requirement: Region of Interest scalability

• useful when bandwidth limitations

• real-time zooming and panning at receiver side

26


53

Additional Requirements

• Complexity scalability– Including a low complexity codec

• Coding efficiency performance– Coding penalty of max. 10% for same perceived quality

• Low end-to-end delay / fast random access• Robustness to transmission errors• Uniform way to manipulate and adapt scalable

streams, to be mapped to widely used protocols


54

MPEG-4 SVC Targeted Applications (1/3)

• Wireless LAN video in home network– time-varying bandwidth fine grain SNR plus limited spatial scalability for managing

wide bandwidth variations

– device adaptation combined spatial/temporal/SNR, complexity scalability, arbitrary spatial resolution ratio’s, hybrid combinations of progressive and interlaced

– Thomson, France Telecom, Philips, Siemens

• Broadband video distribution – ADSL clients, from DVB to DSL

– client: display/processing adaptive video stream from low-quality (400Kbps@CIF) to HD-TV combined spatial/temporal/SNR/complexity with medium grain

– operator/service: decoupling encoding from streaming, saving storage and bandwidth, content leverage simultaneous management of HD/SD (combined scalability)

– France Telecom, Thomson, Philips

• Scalable video storage with erosion functionality– importance of the recorded video data, and thus its quality, decreases over time

– Bosch, GE, Panasonic

27


55


• MPEG-21 DIA based adaptation– technology to adapt scalable bit-streams to different devices and network

standardized system interface (e.g. bit-stream syntax)

– Siemens AG, Deutsche Telekom, France Telecom

Media Server

Business Users

Home Users

Business Users

Meta-Database

Proxy Server

Router

WAN

LAN

LAN

LAN

LAN

Last Mile

Applications and Requirements for SVC [N6880, Jan 2005]


56


• Video telephony/conferencing– Nokia, NTT, Siemens

• Mobile streaming/broadcast video– Thomson, Nokia, France Telecom, Orange, SFR, Samsung, Siemens

• Professional video manipulation– Thomson

28


57

History

66, BrisbaneOct ‘03

71, Hong KongJan ‘05

76, MontreuxApr ‘06

CfP[N5958]

WD

79, MarakechJan ‘07

FDIS

WaveletExploration Group

N8043

MPEG-21 Part 13

MPEG-4 Part 10 Amd 3


58

History




CfP

WD


FDIS


N8043

MPEG-21 Part 13


Call for Proposals

[N5958]

29


59

History




CfP[N5958]

WD


FDIS


N8043

MPEG-21 Part 13

MPEG-4 Part 10 Amd 3Exploration phase

Temporal: MCTF/Hierarch. B-framesSpatial: wavelets/DCT-basedQuality: bitplane/layered coding


60

History




CfP[N5958]

WD


FDIS


N8043

MPEG-21 Part 13


Working draft:Multiscale pyramid 2D+t

subjective and objective comparison

Temporal: MCTFSpatial: Layered approachQuality: Layered approach

30


61

History




CfP[N5958]

WD


FDIS


N8043

MPEG-21 Part 13


Vidwav AdHoc Group

wavelets promising for high resolution, extended scal. and fine grain scal.

Three working modes:T+2D: drift problems at full spatial resolutions2D+T: compensate for shift-invariant nature of

wavelets2D+T+2D: intermediate solution


62

History




CfP[N5958]

WD


FDIS


N8043

MPEG-21 Part 13


Status report on Wavelet Video Coding

subjective test indicate (compared to JSVM):• significantly worse for combined scalability• comparable/slightly worse for standalone SNR scal.

“Exploration discontinued until further proof of usefulness”

31


63

History




CfP[N5958]

WD


FDIS


N8043

MPEG-21 Part 13


M11756: Proposed requirement modification

“The base layer of the standard shall be compliant with MPEG-4 AVC. The standard shall be capable of

decoding MPEG-4 AVC bit streams”

MPEG-4 Part 10 Amd3

SVC continued in JVT


64

History




CfP[N5958]

WD


FDIS


N8043

MPEG-21 Part 13


Further exploration

Core ExperimentsAdHoc Groups

signaling layer dependencieshigh-level syntaxFGS refinement

additional inter-layer modes…

32


65

History




CfP[N5958]

WD


FDIS


N8043

MPEG-21 Part 13


FDIS

• Temporal scalability:Hierarchical B-frames

• Spatial scalability: Layered approachESS

• Quality scalability: Layered approach for CGSMGSBitplane coding for FGS


66

SVC Encoder Block Diagram

spatialdecimation (2)

MC & Intraprediction


base layercoding

base layercoding

mux

inter-layer prediction techniques

0010110110100101101011010110110001011011001011011110101

texture

motion

texture

motion


base layercoding


texture

motion

progressive SNR refinement

texture coding


texture coding

H.264/AVC compatible bitstream


texture coding

spatialdecimation (2)

generates M (max=3) FGS layers

generates N CGS layers

33


67

Temporal Scalability in MPEG-4 SVC

• Hierarchical B-frames

• Concept already existed in H.264/AVC


entropycoding

0010110110100101101011010110110001011011001011011110101

texture

motion


68

Hierarchical B-frames

key picture(IDR)

key picture

Group of pictures (GOP)

T2

T1

T0

T3

temporal layers

34


69

Hierarchical B-frames: performance• high spatial detail• smooth motion strong local motion

• H.264/MPEG-4 AVC: classical prediction, i.e. 1st picture IDR, following ones Ps and Bs (IPP…, IBPBP…, IBBPBBP…) vs. hierarchical B-frames (GOP = 4, 8, 16, 32)

• NOTE: encoder optimizations to improve coding eff. of hierarchical prediction

[H. Schwarz, D. Marpe, and T. Wiegand: Analysis of Hierarchical B Pictures and MCTF, Proc. IEEE International Conference on Multimedia & Expo (ICME 2006), Toronto, Canada, July 9-12, 2006]

hierarchical B-frames

Hierarchical B-frames coding efficiency gain


70

Spatial Scalability in Simulcast

spatialdecimation



entropycoding

entropycoding

0010110110100101101011010110110001011011001011011110101

texture

motion

texture

motion

35


71

Spatial Scalability in MPEG-4 SVC


spatialdecimation



entropycoding

entropycoding

0010110110100101101011010110110001011011001011011110101

texture

motion

texture

motion


72

Inter-layer Prediction Techniques

• Layered approach

• Three techniques:– Inter-layer Intra Texture Prediction

– Inter-layer Motion Prediction

– Inter-layer Residual Prediction

• Some concepts already existed in MPEG-2/4 for spatial scalability

36


73

Inter-layer Intra Texture Prediction

• provided in the Intra_Base macroblock mode• constrained, allowing for single loop decoding at

target layer (no multiple motion compensation processing)

layer k+1

layer k

I B2 B2B1 P

I B2 B2B1 P


74

Inter-layer Motion Prediction

• Base Layer Mode– macroblock partitioning, motion vectors and reference indices of

base layer are used in enhancement layer

– motion vectors are multiplied by the resolution ratio between the enhancement layer and its base layer

• Quarter Pel Refinement Mode– same as base layer mode

– for each motion vector a quarter-sample motion vector refinement is additionally transmitted and added to the derived motion vectors

37


75

Inter-layer Residual Prediction

• only code the difference between current layer residual information and previous layer residual information

• adaptive, since only beneficial when motion vectors are (nearly) identical between both layers

• residual information is upsampled according to the spatial resolution ratio between the layers


76

Inter-layer Prediction Techniques: Performance

[Schwarz_IWSSIP_2005]

This difference is the cost of spatial

scalability!

38


77

Inter-layer Prediction Techniques: Performance

[Schwarz_IWSSIP_2005]

This difference is the cost of spatial

scalability!


79

Quality Scalability in MPEG-4 SVC

• Coarse Grain Scalability

• Fine Grain Scalability– aka Progressive Refinement

• Medium Grain Scalability

3 types of quality scalability technologies:

39


80

Coarse Grain Scalability (1/3)



entropycoding

entropycoding

multiplexer

0010110110100101101011010110110001011011001011011110101

texture

motion

texture

motion

QP2

QP1

QP2 < QP1

QP = Quantization Parameter

R2

R1


81




entropycoding

entropycoding

multiplexerinter-layer prediction techniques

0010110110100101101011010110110001011011001011011110101

texture

motion

texture

motion

QP2

QP1

QP2 < QP1

QP = Quantization Parameter

R1

R2-R1

40


82


• Same concepts as with spatial scalability, but without the upsampling operations

• Coding efficiency is optimized for coarse rate graduations (factor 1.5-2 from one layer to the next)


83

Fine Grain Scalability (1/2)


base layercoding

multiplexer

0010110110100101101011010110110001011011001011011110101

texture

motion

progressive SNR refinement texture coding

41


84

Fine Grain Scalability (2/2)

• Bitplane coding• Ordering of data from most to least significant

• enables truncation of NAL units (bit packets) at any arbitrary (byte-aligned) point

• Useful when in need to reduce bit rate by a few per cent to adapt to current network conditions


85

Medium Grain Scalability (1/2)

• CGS is computationally simple, but only allows a discrete set of extraction points

• FGS allows a fine granular set of extraction points, but is computationally complex

• MGS increases the number of CGS extraction points by reusing the quality level and layer syntax used in FGS

42


86

Medium Grain Scalability (2/2)

SOCCER_CIF15

30

31

32

33

34

35

36

37

38

0 100 200 300 400 500 600 700Rate (kbs/s)

PSN

R (d

B)

CGS

CGS_QL_DEP

CGS_QL_DEFAULT

Adaptive Motion FGS + Quality Layers

[Amonou_JVT-T054]

extraction point

in FGS any point is an

extraction point


87

Quality scalability: performance

• Single/base layer: H.264/MPEG-4 AVC, 1st picture IDR, GOP = 16• QP lowest PSNR layers – QP highest PSNR layers = 12 (factor 4 in bit-rate)• DQP = QP PSNR layer_i – QP PSNR layer_(i+1)• Motion = motion parameters of the enhacement layer are adaptively refined

[Sch

war

z_IW

SSIP

_2005]

This difference is the cost of quality

scalability!

43


88

Bringing it Together: Combined Scalability

no strict notion of layer

CIF@30Hz

QCIF@30HzCIF@15Hz

[Schwarz, ICME’05]

3D Scalability Space


89

Embedded Scalability: Coding Efficiency Cost

[Schwarz, ICME’05]

The difference between the two

points is the cost of embedded scalability!

Note: the original target (from MPEG Req.) was 10% cod. eff. loss in exchange of

embedded scalability

44


90

Media Aware Network Element (MANE)

• A network element, such as a router or application layer gateway, that is capable of parsing certain aspects of the RTP payload headers or the RTP payload and reacting to the contents

• MANEs are aware of the bitstream signalingMANEs allow packets to be dropped according to the needs of the media coding.

Example: if a MANE has to drop packets due to congestion on a certain link, it can identify those packets whose dropping has the smallest negative impact on the user experience and remove them in order to remove the congestion and/or keep the delay low.


91

Bitstream Adaptation (scenario 1)

45


92



93


46


94



95

Profiles and Levels: not there yet!

• Profiles– Restrict which tools to be used

• Levels– Restrict on implementation issues (memory usage, resolution, …)

• Profile and Level discussion started in previous meeting (October 2006)– Expected results at next meeting

• New issues introduced by scalability– traditional conformance point (profile@level) not sufficient, need to explain

how scalability layers interact

– base and enhancement layers may belong to different profiles, how to express that to limit decoder complexity?

47

Chapter 3 - MPEG-4 Multi-view Video Coding


97

MPEG-4 Multi-view Video Coding - Outline

• History, context, and motivations• Typical multi-view system setups and application

scenarios• Standardization and requirements on MVC• Basic concepts and principles of MVC• Coding tools currently considered

48


98

MPEG-4 Multi-view Video Coding - History, Context, and Motivations

2D color TV3D color TV

2D 3D

Interactive multi-view video

Passive single-view video

Passive Interactive

1-view N-view

Courtesy of Philips and The Matrix


99

3D Video/Stereo Video [JVT-T100]

• Generating 3D depth impression from separate views for each eyes

• Not necessarily limited to one user: multiple users can watch it simultaneously from a wide range of positions, with multiple inputs/outputs

Courtesy of HHI

49


100

Free Viewpoint Video (FVV) [JVT-T100]

• Aiming at free navigation and interactivity• Some challenges to tackle:

– Multiple video streams need to be synchronized

– Huge amount of data needs to be compressed and transmitted

– Inter-viewpoint redundancy to be exploited (effective only for dense camera setups, not for dome-type arrangements)

Courtesy of Microsoft Research and HHI


101

Typical Multi-view Camera Configuration

Parallel view Convergent view Divergent view

Stanford Multi-Camera Array with 128 synchronized CMOS cameras[Levoy et al,SIGGRAPH’96]

8 high resolution PtGrey cameras along a 1D arc[Zitnick et al,SIGGRAPH’04] Dodeca 1000 camera –

11 videocameras in a dodecahedral array

50


102

Interactive Multi-view Realization: Free View-point Video (FVV)

Courtesy of HHI and Microsoft Research

• A classical trade-off between costs (e.g. camera#, processors) and benefits (e.g. navigation range, virtual views quality)

• Depth (or disparity) map to be computed using computer vision techniques


103

High-quality Video Interpolation using a Layered Representation [Zitnick, SIGGRAPH’04]

51


104

High-quality Video Interpolation using a Layered Representation: Set-up and Results

balletB-half.avi day1-lazy0-half.avi


105

Application Scenarios [N5877]

Entertainmente.g. concert, sport,

movie, game...

Educatione.g. instruction video, cultural archieves

Medical surgeryViewing with

exploration e.g.

museum, shopping

Surveillance

Immersive video conferenceAdvertisements

Event broadcastingMVC

FVV system

52


106

Video Resource Management Info

MVC Video Elementary Stream Info

Timing Info

Camera Parameters Info

MVC Decoder

View Generation

Shared Memory

Video Resource Management Info

MVC Video Elementary Stream Info

Timing Info

Camera Parameters Info

MVC Decoder

View Generation

Shared Memory

• Basic components of an example FVV system

• Example architecture of a FVV decoder

Video Capture Correction MVC

EncoderMVC

DecoderView

Generation Display

FVV System and FVV Decoder [N8064]


107

Standardization Progress [JVT-T100]

12/20

01

10/20

03

07/20

05

07/20

06

01/20

06

01/20

07

07/20

07

01/20

08

07/20

03

53


108

Requirements for MVC (1/2) [N8064]

√Robustness (or error resilience)

√Picture quality among views

√Resolution, bit depth, chroma sampling format

√Low delay

√Resource consumption

√Spatial/temporal/SNR scalability

√Free viewpoint scalability

√View scalability

√Compression efficiency

ShouldShallFeatures

Compression related requirements


109

Requirements for MVC (2/2)

√Non-planar imaging and display systems

√Synchronization

√Camera parameters

√View generation

System support related requirements

√Parallel processing

√Resource management for decoders

√Spatial random access

√View random access

√Temporal random access

ShouldShallFeatures

54


110

Synchronized Multi-view Video Streams


111

Inter-viewpoint Correlation and Prediction [Merkle, ICME’06], [m12945]

• T (temporal)• S (inter-view)• L/R (combined)

• Normally, T better than S better than L/R

• Influencing factors:– Frame rate

– Inter-camera distance (baseline)

– Content complexity (motion intensity, illumination effects, etc)

55


114

Promising Coding Tools Currently Considered for MVC Standard [m13394]

• Hierarchical B pictures for temporal dependencies and an adapted prediction scheme (HHI proposal)

• MVC encoder design recommendations• Block level illumination compensation (5

competing proposals)• View synthesis prediction (2 different

proposals)


115




competing proposals)• View synthesis prediction (2 different

proposals)

56


116

Hierarchical B Pictures for MVC: Overview [Merkle, ICME’06], [m12945], [W8019], [JVT-P014]

• Uses hierarchical B-pictures combined in temporal and interview dimension, proposed by FhG-HHI

• Proved best performance out of 8 proposals in response to a CfP in formal subjective tests

• Current MVC reference model• Fully compatible to H.264/MPEG4-AVC

• Reorganization of input images into a single stream prior to encoding

• Decoder needs to de-interleave the decoded pictures into MV video streams


117

Hierarchical B Pictures for MVC: Coding Structure (GOP size 8)

[Sm

olić

, JV

T-T

100,

July

2006]

temporal prediction

inte

r-vie

w p

red

icti

on

combined prediction

57


118

Hierarchical B Pictures for MVC: Coding Results

for “Ballroom” for “Exit”

• Inter-view prediction: hierarchical B pictures for temporal and inter-view dimensions

• Simulcast: hierarchical B pictures for temporal dimension only

• Anchor: H.264/AVC simulcast coding without hierarchical B pictures


119

Hierarchical B Pictures for MVC: Summary

• Hierarchical B pictures (accounts for about 50% coding gain)• Sophisticated reference frame selection (coding structure)• Decoding Picture Buffer Size increases to 2*GOP_length +

views#• Coding efficiency vs low-delay decoding: the coding

efficiency drops significantly as GOP_length decreases.• In general, significant

coding gain over simulcast anchor andsimulcast hierarchicalB picture coding, when GOP_lengthis set to 12 and 15.

58


120




competing proposals)• View synthesis prediction


121

MVC encoder design recommendations

• Low delay to enable fast temporal random access– Ex: prediction structure

corresponding to GOP with 1 frame (“Short”)

• Simplified prediction structures to reduce the number of reference candidates– Ex: inter-view prediction is

only applied to anchor frames

• Alternative prediction structures (e.g. placing reference I in middle) to improve coding efficiency

B

P

I

B

P

B

P

P

I

B

P

B

P

B

P

P

59


122






123

Block Level Illumination Compensation: Illumination/Color Mismatch Problem [JVT-T100: CE2]

• Imperfectly calibrated cameras• Different perspective projection direction• Different reflection effects

Courtesy of HHI

60


124

Block Level Illumination Compensation [JVT-T100: CE2]: Current status

• Weighted prediction (WP) existing in H.264/AVC for slice level illumination compensation

• 5 competing proposals aiming at compensate the illumination differences using various local techniques (see details in [JVT-T110], [JVT-T113], [JVT-T114], [JVT-T115], [JVT-T117]) – Some proposals use both scale and offset; others use offset only

– Three proposals employ predictive coding of DC residue; one proposal combines slice-level WP with block-level delta-WP; the other proposal uses 2D (W,O) vector quantization

– Some proposals support only 16x16; others include smaller block sizes for IC

– All proposals introduce new syntaxes

– Encoder aspects: all proposal involve illumination invariant matching cost calculation in motion/disparity search; in general IC requires extra cost in mode decision

• Comparison against reference w/o WP 0.1 to 0.6 dB PSNR gain• As of July ’06, no convergence has been reached


126





61


127

View Synthesis Prediction

• Require camera parameters and multi-view geometry

• Exploit camera parameters and multi-view geometry (i.e. corresponding points across different views lie on the epipolar line) to generate intermediate views which may be a more accurate reference candidate

• Ex in picture: MERL [JVT-T123] approach

Temporal Prediction

View Prediction

time

view

View Interpolation

View Warping

captured frames

virtual frames

Conclusions

62


129

Conclusions

• MPEG and ITU-T Standardization Bodies• MPEG-* and H.26*• The H.264/MPEG-4 video standards

• H.264/MPEG-4 Part 10 AVC

• SVC extension of H.264/MPEG-4 AVC (FDIS Jan ’07)

• MVC extension of H.264/MPEG-4 AVC (FDIS Jan ’08)

Annex

63


131

Annex - Outline

• What else is going on as far as video coding standards (MPEG, ITU-T, SMPTE)

• Bibliography and reference points• Acronyms and abbreviations


132

Other MPEG Video Activities (1/2)

• ISO/IEC 23002-1 and 23002-2 MPEG Visual Technologies– Accuracy specification for implementation of integer-output IDCT (Part 1) to replace IEEE 1180

(same text), issue CORs

– CfP for fixed-point implementation (i.e. no drifting) of DCT/IDCT passing IEEE 1180 and causing no problem with existing codec and streams

– 9 proposals on October 2005

– interesting dynamic between MPEG-2 “old timers” and the proponent of this activity (ex: how much precision is needed?, do we really need this work?)

– SW test-bed and test plan to evaluate proposals

– WD 2.0 of ISO/IEC 23002-2 Integer DCT Transforms

• ISO/IEC 23002-3 MPEG Visual Technologies: Auxiliary Video Data Representation

– stereoscopic applications (ex: stereo video display)

– encode depth/disparity map re-using existing SoA (low hanging fruit)

– ISO/IEC 23002-3 timeline:

• WD on Jan 2006• CD on April 2006• FDIS on January 2007

– current status: FCD

64


133

Other MPEG Video Activities (2/2)

• Reconfigurable Video Coding (RVC)– formally known as VCTR

– toolbox of standard video coding tools with standard interfaces (FU)

– mechanism to describe decoders in machine readable format

– a specific decoding solution is implemented as a set of FUs connected to a global control unit

– benefits:

• reconfigurable decoders comply with multiple standards at a minimal implementation cost

• componentization of codec design and reusability• higher level of flexibility in interoperability• fast standardization of new coding tools new standard codecs

– status: requirements definition and draft of CfP

– time line:

• CfP April 2006, • proposals by July 2006• FDIS July 2007

– current status: WD 1.0


134

Other JVT Activities

• Advanced 4:4:4 Profiles– replace High 4:4:4 (FRExt H444P)

– 14496-10:2006 Amd. 2 Advanced 4:4:4 Profiles

– Two 4:4:4 profiles (one for Intra only and one for Intra+Inter): tentativenames Hi444Intra (only containing I slices) and Hi444Predictive (further input requested at the next meeting)

– Current status : FPDAM2

• SVC File Format (FF)

65


135

Video Standards on the Horizon

• MPEG Video: exploration activity for technologies to increased video compression efficiency (workshops)

• ITU-T VCEG: H.265 on hold to send a clear message

• Society of Motion Picture and Television Engineers (SMPTE)

– Tech. Committee on Television Video Compression (C24), AhG on Video Codec (VC)-9 Compressed Video Bitstream Format and Decoding Process

– MS push to standardize WMV-9 (failed in MPEG) in preparation for HD-DVD battle

– Simple and Main profiles: “rubber stamp”, as forced to be WMV-9 backward compatible

– Advanced profile: more open standardization approach, as innovation is required to support interlaced content

– Current status: completed

• VC-1 specification – SMPTE 421M• VC-1 conformance – SMPTE RP228• VC-1 transport – SMPTE RP227


136

Some Interesting Papers Relevant to MVC

• 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes [Matusik, SIGGRAPH’04]

• Disparity compensated prediction [Zitnick, SIGGRAPH’04]• Video transcoding for special visual effects in MVC, e.g. frozen moment,

view switching [Lou, MM’05]• 3D voxel model based MVC [Gao, ICIP’05]• MVC based on global motion model [Guo, PCM’04]• A framework for MVC using layed depth images [Yoon, PCM’05]

• Ensuring color consistency across multiple cameras [Ilie, ICCV’05]

• Dependent bit allocation for MVC [Kim, ICIP’05]

• Predictive fast motion/disparity search for MVC [Lai, VCIP’06]• Epipolar geometry assisted motion estimation for MVC [Lu, ICIP’06]

• Multicast of real-time MVC [Zuo, ICME’06]• Interactive transport of MVC for 3DTV [Kurutepe, PV’06]

66


137

MPEG and JVT Documents (1/2)

• [JVT-T100] ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, “Overview presentation of multi-view video coding,” Doc. JVT-T100, July 2006.

• [N8064] ISO/IEC JTC1/SC29/WG11, “Requirements for multi-view video coding v.6,” Doc. N8064, April 2006.

• [N5877] A. Smolić and H. Kimata, “Applications and requirements for 3DAV,”ISO/IEC JTC1/SC29/WG11, Doc. N5877, July 2003.

• [m12945] P. Merkle, K. Műller, A. Smolić and T. Wiegand, “Multiview coding using AVC,” ISO/IEC JTC1/SC29/WG11, Doc. m12945, Jan. 2006.

• [JVT-T138] X. Xu, et al., “MVC camera position parameters coding,” Doc. JVT-T138, July 2006.

• [W8019] ISO/IEC JTC1/SC29/WG11, “Description of Core Experiments in MVC,” Doc. W8019, April 2006.

• [m13394] A. Smolić and H. Kimata, “Report of AHG on MVC,”” ISO/IEC JTC1/SC29/WG11, Doc. m13394, July 2006.

• [JVT-T110] Y.L. Lee, et al., “Results of CE2 on multi-view video coding,”Doc. JVT-T110, July 2006.

• [JVT-T123] S. Yea, et al., “Report on core experiment CE3 of multi-view video coding,” Doc. JVT-T123, July 2006.


138

MPEG and JVT Documents (2/2)

• [JVT-T119] M. Kitahara, et al., “Report of core experiment on view interpolation (multi-view video coding CE3),” Doc. JVT-T119, July 2006.

• [JVT-T130] P. Pandit, et al., “MVC high-level syntax for random access,” Doc. JVT-T130, July 2006.

• [JVT-T134] H. Kimata, et al., “Proposal on a coding scheme for free viewpoint scalability (MVC),” Doc. JVT-T134, July 2006.

• [JVT-T136] Y. S. Ho, et al., “Global disparity compensation for multi-view video coding,” Doc. JVT-T136, July 2006.

• [JVT-T137] Y. S. Ho, et al., “Complete coding result of layered depth image frames,” Doc. JVT-T137, July 2006.

• [JVT-P014] H. Schwarz, D. Marpe and T. Wiegand, “Hierarchical B pictures,”Doc. JVT-P014, July 2005.

• [N5958] ISO/IEC JTC1, “Call for Proposals on Scalable Video Coding Technology”, ISO/IEC JTC1/WG11 Doc. N5958, Oct. 2003

67


139

References (1/4)

• [Leyoy, SIGGRAPH’96] M. Leyoy and P. Hanrahan, “Light field rendering,”Proc. of ACM SIGGRAPH, 1996, pp. 31-42.

• [Smolić, CSVT’04] A. Smolić and D. McCutchen, “3DAV exploration of video-based rendering technology in MPEG,” IEEE Trans. on Circuits and System for Video Technology, Mar.2004, vol.14, no.3, pp. 348-356.

• [Smolić, IEEEProc’05] A. Smolić and P. Kauff, "Interactive 3-D video representation and coding technologies," Proceedings of the IEEE, Special Issue on Advances in Video Coding and Delivery, Jan.2005, vol. 93, no.1, pp. 98-110.

• [Matusik, SIGGRAPH’04] W. Matusik and H. Pfister, “3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes,” Proc. of ACM SIGGRAPH, 2004, vol.23, pp. 814-824.

• [Zitnick, SIGGRAPH’04] C. L. Zitnick, et al., “High-quality video interpolation using a layered representation,” Proc. of ACM SIGGRAPH, 2004, vol.23, pp. 600-608.

• [Lou, MM’05] J.G. Lou, H. Cai, and J. Li, “A real-time interactive multi-view video system,” Proc. of ACM on Multimedia (ACMMM), 2005, pp. 161-170.

• [Gao, ICIP’05] Y. Gao and H. Radha, “Multi-view image coding using 3-D voxel models,” IEEE Intl. Conf. on Image Processing (ICIP), 2005, vol.2, pp. 257-260.

• [Guo, PCM’04] X. Guo and Q. Huang, “Multiview video coding based on global motion model,” LNCS 3333, Pacific-Rim Conf. on Multimedia (PCM), 2004, pp. 665-672.


140

References (2/4)

• [Yoon, PCM’05] S.U. Yoon, E.K. Lee, S. Y. Kim and Y.S. Ho, “A framework for multi-view video coding using layered depth images,” LNCS 3767, Pacific-Rim Conf. on Multimedia (PCM), 2005, pp. 431-442.

• [Ilie, ICCV’05] A. Ilie and G. Welch, “Ensuring color consistency across multiple cameras,” IEEE Conf. on Computer Vision (ICCV), 2005, vol.2, pp. 1268–1275.

• [Kim, ICIP’05] J.H. Kim, J. Garcia and A. Ortega, “Dependent bit allocation in multiview video coding,” IEEE Intl. Conf. on Image Processing (ICIP), 2005, vol.2, pp. 293-296.

• [Lai, VCIP’06] P.L. Lai and A. Ortega, “Predictive fast motion/disparity search for multiview video coding,” Proc. of SPIE Visual Communications and Image Processing (VCIP), 2006, vol.6077.

• [Lu, ICIP’06] J. Lu, H. Cai, J.G. Lou and J. Li, “An effective epipolargeometry assisted motion estimation technique for multi-view image and video coding,” to appear in IEEE Intl. Conf. on Image Processing (ICIP), 2006.

• [Zuo, ICME’06] L. Zuo, J.G. Lou, H. Cai and J. Li, “Multicast of real-time multi-view video,” IEEE Intl. Conf. on Multimedia and Expo (ICME), 2006, pp. 1225-1228.

• [Kurutepe, PV’06] E. Kurutepe, M. R. Civanlar, and A. M. Tekalp, “Interactive transport of multi-view videos for 3DTV applications,” Packet Video Workshop, 2006.

68


141

References (3/4)

• [Merkle, ICME’06] P. Merkle, K. Műller, A. Smolić and T. Wiegand, “Efficient compression of multi-view video exploiting inter-view dependencies based on H.264/MPEG4-AVC,” IEEE Intl. Conf. on Multimedia and Expo (ICME), 2006, pp. 1717-1720.

• [Schreer, ICME’06] O. Schreer, C. Fehn, N. Atzpadin, M. Müller, R. Tanger, and P. Kauff, “A flexible 3D TV system for different multi-baseline geometries,” IEEE Intl. Conf. on Multimedia and Expo (ICME), 2006, pp. 1877-1880.

• [Ozbek, ICME’06] N. Ozbek and A. M. Tekalp, “Scalable multi-view video coding for interactive 3DTV,” IEEE Intl. Conf. on Multimedia and Expo (ICME), 2006, pp. 213-216.

• [PCM’06] J. Lu, G. Lafruit and F. Catthoor, “Streaming-mode MB-based integral image techniques for fast multi-view video illumination compensation,” LNCS 4261, Pacific-Rim Conf. on Multimedia (PCM), 2006, pp. 414-423.

• Online resources:– MPEG site: http://mpeg.nist.gov/welcome.php

– JVT site: http://ftp3.itu.ch/av-arch/jvt-site/

– MPEG-4 working documents: http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-4

– HHI SVC page: http://ip.hhi.de/imagecom_G1/savce/


142

References (4/4)

• [Schwarz, IWSSIP’05] H. Schwarz, D. Marpe, and T. Wiegand, “Basic Concepts for Supporting Spatial and SNR Scalability in the Scalable H.264/AVC Extension”, IWSSIP 2005

• [Schwarz, ICIP’05] Heiko Schwarz, Tobias Hinz, Detlev Marpe, and Thomas Wiegand, “Constrained Inter-Layer Prediction for Single-Loop Decoding in Spatial Scalability”, IEEE International Conference on Image Processing (ICIP'05), Genova, Italy, September 2005

• [Schwarz, ICIP’05, 2] Heiko Schwarz, Detlev Marpe, Thomas Schierl, and Thomas Wiegand, “Combined Scalability Support for the Scalable Extension of H.264/AVC”, IEEE International Conference on Image Processing (ICIP'05), Genova, Italy, September 2005

• [Sullivan_SPIE_2004] Gary J. Sullivan, Pankaj Topiwala, and Ajay Luthra, “The H.264/AVC Advanced Video Coding Standard: Overview and Introduction to the Fidelity Range Extensions”, SPIE Conference on Applications of Digital Image Processing XXVII, August 2004

69


143

Acronyms and Abbreviations (1/3) – SVC: Scalable Video Coding (Amd3 to MPEG-4 Part 10)

– AVC: Advanced Video Coding (MPEG-4 Part 10)

– MPEG: Moving Picture Expert Group

– ITU – T: International Telecommunication Union - Telecommunication (formally CCITT)

– VCEG: Video Coding Expert Group

– JVT: Joint Video Team

– SMPTE: Society of Motion Picture and Television Engineers

– AhG: Ad-hoc-Group

– CfP: Call for Proposal

– SoA: State-of-the-Art

– CA-VLC: Context Adaptive Variable Length Coding

– FRExt: Fidelity Range Extension

– COR: Corrigendum

– Amd: Amendment

– WD: Working Draft

– CD: Committee draft

– FDIS: Final Draft International Standard

– IS: International Standard

– PDAM: Preliminary Draft Amendment

– FPDAM: Final Preliminary Draft Amendment

– FDAM: Final Draft Amendment

– CE: Core Experiment

– DAB: Digital Audio Broadcasting

– T/S-DMB: Terrestrial/Satellite Digital Multimedia Broadcasting

– ISMA: Internet Streaming Media Alliance


144

Acronyms and Abbreviations (2/3)

– MPEG-21 DIA: MPEG-21 Digital Item Adaptation

– DVB: Digital Video Broadcasting

– DVB - T: Digital Video Broadcasting Terrestrial (terrestrial television)

– DVB – H: Digital Video Broadcasting Handheld (terrestrial television for handhelds)

– 3GPP: 3rd Generation Partnership Project

– 3GPP2: 3rd Generation Partnership Project 2

– ARIB: Association of Radio Industries and Businesses (standardization organization in Japan)

– ISDB-T: Integrated Services Digital Broadcasting-Terrestrial

– IPR: Intellectual Property Right

– RVC: Reconfigurable Video Coding (formally VCTR)

– VCTR: Video Coding Tools Repository

– MVC: Multi-View Coding

– FF: File Format

– WMV-9: Windows Media Version 9

– AAP: Alternative Approval Process (ITU)

– ISO: International Standards Organization

– IEC: International Electrotechnical Commission

– DSL/ADSL: Digital Subscriber Line/Asynchronous Digital Subscriber Line

– DCT/IDCT: Discrete Cosine Transform/ Inverse Discrete Cosine Transform

– Vidwav: Wavelet Video Coding

– PSP: PlayStation Portable (Sony)

– MMS: Multimedia Messaging (3GPP)

– PSS: Packet Switched Stream (3GPP)

– PSC: Primary Synchronization Code (3GPP)

70


145

Acronyms and Abbreviations (3/3)

– MVC: Multi-view Video Coding (in the Annex of the full-set slides already)

– FVV: Free Viewpoint Video

– 3DAV: 3D Audio and Video

– GOP: Group of Pictures

– DPB: Decoded Picture Buffer

– RPL: Reference Picture Lists

– WP: Weighted Prediction

– VSP: View Synthesis Prediction

– DCVP: Disparity Compensated View Prediction

– VPS: View Parameter Set