Scalable Video Coding Systems-Basics, Open-Source Approaches and Advances

PARITANTRA - Journal of Systems Science and Engineering, Vol. 21&22, No. 1, 2013, pp. 19-28

Scalable Video Coding Systems: Basics, Open-SourceApproaches and Advances

A. A. BHURANE1, P. R. CHAPLOT2 AND V. M. GADRE3

1Research Scholar, Department of Electrical Engineering,Indian Institute of Technology, Bombay, India.

2Student of Dual-degree (B.Tech.+ M.Tech.), Department of Electrical Engineering,Indian Institute of Technology, Bombay, India.

3 Professor, Department of Electrical Engineering, IndianInstitute of Technology, Bombay, India.

ABSTRACT

Scalability is an important systems concept. Scalable video coding systems are gainingimportance in communication. In this paper we modify an open-source, wavelet-basedvideo codec, namely Dirac which is developed by the British Broadcasting Corporation(BBC) Research group. The paper describes our efforts to integrate spatial scalability inDirac, which was not initially intended to be a scalable video codec. The comparison ofthis scalable extension of Dirac video codec across a single layer and a traditional simulcastis carried out. The remainder of the paper gives a glimpse of the current efforts in thestandardization of a scalable video codec.

I. INTRODUCTION

Increasing demands of video ranging from internetstreaming, IPTV, video on demand (VOD), and so onhave accelerated the research in video compression.This research continues to aim at delivering betterquality of video to the end users. For this,standardization groups ITU-T Video Coding ExpertsGroup (ITU-T Q.6/SG 16) [1] and ISO/IEC MovingPicture Experts Group (ISO/IEC JTC 1/SC 29/WG11) [2] are constantly working towards developingthe state-of-the-art technologies to satisfy userdemands. Important video codecs can broadly beclassified into two categories: DCT-based and wavelet-based. With increasing licensing cost of proprietaryvideo codecs, the open-source community has startedgaining popularity apart from the efforts of standardi-zation committee. Many open source communitieseventually started working on royalty-free video codecswith an aim to give comparable quality to that ofstate-of-the-art video codecs. This included both DCT-based (e.g. Theora[3]),ongoing WebM[4]and wavelet-based approaches (e.g. Dirac [5]).

The variety of devices available in the range frommobile phone, tablets, internet TV, DVD players, and

so on, have different capabilities, requirements andvarying connection quality. This motivates us to haveadaptability within a bitstream so that the bitstreamgets automatically adapted to various needs. This canbe done at the network side by truncating the bit streamas per the requirement of the end user. This leads tothe concept of scalability in the video bit stream.

This paper gives an overview of scalable videocoding in Section II. Section III describes the wavelet-based approaches for scalable video coding. Anoverview of an open-source wavelet-based codec,namely Dirac, is then presented in Section IV. SectionV then describes the integration of spatial scalabilityinto the Dirac video codec. Results and conclusionsare drawn in Section VI and Section VII respectively.

II. SCALABLE VIDEO CODING

Video scalability is not a new concept. In fact, itis present in the previous video coding standardsranging from MPEG-2, MPEG-4 Visual and also asan extension in the state-of-art AVC|H.264. However,

20 A. A. BHURANE, P. R. CHAPLOT AND V. M. GADRE

researchers are still striving towards an optimal scalablesolution: optimal in terms of coding gain obtainedwith the cost of additional complexity.

A video bit stream is said to be scalable if it hasthe ability to adapt to multiple decoders. This allowsus to encode a video once and decode as per therequirements of end users. Apart from the benefit ofsingle encoding-multiple decoding, scalability offersother benefits like backward compatibility and smoothcontrol over the tradeoff between degradation andavailable bitrate.The performance of a scalable bitstream is evaluated by comparing it with the twoextremes of single layer and simulcast bit streamsbased on various parameters like Peak Signal to NoiseRatio (PSNR), structural similarity (SSIM) index, andVideo Quality Metric (VQM). A Single layer bit streamis a bit stream which does not provide scalability. Inthe case of simulcast, two or more individuallydecodable bit streams each catering to a particularquality and/ or resolution are bundled together to forma single bit stream. The total available channelbandwidth is divided between these layers.

Traditionally, a layered approach is used forscalability. In this approach various levels of qualityand resolution are packed in a bit stream. The lowestpossible quality of a video bit stream that can beextracted from the encoded scalable bit stream istermed as the base layer and subsequent layers asenhancement layers. A scalable video bit streamcomprises of a base layer and one or moreenhancement layers. Only an enhancement layer byitself is not useful to decode the video at that level.The enhancement layer at any level needs to becombined with all its preceding layers.

There are various types of scalability. Three mostpopular dimensions of scalability include spatial (Fig.1.), temporal (Fig. 2) and quality (Fig. 3). Othersinclude scalability with respect to bit-depth, croma,complexity, object, region-of-interest (ROI), and view.Hybrid scalability can also be achieved by using acombination of two or more scalability techniques.

A. Spatial Scalability

A video bit stream is said to be spatially scalableif it can be decoded at multiple spatial resolutions.Therefore, a spatially scalable bit stream whentruncated at a point and decoded gives video at one

resolution and when decoded further builds the videoat higher resolution. The ratio of resolutions betweentwo consecutive layers is in powers of two for thedyadic case of spatial scalability. Dyadic case whenextended for ratios other than two is called extendedspatial scalability [21].

A typical structure for two levels of dyadic spatialscalability is shown in Fig. 4. The raw input CIFvideo is resized to QCIF resolution and fed as aninput to the base layer encoder, which is a normalsingle-layer encoder, to get a base-layer bit stream.The encoded base-layer video is then decoded by thelocally available decoder at the encoder side. Thisdecoded base-layer bit stream is resized back to theoriginal video resolution at the input. The differencebetween this resized video and original video is calledinter-layer residue. The inter-layer residue is thenencoded by the enhancement-layer encoder to give anenhancement level bit stream. Both base layer andenhancement layer bit streams are then multiplexed toform a single spatially scalable bit stream. The spatialresolution levels can also be non-dyadic. Also the levelsof scalability may be greater than two. The maximumresolution that can be achieved by a scalable bit streamis of course the same as that of the original videoresolution.

B. Temporal Scalability

Temporal scalability refers to the ability to decodea video at multiple frame rates. A base layer will haveminimum frame rate and the frame rate increases aswe go up the layers. A typical block diagram for dyadictemporal scalability is shown in Fig. 5. The raw inputvideo at 30 Hertz is given as input to temporaldecimator which results in an output video at 15 Hertz.This low frame rate video is encoded by the baselayer encoder to give the base-layer bit stream. Thebase-layer bit stream is decoded by the locally availablebase layer decoder. The locally decoded base layervideo is then interpolated back to the original framerate using motion prediction residuals,either from thebase layer itself or from the original video frames.These residuals are then encoded to form anenhancement layer bit stream. The base layer andenhancement layer bit streams are then multiplexed toform a temporally scalable bit stream. The maximumframe-rate at which a scalable video can be extractedis equal to the original video frame rate.

Scalable Video Coding Systems: Basics, Open-Source Approaches and Advances 21

Fig. 1. Illustration of Spatial Scalability

Fig. 2. Illustration of Quality (PSNR) Scalability

Fig. 3. Illustration of Temporal Scalability

C. Quality Scalability

Quality scalability enables decoding a video bitstream at multiple qualities. The quality scalability isalso termed as PSNR scalability as the quality of thevideo is quantified by the most popular metric, PSNR.Fig. 6 shows the block diagram for a typical two layerquality scalability. The PSNR scalability is obtainedby modulating the quantization parameter Q which isvaried at different layers.

The quantization parameter directly affects thequantization matrix for transform coefficients at theencoder. Initially Q is selected as Q

0 which is a specific

coarse quantization parameter selected in order tosatisfy requirements at the base layer. The input videois encoded at this Q

0 to obtain a base layer bit stream.

The base layer bit stream is decoded by the localdecoder module available at the encoder side withquantization parameter Q

0-1. The difference between

the original video and the decoded video is thenencoded at a higher precision Q

1, to obtain an

enhancement layer bit stream. The difference can betaken in the transform domain. The two resultant bitstreams are then multiplexed to form a single qualityscalable bit stream.

III. WAVELET-BASED APPROACHES FOR

SCALABLE VIDEO CODEC

Wavelet transforms have been often reported togive better performance as compared to the DCT inimage compression. A detailed comparison betweenthe DCT and wavelet transform for image compressioncan be found in the work by Zixiang et al. [6]. Therehave been several attempts for incorporating waveletsin video compression. Wavelet coding techniques ofvideo can be classified into three categories:

(1) spatial-domain motion compensation followedby 2-D wavelet transform;

(2) wavelet transform followed by frequency-domainmotion compensation [22];

(3) 3-D wavelet transforms without [7] [8] and with[9] [10] motion estimation based approaches.

22 A. A. BHURANE, P. R. CHAPLOT AND V. M. GADRE

Fig. 4. Typical Block Diagram For Two Level Dyadic Spatial Scalability

Fig. 5. Typical Block Diagram For Two Level Dyadic Temporal Scalability

Fig. 6. Typical Block Diagram For Two Level PSNR Scalability

Scalable Video Coding Systems: Basics, Open-Source Approaches and Advances 23

What makes the wavelet-based approach moreattractive is its inherent scalable nature. Videoscalability refers to adaptability of the bit-stream toheterogeneous devices. However scalable extensionsare also available for non-wavelet based video codecs.

Considering the need of scalability and popularityof newly emerged 3-D wavelet video coding schemeswith Motion Compensated Temporal Filtering (MCTF),in October 2003, ISO/MPEG issued call for proposalson scalable video coding technology. Most of theproposals were wavelet-based.

Among the wavelet-based approaches, the one byMicrosoft Research Asia (MSRA), namely VidWav(video wavelet) was found to be the best inperformance.The objective and subjective comparisonbetween non-wavelet based model and VidWav laterconfirmed that the performance of VidWav was inferiorto that of non-wavelet based model. It was thereforerecommended by the investigation group to temporarilydiscontinue further exploration of wavelet basedapproach for scalability in standardization of scalablevideo coding. However, future exploration of waveletvideo coding as an alternative in standardizationframework was encouraged [11]. Presently, the state-of-art scalable video codec is JSVM (Joint ScalableVideo Model) which is a non-wavelet based approachbut the model does provide an option for MCTF.

Several other attempts were made for incorporatingwavelets at the core of video compression. A goodoverview of wavelet-based approaches for scalablevideo coding can be found in the work by Adami etal. [12]. One of the promising attemptsforincorporating the wavelet transform in a video codecwas made by BBC.

IV. DIRAC: A WAVELET-BASED VIDEO CODEC

Dirac is an experimental, general-purpose, open-source, royalty-free video codec developed by BBC.It is named after the famous British physicist, Paul A.M. Dirac. It is one of the several attempts to incorporatethe wavelet transform in a video codec. The Diracproject maintains two encoder implementations: Dirac-research, a research encoder, and Schrödinger, whichis meant for user applications [5]. We will be discussingabout the Dirac-research implementation. Started in

2003, the first experimental version of the codec wasreleased as Dirac 0.1.0 in March, 2004. A stable versionof the Dirac-research codebases, Dirac 1.0.0, wasreleased in September, 2008. The complete timelinefor Dirac video codec can be found in the workbyKeonget al. [13].

Dirac has been reported to give comparableperformance to state-of-the-art (H.264|AVC) videocodec at higher bitrates [13] [19]. Recently, the DiracPro (intra-only version of Dirac) [18] has also beenstandardized by the SMPTE as “VC-2”.

While conventional video standards have a DCT-cum-motion compensation based hybrid structure,Dirac is based on wavelets and motion compensation.Dirac is simple, available freely to implement withoutany royalty. Even so, it has highly configurable tools.Dirac supports a range of resolutions from QSIF(176×120) through UHDTV8K (7680×4320). Like anyconventional video codec, Dirac architecture comprisesthe following main elements: transform (wavelet),motion estimation and compensation, quantization andentropy coding. The Dirac video codec architecture isshown in Fig. 7.

Three types of frames are defined in Diracencoder: Intra frame (I) which is coded in its ownright, Level 1 frame (L1) which is forward predictedand Level 2 frame (L2) which is predicted in bothforward and backward directions. The frame structuredefined in Dirac is shown in Fig. 8

Fig. 7. Dirac Video Codec Architecture[14].

24 A. A. BHURANE, P. R. CHAPLOT AND V. M. GADRE

Fig. 8. Group of Pictures (GOP) Structure in Dirac [14]

The size of GOP can be varied by specifying thenumber of L1 frames and separation of L1 framesusing formula:

GOP size = (Num. L1 + 1) × (L1 separaation)

Setting the number of L1 frames as zero implies I-only coding.

Dirac uses hierarchical motion estimation in whichmotion estimation is performed initially on the lowerresolution level and refined gradually to the highestresolution as shown in Fig. 9. Motion vector precisionup to 1/8 pixel accuracy is supported in Dirac. Motioncompensation is carried out using Overlapped Block-based Motion Compensation (OBMC).

Motion compensated residuals are then wavelettransformed by the famous parent-child relationshipin wavelet sub-bands. This relationship says that acoefficient is more likely to be significant if its parentis marked to be significant as illustrated in Fig. 10.The coefficients are then quantised using the ratecontrol algorithm suggested in the work by Tun et al.[16].

The quantized coefficients are then scanned in aparticular order and entropy coded based on threestages: binarisation, context modelling and adaptivearithmetic coding.

Fig. 9. Hierarchical Motion Estimation [14].

Fig. 10. Parent-Child Relationship in Dirac [14]

Fig. 11. Integrating Spatial Scalability in Dirac Video Codec

V. INTEGRATING SPATIAL SCALABILITY IN

DIRAC VIDEO CODEC

In this section, we present a modification of theDirac codec to include spatial scalability. AlthoughDirac was not originally envisaged as a scalable codec,keeping the video codec untouched at its core, we canintegrate external spatial scalability for Dirac videocodec. We have done so and we present ourmethodology and results.

Here, we have restricted ourselves for intra-onlycoding as it provides the best case for scalable coding.Scalable coding for inter-frame case needs to beinvestigated in greater depth.We adopted a simple,basic model for spatial scalability as shown inFig. 11.

The experimentations are carried out for two-leveldyadic case of spatial scalability. Original raw input

Scalable Video Coding Systems: Basics, Open-Source Approaches and Advances 25

video is downsampled by filtering the frames with the MPEG-4 downsampling filter with point spread function(PSF) given by {2, 0, -4, -3, 5, 19, 26, 19, 5, -3, -4, 0, 2}/64 and the output is presented by every second sample(in horizontal and vertical direction) of the filtered images. This downscaled video acts as an input to Diracintra-only base-layer encoder. The resultant bit stream is decoded by the Dirac intra-only local base-layerdecoder. The locally decoded base-layer video is up-scaled back to the original resolution.

Fig. 12. PSNR Versus Compressed File Size for CITY Sequence

Fig. 13. PSNR Versus Compressed File Size for CREW Sequence

26 A. A. BHURANE, P. R. CHAPLOT AND V. M. GADRE

Fig. 14. PSNR Versus Compressed File Size for FOOTBALL Sequence

Fig. 15. PSNR Versus Compressed File Size for FOREMAN Sequence

Scalable Video Coding Systems: Basics, Open-Source Approaches and Advances 27

Fig. 16. PSNR Versus Compressed File Size for WAVE_VMG Sequence

This is done by preserving every second samplein horizontal and vertical direction of the input image,and the missing luma and croma samples areinterpolated using the AVC half-sample interpolationfilter with PSF {1, -5, 20, 20, -5, 1}/32 and {16,16}/32 respectively. Choice of the downsampling filters isas per the recommendations for JSVM [15] [21].

TABLE ITest Sequences

Sr.no. Name Frame rate

1 CITY.yuv 15

2 CREW.yuv 30

3 FOOTBALL.yuv 15

4 FOREMAN.yuv 15

5 WAVE_VMG.yuv 25

VI. EXPERIMENTAL RESULTS

Simulations for spatial scalability in Dirac, asdescribed in Section V were carried out on five differentCIF (352×288) sequences as shown in Table 1.

The base layer resolution was kept at QCIF(176×144) and enhancement layer at CIF (352×288)resolution. The target bitrates were varied andcorresponding PSNR values were noted. Fig. 12

through Fig. 16 shows the results for spatial scalabilitywhich is compared with single layer and simulcastencoding. The curves for a scalable bit stream clearlyindicated better performance over simulcast encoding.The gain of around 1.5 dB was obtained as comparedto the simulcast case.

VII. CONCLUSIONS AND CURRENT EFFORTS

This paper started with an overview of scalabilityin video coding. A promising wavelet-based videocodec, namely Dirac was introduced. Although Diracwas not designed to be scalable, it was extended tohave external spatial scalability using some of thefeatures adopted from state-of-art scalable videoextension i.e. JSVM. Simulations show that a resultantgain of around 1.5 dB can be obtained with respect tothe simulcast case restricted for intra-only coding.However, incorporating scalability in a long-GOPversion of Dirac needs to be considered. Also, moreefficient encoding of inter-layer residuals is requiredso as to compete with the de-facto standard.

Currently, HEVC is the joint standardization effortby ITU-T Video Coding Experts Group (ITU-T Q.6/SG 16) and ISO/IEC Moving Picture Experts Group(ISO/IEC JTC 1/SC 29/WG 11). HEVC aims atreduction of around 50% of bitrate while preservingthe same quality as that of the state-of-art AVC|H.264.

28 A. A. BHURANE, P. R. CHAPLOT AND V. M. GADRE

At the time of writing this paper, out of 21 responsesfor the scalable extension of HEVC (SHEVC), 6proposals were selected to begin with. Although theinitial software for the scalable HEVC project is yetto be finalized, an initial software candidate based onHM 8.1 version Inter-layer texture prediction, whichis the most common denominator among the call forproposals, is being considered [17].

ACKNOWLEDGMENT

The authors would like to thank Tim Borer, BBCand Anant Malewar, Nex-Robotics for theirencouragement and suggestions.

REFERENCES

[1] ITU-T [online] http://www.itu.int/en/pages/default.aspx(Date accessed: 18 Oct 2012)

[2] MPEG [online] http://mpeg.chiariglione.org/ (Dateaccessed: 18 Oct 2012)

[3] Theora Project [online] http://www.theora.org/ (Dateaccessed: 18 Oct 2012)

[4] WebM Project [online] http://www.webmproject.org(Date accessed: 18 Oct 2012)

[5] Dirac [online] http://diracvideo.org/ (Date accessed:18 Oct 2012)

[6] ZixiangXiong; Ramchandran, K.; Orchard, M.T.; Ya-Qin Zhang; , A comparative study of DCT- and wavelet-based image coding, Circuits and Systems for VideoTechnology, IEEE Transactions, vol.9, no.5, pp.692-695(Aug 1999).

[7] JB-J. Kim, Z. Xiong, and W. A. Pearlman, Low Bit-Rate Scalable Video Coding with 3D Set Partitioningin Hierarchical Trees (3D SPIHT), IEEE Trans. Circuitsand Systems for Video Technology, Vol. 10, pp. 1374-1387 ( Dec. 2000).

[8] Beibei Wang, Yao Wang, Ivan Selesnick and AnthonyVetro, Video Coding Using 3D Dual-Tree WaveletTransform, EURASIP Journal on Image and VideoProcessing(2007).

[9] Ohm, J.-R, Motion-compensated 3-D subband codingwith multiresolution representation of motionparameters, Image Processing, 1994. Proceedings. ICIP-94., IEEE International Conference , vol.3, no., pp.250-254 vol.3 (13-16 Nov 1994).

[10] Seung-Jong Choi; Woods, J.W., “Motion-compensated3-D subband coding of video,” Image Processing, IEEETransactions on , vol.8, no.2, pp.155-167, Feb 1999.

[11] ISO/IEC JTC 1/SC 29/WG 11, Report of 76th meeting,N7983 Montreux, CH ( April 2006).

[12] Adami, N.; Signoroni, A.; Leonardi, R., State-of-the-Art and Trends in Scalable Video Compression WithWavelet-Based Approaches, Circuits and Systems forVideo Technology, IEEE Transactions , vol.17, no.9,pp.1238-1255( Sept. 2007).

[13] Keong, Kok, MyoTun and Yoong Choon Chang. DiracVideo Codec: Introduction,Streaming MediaArchitectures, Techniques, and Applications: RecentAdvances. IGI Global, 2011. 58-94. Web. 18 (Oct.2012).

[14] Dirac [online] http://dirac.sourceforge.net/

[15] HHI [online] http://www.hhi.fraunhofer.de/endepartments/image-processing/image-video-coding/svc-extension-of-h264avc/jsvm-refere nce-software/

[16] M. Tun, K.K. Loo, J. Cosmas, Rate control algorithmbased on quality factor optimization for Dirac videocodec, Signal Processing: Image Communication,Volume 23, Issue 9, pp 649–664,(October 2008).

[17] Andrew Segall, BoG report on SHVC, dry ITU-T SG 16WP 3 and ISO/IEC JTC 1/SC 29/WG 11,JCTVC-K0354, 11th Meeting: Shanghai, CN, (10–19 Oct. 2012).

[18] BBC http://www.bbc.co.uk/rd/projects/dirac/diracpro.shtml

[19] Kalra, V.; Wahid, K.; Dinh, A., Video codec comparativeanalysis between h.264 and DIRAC PRO/VC-2,24thCanadian Conference on Electrical and ComputerEngineering (CCECE), 2011, vol., no., pp.000951-000955 (8-11 May 2011).

[20] Schwarz, H.; Marpe, D.; Wiegand, T. ,Overview of theScalable Video Coding Extension of the H.264/AVCStandard,Circuits and Systems for Video Technology,IEEE Transactions on , vol.17, no.9, pp.1103-1120 (Sept.2007).

[21] Segall, C.A.; Sullivan, G.J., Spatial Scalability Withinthe H.264/AVC Scalable Video Coding Extension,Circuits and Systems for Video Technology, IEEETransactions on , vol.17, no.9, pp.1121-1135(Sept.2007).

[22] YiannisAndreopoulos et al., Comparison Between“t+2D” and “2D+t” Architectures with AdvancedMotion Compensated Temporal Filtering,ISO/IECJTC1/SC29/WG11 MPEG2004/SVC-CfP, m11045,Redmond, WA, USA (July 2004)