Video Coding Past, Present & Future Georgios Diamantopoulos Digital Systems and Vision Processing Group.

Video Coding

Past, Present & Future

Georgios Diamantopoulos

Digital Systems and Vision Processing Group

Today’s agenda

History of video compression standards State-of-the-art Showcase Relative performance analysis

Advanced video coding technology Motion Compensation demo

Image coding with JPEG2000 Features, how it works, demo

Future of video coders Video Quality Evaluation

Relation to Image Quality and overview of Metrics Getting started with video coding

About me

Graduated with BEng in Computer Systems Engineering in 2004

Started PhD the same year: Improve video compression Link psychology of vision with engineering Investigating how to re-arrange bits such that

perceptual quality is increased

Personal research interests: Video Quality Evaluation [Video Quality Studio]

How to use vision models to evaluate video automatically

The history

The early days (1950-1990)

First picture phone systems in the ’50s (B&W) First standardized effort (1982):

International Telecommunications Union-Telecommunication (ITU-T) H.120: a Differential Pulse Code Modulation (DPCM) codec

Target bitrate: 2Mbit/s for Europe, 1.544 for North America Poor temporal quality

The early days (1950-1990)

Need for less than 1 bit per pixel Solution: Block based coding

Discrete Cosine Transform versus Vector Quantization DCT won because of progressive image transmission JPEG 1984-1988

H.261: Video conferencing ITU-T, completed in 1990 Combination of DPCM and DCT Motion Compensation P x 64kbps

Familiar things (1990-1995)

MPEG-1: “Coding of moving pictures and associated audio for digital storage media” (1992) Target was VHS Quality at 1.5MBits/s Basis of Video-CD MP3 is still with us! (MPEG-1 Layer 3)

Familiar things (1995-2000)

MPEG-2: “Generic coding of Moving Pictures and Associated Audio”

Broadcasting and storage Bitrates: 4-9 MBits/s Satellite TV, DVD

MPEG-3? Aimed to do High Definition TV

(HDTV) MPEG-2 could do that anyway Folded into MPEG-2

MPEG-4: “Coding of audio-visual objects” Started as very low-bitrate project Turned out to be much more:

Coding of media objects 64kbps to 240Mbps (Part 10/H.264) Synthetic/Semi-synthetic objects XMT: Like HTML, but to build videos First standard with Intellectual Property

Management

Present & Future (2000-2010)

MPEG-4 Part 10: Advanced Video Coding / H.264 Designed by a Joint MPEG and

ITU-T group Claims 50% bitrate savings to

MPEG-2, 30% over MPEG-4!

H.265 aims to have 50% better compression

MPEG-7: “Multimedia Content Description Interface” (2001) Describing audio/video

Applications Indexing of video databases Search & Retrieval Browsing

Independent players

Efforts from independent companies in video compression

Main competitors: RealNetworks – http://www.real.com Microsoft Windows Media (WM) – http://www.microsoft.com Apple Quicktime – http://www.apple.com On2 – http://www.on2.com

Standards have wider acceptance Recently Microsoft wrapped their WM technology

Standard: VC-1 An attempt to compete with MPEG/ITU… succeeded

The next DVD format (Blu-Ray) will support H.264 and VC-1

Where is MPEG used? Most probably.

MPEG-1 Video-CD Usually .mpg or .mpeg files are MPEG-1 DAB Digital Radio is MP2 (MPEG-1 Layer 2) MP3 files (MPEG-1 Layer 3)

MPEG-2: .vob, .m2v, rarely .mpg files Anything to do with DVD

Camcorders, DVD players, DVD recorders, TiVo Digital TV

MPEG-4: High Quality AVI files Video Phones DivX Some advanced audio players support MPEG-4 Advanced Audio Coding (AAC)

NetMeeting and similar video-chat H.263/+/++

H.264 Some content has appeared recently, mainly trailers

Even if you haven’t…

MPEG-1 VideoCD-like (352 x 240/288 @ 1200 kbps)

MPEG-2 DVD-like (720 x 576 @ 6500-7000 kbps)

MPEG-4 High Quality (720 x 576 @ 1000-1800 kbps)

HDTV (1920 x 1080) H.264 Windows Video 9

* Material encoded from H.264/WMV9 original with TMPGEnc (MPEG-1/2) and XviD (MPEG-4).

R-D Performance of MPEG Codecs

32

34

36

38

40

42

44

46

48

50

350 450 550 650 750 850 950 1050

Bit rate (kbps)

PS

NR

(Y

)

MPEG-1 MPEG-2 MPEG-4 H.264

Mission Impossible 3 HDTV trailer (1080p) resized to 720x304. Encoders: TMPGEnc (MPEG-1/2), XviD (MPEG-4), x264 (H.264)

CODEC Design

Reproduced from “H.264 and MPEG-4 Video Compression” by I. E. G. Richardson.

Motion Compensation

It aims to reduce the data transmitted by detecting the motion of objects Use the previous as reference In steps:

Split the current frame in blocks. For each one: Find the best-matching block in the reference frame

How? We search an area in the reference frame and compare, using Sum of Absolute Differences (SAD) Mean SAD (MSAD)

The best matching block is coded and transmitted Next frame can be used a reference too

Demo

Advanced Video Coding

Bidirectional frames When performing motion estimation, each block can

be assigned a match from either a backward or forward reference

Global Motion Compensation A set of vectors that describe the whole object plane

can be transmitted

Advanced Video Coding

Other tools Quarter-pixel motion compensation precision

Bigger vectors but better prediction (less error) Spatial prediction (intra-level)

Prediction of pixels spatially Motion vectors off-picture 4x4 integer transform

Faster, reversible, exploits spatial correlation better Deblocking filter

Executed on the encoder, reduces blocking Multiple reference frames

JPEG2000 – What’s new

Resolution and quality scalability Encode once, decode at a resolution and quality

appropriate to the client e.g. Mobile phone: low resolution/quality High performance PC: high resolution/quality

Random access Any part of the stream can be decoded independently

Lossless coding is possible Fast, highly efficient in terms of compression Data is organized in layers with different

capabilities

JPEG2000 – How it works Component transform is

RGB to YUV Wavelet transform gives a

multi-resolution representation of the image

Quantization is optional

Bytes are organized in bit planes

Post-compression R-D optimisation “truncates” the planes to achieve optimal distortion for the specified rate

JPEG2000 - Showcase

NASA Image 8688 x 8296 pixels RGB Data = 206MBytes

JPEG2000 Tiles 200x300 Wavelet decomposition/Resolution levels = 8 6 Quality layers = 0.01, 0.05, 0.1, 0.25, 1.0, lossless Output size = 101MBytes

Demo

The future standard is…

MPEG-21: JPEG2000 for videoScalability

Resolution Quality But also… Temporal

In other words, from one stream, you can get streams with variable resolution, quality and frame-rate

In the same fashion we saw with JPEG2000

The future product is…

High Definition DVD1080 pixels vertical resolution, 60 HzBlu-Ray discs

25/50GB of storage 4 hours of HD video

H.264 and Windows Media Video 9First prototype set-top boxes have appeared

Video Quality Evaluation

Similarities to Image Quality Evaluation Eye more sensitive to chrominance Some areas attract more attention that others Blocking/ringing artifacts are annoying …

Differences Temporal dimension

Moving versus stationary objects Frame rate plays a significant role

Any less than 24 Hz is not perceived as moving images but as slideshow

Video Quality Evaluation

SubjectiveA human “subject” rates the video on a scale

Double Stimulus Continuous Quality Scale Method Hidden scale of 0-100 Difference is calculated as the

actual rating

Video Quality Evaluation

ObjectiveA computer algorithm judges the distortion

between videosAttempts to model a human observerThere is currently no standard method

Objective Metrics

Peak Signal-To-Noise Ratio (PSNR)Used widely in evaluating coding performancePurely mathematical difference

Can be tricked quite easily

Objective Metrics

How to trick PSNRTake a natural imageGive more bits to areas you look at moreGive less bits to areas you look at lessSubjective rating will be high, PSNR low

Better metrics Not perfect

Video Quality Experts Group

Evaluation of objective metrics

Video Quality Studio

Getting started - Programming

Learn to read and write RAW images Process them somehow

You can move on to YUV video sequences Get visual

Win32 API, Windows Forms (.NET) Processing video files

AVI files – Video For Windows API Tons of open source MPEG/H.xxx libraries (libmpeg2, libavcodec, MPEG4IP,

XviD etc.) Advanced editing

DirectShow API Very complex, get a book

Languages C/C++ is the de-facto standard C++/CLI (Managed C++) if you want to access .NET libraries VB: Definitely a no-no, slow and cumbersome C#: Better than VB but still not for performance coding

DirectShow

Getting started - Tools

Jasc Paint Shop Pro Now owned by Corel GIMP is a free alternative

IrfanView Excellent & free image viewer with saving capabilities Supports JPEG2000!

Virtualdub Free video processing toolkit Book: VirtualDub Video: Capture, Processing and Encoding

Media Player Classic Lightweight, open source media player Replica of Windows Media Player 6.4 A LOT of options

TMPGEnc Free MPEG-1/2 encoder

Getting started – Codecs/Content Mostly open-source

XviD Ogg Theora (wavelets) BBC Dirac (wavelets) On2 VP3 x264 OpenJPEG Jasper

Commercial, free versions DivX

Where to get content Extract them from your DVDs – be careful who you show it to YUV Sequences

http://www.cipr.rpi.edu/resource/sequences/ ftp://ftp.crc.ca/crc/vqeg/TestSequences/

High Definition Apple trailers: http://www.apple.com/trailers/ Microsoft showcase:

http://www.microsoft.com/windows/windowsmedia/content_provider/film/ContentShowcase.aspx

Summary

Historic development of video coding Who makes codecs/standards Examined samples of each generation and their relative performance

Codec design Motion estimation demo Advanced video coding

Wavelet coding: JPEG2000 Features Technology Demo

What the future holds Video Quality Evaluation

Objective/Subjective Evaluation/Metrics Video Quality Studio

Video coding how-to

Get in touch

If you like programming and want to get involved with cool real-world projects, get in touch

I am willing to give more technical/practical seminars if there is interest so let me know

Any Questions?

Contact information

Web: http://postgrad.eee.bham.ac.uk/gxd186/

E-mail: [email protected]

Bibliography

Video Quality Studio: http://www.visumalchemia.com/vqstudio/

History: http://www.pha.jhu.edu/~sundar/intermediate/history.html http://myhome.hanafos.com/~soonjp/vchx.html

F. Pereira, T. Ebrahimi. The MPEG-4 Book. I. Richardson. H.264 and MPEG-4 Video Compression. MPEG: http://www.chiariglione.org/mpeg/ VQEG: http://www.vqeg.org Video For Windows:

http://msdn.microsoft.com/library/en-us/multimed/htm/_win32_video_for_windows.asp

DirectShow: http://msdn.microsoft.com/directx/sdk/