Multimedia object types: video ISMT Multimedia 2001 Dr Vojislav B Mišić.

Multimedia object types:video

ISMT Multimedia 2001Dr Vojislav B Mišić

ISMT Multimedia Lecture 05/2 © 2001 Dr. Vojislav B. Mišić

Lecture Outline

Basic characteristics of moving imagesHuman perception of motionSome technical issues of computer representation of video, with emphasis on color

This includes a rather detailed presentation of principles of color TV, I’m afraid

Video compression: principles and implementation


Vision

Once again: vision is probably the most important among our sensesHumans are able to perceive intensity as well as colorHuman eye is more sensitive to detail in black-and-white (small details are not perceived in color) – designers of color TV made use of these characteristicsColor is not perceived well under low lighting conditions


Pictures in motion

Persistence of vision allows images shown in rapid succession to produce the illusion of motionMotion obtained above about 15 to 20 images (or frames) per second

At lower frame rates, individual frames are noticeable – “flickering”

At higher frame rates, motion becomes smooth


See the difference … This small video has 4 frames per second …

… and this one has about 24 (but the steps are still noticeable, maybe because of the computer itself)


A simple arithmetic

An image is worth a thousand wordsEach second of video has about 25 imagesTherefore, a second of video is worth about 25,000 wordsWhich amounts toquite fast talking …


History (and how it repeats itself)

Before we investigate the issues of computer video, let’s take a brief look at how moving images were produced before the computer age – in movies and TV setsMovies: the first approach to producing moving pictures (that’s why they are called movies, after all)


Movies work like this …

Frame rate: 24 frames per secondEach frame is illuminated twice,for a total of 48 framesNo flickering – and no attempts toincrease the frame rate


More on movies

24 fps is enough - no attempts have been made to increase the frame rate

Although cameras can shoot at higher rates, and often do – in order to produce smooth motion when the scene is played back in slow motion

Width-to-height ratio (usually called the aspect ratio) of 4 to 3Wider picture produced with special optics (Cinemascope, Panavision), with some success


TV pictures

Why not enjoy moving images at home?TV: an ingenious solution of a technically difficult problem – how to transmit 2D data through a 1D channel (compare that to sound!)First attempts in late 30s (monochrome)Monochrome, unfortunately – because the design of monochrome TV has been the prime culprit for a multitude of problems, usually referred to as two-way compatibility, when…Color TV was introduced in early 60s


TV basics

An electron beam scans the image in horizontal rows – lines (525 in USA, but 625 in Europe, Hong Kong included)Frame rate synchronized to the power frequency: 30Hz in USA, 25Hz in Europe/HK

Of course, frame rates are quartz-controlled now … but the numbers remain (almost) the same

Scanned image transmitted line by line, with appropriate synchronization informationAnd the lines look like this …



TV display

Image re-created on the receiver sideTV picture tube: scanning again, this time with the electron beam intensity modulated by the signalPhosphor dots illuminated through a mask (grid) radiate visible lightSomething like this …


[tv display from encarta]


Actually, dots are not circular

Not any more, that is …And Trinitron uses a striped mask, givingmuch brighter image


Interlacing

Devised because of less-than-perfect electronicsEven-numbered lines scanned in one half-frame, odd ones in the other, resulting in twice as many half-framesbut: lo and behold!At the same time, flickering is reduced …


Enter the Colors …

How to add color?…without disrupting the existing monochrome signalSeparate carriersignal (chromasubcarrier)Clever codingof color info(YIQ in USA,later YUV inEurope)


In Terms of Frequency…


TV systems in use

NTSC (US): the first to use YIQ codingPAL (Germany): an improved NTSC

Chroma subcarrier at 4.43MHz

One of the signals (R-Y) is inverted in alternate lines (hence the name), which leads to better color stability

Severe phase changes in the transmission of a PAL signal will be seen as weak—but correct—colors

Severe phase changes in NTSC transmission will be seen as strong colors – but the wrong ones! (green faces etc.)

Also: PAL uses 50 fps, not 60 like NTSC


Other did exist …

SECAM (Sequentiel Couleur a Memoire, in France): humans don’t perceive the color of small details well

One color component (say, U) transmitted in odd-numbered lines, another (V) in even-numbered ones

The missing color information is simply copied form the previous line

Chrominance channel has only half the resolution (409 lines) of the luminance channel (819)

Plus, FM modulation keeps the colors stable

Nowadays, SECAM is virtually extinct – but modern digital coding schemes still exploit the same effect…


A word about signals & connectors …

(ordered by increasing quality)RF: like the original broadcast signal, minus the noiseComposite: like the RF, minus the modulationS-video: luminance signal uses a separate wire from the two chrominance signals, better quality than compositeComponent: each of the signals has a separate wire (luminance and two chrominance)RGB: if only you could get the one directly from the camera …


Modern TV developments

analog HDTV many years and much money spent in Japan little acceptance because of compatibility problems

digital TV and/or HDTV (USA and Europe): over 1000 lines (1125 or 1200) Wider aspect ratio (16:9 and even more) All digital processing expected to take off soon


Analog Storage

Analog TV recorded on magnetic tapeHowever, the bandwidth required dictates high speed… which in turn dictates helical recording, rather than longitudinalAt the same time,FM modulation isused to improveperformanceAll in all: VERYcomplicated!


Analog Problems

Tape is NOT a good medium, it is prone to wear and tearEach subsequent copy is worse than the original…and the original is not that good anyway

Usable resolution: about 200 to 250 lines, NOT 500

And of course we should mention the VCR wars between Beta and VHS (read: Sony vs. the world)

Was Beta technically better? … not really Beta cassettes were smaller (and, possibly, sturdier to

handle) … but VHS could store two hours instead of one


Digital video

Video signal obtained by scanning is continuous in both time and amplitudeAs always, it must be sampled and digitizedDigital processing has a few advantages, primarilyLossless archiving and copyingBetter quality and better scalability

Plus, more processing options available Processing can be partially automated (batch

processing)


A brute-force approach

Digital video – a sequence of digitized images, played back at sufficient frame rateIf we were to transmit a sequence of color images, we would need...

16 or 24 bits per pixel (for good color) at least 640 x 480 pixels (to obtain acceptable

resolution) at least 25 frames per second, to obtain flicker-free

display

Which comes out at about 15MB per second, or over 100 GB for a full feature movie


Possible Cure: Compression

Video signal contains some redundancy (actually, a lot of it, most of the time)Compression exploits that redundancy in order to reduce file sizes and transmission speeds requiredTwo basic approaches to compression exploit two types of redundancy: intra-frame and inter-frame


Intra-frame compression

Deals with frames on an individual basis, and tries to exploit the redundancy present in each frameEssentially the same as for still images (most still image algorithms can be used for video)However, compression and decompression must be performed in real-time – making some algorithms (marginally) less suitable than others


Inter-frame compression

Exploits redundancy present between successive framesAssumption: only some pixels in an image will be changed from one frame to the next oneMost often based on some sort of prediction (or estimation) how the pixels will change from frame to frame


Well?

Let’s look at an example: some cyclists running in circles

The differences are small indeed …


Pixel-based predictive coding

Color does not change too quickly between successive pixelsTherefore, the color of a pixel may be estimated based on the colors of its neighborsWe code the difference between the actual value and the predicted one (possibly with fewer bits) and transmit

But compression ratios are not high (about 2:1 only)

If a frame is lost in transmission, a noticeable distortion will appear …


Macro blocks

Macro block: an area, or block of pixels that may move – but does not changeSignificant reductions, but a sophisticated algorithm for identifying macro blocks is needed


More on compression

Another classification: lossless vs. lossyAs before, lossless compression techniques don’t compress muchLossy compression techniques are better… but there is a price: loss = distortion

type of distortion depends on the compression technique

amount of distortion depends on the compression (loss) ratio


Computer video formats

A wide variety, with different levels of flexibility and sophisticationAmong the most popular ones are:

AVI (Microsoft and IBM) MPEG (MPEG) QuickTime (Apple)

Even more formats when we discuss multimedia on the Internet


AVI format

A single stream of interleaved audio and video frames, thus providing automatic synchronization between the twoImage size,frame rate, andcolor depth maybe adjustedindependently


MPEG

MPEG (Motion Picture Experts Group) is another ad hoc standards body, specifically intended to create standards for digital media transmission and storageMPEG-1: first incarnation of the standard, widely used for VCDMPEG-2: the second generation of the MPEG standard, used in DVDMPEG-4, MPEG-7 standards in the works (will take time to materialize)


MPEG-2

Second generation of the MPEG standardHardware decoders/add-on cards still required (software-only decoders work with MPEG-1)Different bit rates with different compression ratios (and different quality)


MPEG-2 frame types

I-frames (intra-coded images): essentially still images; compressed as JPEG (but in real-time)P-framesB-framesD-frames


P-frames

Predictive-coded: require information of the previous I-frame and subsequent P-framesCoding based on image areas (macro blocks) that may shift between frames but do not change at all (motion estimation)Actual algorithm not specified, but the format for transmitting the motion vector is specifiedSmall differences between macro blocks in successive frames allowed


B-frames

When an object moves, it reveals the background – which has to be known …But we can fetch it from the futureMPEG solution: B-frames, playback requires both previous and following I- and P-frames


More frame types

A fourth type: D- (or DC-coded) framesD-frames contain only the lowest frequencies of an imageUsed for playback during fast-forward or rewind operationsI-frames could be used instead, but would require more storage space and more time to display


QuickTime

Widely used on both Macs and PCsA QuickTime movie is actually a container, with components called tracks

Each track contains digital media data of a single type

Each track is played at a predefined time

Recently adopted by ISO for use as the unified digital media storage format for MPEG-4


QT-supported track types

Video – a sequence of images in appropriate format, with defined image size, frame rate, and codec used (more on codecs in the next section)Sound (digitized audio)Music (based on the MIDI standard, and played through a synthesizer)Text track used to provide captions or indexing information


More track types

3D track, with geometric definitions of objects and descriptions of objects’ surfaces, including textures and lighting

A 3D track needs a tween track to define object motion

Sprite track contains graphic objects (sprites) that can be moved and resized independently

Each sprite stored only once, individual frames contain only motion information


Yet more track types

Some track types, such as timecode, tween, and base, do not correspond to specific media types. They are used for synchronization (during playback) and editingThe QuickTime Pro package provides basic editing capabilities at the track levelMore sophisticated editing packages (such as Adobe Premiere) are available


QuickTime VR

A special format for panoramic and object moviesA QuickTime VR movie must have

one video track a VR Panorama or a VR Object track

Additionally, hot spots can be defined for navigation to other points in the movie, or to launch external applications


QTVR

Panoramic Movies: camera is rotated to give a panoramic view of a landscapeObject movies: object is rotated in front of the camera, so that it can be seen from all sides


Codecs

Both QuickTime and AVI provide just the basic format for packaging the frames into a movie file, but make no assumptions on the actual compression algorithm usedDifferent compression algorithms can be implemented and used – codecsMost codecs can be added to an existing installation of recorder or player as plugins


Most common codecs

Apple Video Codec: basic codec provided with QuickTime, reasonable compression and speedApple Animation Codec: RLE compression, good for animationsMicrosoft Video Codec: basic for AVI, reasonable compression and speedCinepak: much better quality and compression than Apple/MS, not too fast


More codecs

JPEG: each frame is compressed as a JPEG image, fast compression, adjustable quality and compression ratioMotion JPEG: similar to JPEG, but with predictive coding (hardware support available)Intel Indeo: very good quality and compression, only on Intel-based platforms


Video Streaming

Streaming: timely delivery of video data so that it can be played as soon as it arrivesMultiple formats available

RealMedia (audio, video, …) QuickTime (long time leader) Windows Media (Microsoft)

More details on streaming later…


Summary

Video is very useful mediumPowerful production and editing tools are availableBut: there are subtleties to masterPlus, different formats exist (fortunately, most players are able to play almost all formats)

Multimedia object types: video ISMT Multimedia 2001 Dr Vojislav B Mišić.

Documents

tv displayimage

tv setsmovies

whencolor tv

frame ratemore

secondeach frame

color designers of color

issues of computer video

small video