Chapter 2 Enabling Technologies Multimedia Systems.

Chapter 2Chapter 2

Enabling TechnologiesEnabling Technologies

Multimedia Systems

Key PointsKey PointsAllAll types of media are represented digitally as pattern types of media are represented digitally as patterns of s of bitsbits. . Analogue signals must be Analogue signals must be digitizeddigitized, which comprises , which comprises samplingsampling and and quantizationquantization. . The The Sampling TheoremSampling Theorem states that, if the highest freq states that, if the highest frequency component of a signal is at fuency component of a signal is at fhh, the signal can be , the signal can be properly reconstructed if it has been sampled at a freqproperly reconstructed if it has been sampled at a frequency greater than 2fuency greater than 2fhh. . Contemporary personal computers are capable of plaContemporary personal computers are capable of playing back most types of media. ying back most types of media. Multimedia production places much greater demands Multimedia production places much greater demands on computer systems. on computer systems.

Key PointsKey PointsMultimedia production requires the use of Multimedia production requires the use of many different software applications, many different software applications, including including authoring systemsauthoring systems. . Networks, and the Internet in particular, Networks, and the Internet in particular, offer valuable opportunities for distributing offer valuable opportunities for distributing multimedia, but they also present multimedia, but they also present formidable technical difficulties. formidable technical difficulties. The most popular form of online multimedia The most popular form of online multimedia delivery is the World Wide Web, which uses delivery is the World Wide Web, which uses the the Hypertext Transfer Protocol (HTTP)Hypertext Transfer Protocol (HTTP) to organize its communications. to organize its communications. Various formal and informal Various formal and informal standardsstandards govern multimedia data, especially in the govern multimedia data, especially in the context of networks. context of networks.

Production and consumption of Production and consumption of digital multimedia depends on ability digital multimedia depends on ability of digital computer.of digital computer.– Media data in digital formMedia data in digital form

Program can be used to change, combine, Program can be used to change, combine, store and display media of all types.store and display media of all types.

– DistributionDistributionNetworksNetworks

Removable storage: CDROM, DVDRemovable storage: CDROM, DVD

Computers are not only devices that Computers are not only devices that can manipulate digital data.can manipulate digital data.– DVD, set-top box, CD playerDVD, set-top box, CD player

ComputerComputer– Full interactivityFull interactivity

Future: multimedia playerFuture: multimedia player– Cheaper than computerCheaper than computer

Digital RepresentationsDigital Representations

Bits, bytes, words.Bits, bytes, words.– Data structuresData structures– ProgramsPrograms

DigitizationDigitization– Analogue and digital representationsAnalogue and digital representations– Analogue to digital convertersAnalogue to digital converters– Sampling and quantizationSampling and quantization– Digital signals are much more robust than Digital signals are much more robust than

analogue ones, and do not suffer degradation analogue ones, and do not suffer degradation when they are copied, or transmitted over when they are copied, or transmitted over noisy media.noisy media.

Sampling and QuantizationSampling and Quantization

SamplingSampling– Measure the signal’s value Measure the signal’s value

at discrete intervalsat discrete intervals– Sampling rateSampling rate

QuantizationQuantization– Restrict the value to a Restrict the value to a

fixed set of levelsfixed set of levels– Quantization levelsQuantization levels

Analogue to digital Analogue to digital converter (ADC)converter (ADC)

DACDAC

Sample and holdSample and holdFig. 2.4Fig. 2.4– Not a good approximationNot a good approximation

Such a signal is passed to an output Such a signal is passed to an output device such CRT or speaker, these device such CRT or speaker, these discontinuities can be smoothed out.discontinuities can be smoothed out.

FrequencyFrequency– Cycles per second (cps)Cycles per second (cps)– Hertz (Hz): KHz, MHzHertz (Hz): KHz, MHz– Velocity = frequency * wavelength Velocity = frequency * wavelength – FFTFFT

Any periodic waveform can be decomposed Any periodic waveform can be decomposed into a collection of different frequency into a collection of different frequency components.components.

– FrequencyFrequencyTemporal, spatial Temporal, spatial Fig. 2.6, vary periodically in spaceFig. 2.6, vary periodically in space

– Fig. 2.7, Frequency components of a square Fig. 2.7, Frequency components of a square wavewave

– DC component, f=0DC component, f=0– Higher frequency: abrupt transitionsHigher frequency: abrupt transitions– Fig. 2.8, square wave in frequency domainFig. 2.8, square wave in frequency domain

FilterFilter

Remove certain Remove certain frequenciesfrequencies

High pass filterHigh pass filter

Low pass filterLow pass filter

Sampling TheoremSampling Theorem

If the highest frequency component of a signal If the highest frequency component of a signal is at is at ff, the signal can be properly reconstructe, the signal can be properly reconstructed at the sampling rate 2*d at the sampling rate 2*f.f.This limiting value is known as the Nyquist rate.This limiting value is known as the Nyquist rate.Fig. 2.9Fig. 2.9The hearing range of the average person is appThe hearing range of the average person is approximately 20 Hz to 17kHz.roximately 20 Hz to 17kHz.(Sounds above this range is called ultrasonic s(Sounds above this range is called ultrasonic sounds)ounds)Audio cards use 44.1kHz with sample sizes 8 anAudio cards use 44.1kHz with sample sizes 8 and 16 bitsd 16 bits

Undersampling and Quantization noiseUndersampling and Quantization noise

UndersamplingUndersampling– Sampling rate is less than the Nyquist rate.Sampling rate is less than the Nyquist rate.AliasingAliasing– In images, it is usually seen in the form of jaIn images, it is usually seen in the form of ja

gged edges.gged edges.Quantization noiseQuantization noise– Number of quantization levelsNumber of quantization levels

Image: grey levelsImage: grey levelsSoundSound

Hardware RequirementsHardware RequirementsMultimedia PC (MPC): Level 3Multimedia PC (MPC): Level 3

HardwareHardware

PDAPDA– Less powerful devicesLess powerful devices– Difficult to achieve the same performance Difficult to achieve the same performance

of PCof PC– Limitation on the amount of storageLimitation on the amount of storage– Often based on cellular telephone network, Often based on cellular telephone network,

less bandwidthless bandwidth– Challenge of web designersChallenge of web designers

Accessible to powerful PC and to PDAAccessible to powerful PC and to PDA

Multimedia architecture must be scaleableMultimedia architecture must be scaleable– Transparently to devices with different capabilitiesTransparently to devices with different capabilities

High quality multimedia production are demaHigh quality multimedia production are demanding than those for its consumptionnding than those for its consumption– Special-purpose graphics acceleration hardware aSpecial-purpose graphics acceleration hardware a

nd input devices for video and audiond input devices for video and audio– High-speed disks, video capture card, specialized pHigh-speed disks, video capture card, specialized p

eripheralseripherals– High-powered, SGI: 3D animation and video effects, High-powered, SGI: 3D animation and video effects,

common in film and TVcommon in film and TV– Special-purpose array processing in Pentium and PSpecial-purpose array processing in Pentium and P

owerPC (MMX, AltiVec)owerPC (MMX, AltiVec)Blur the distinction between high-end PC and workstationBlur the distinction between high-end PC and workstationss

Producing multimedia contentProducing multimedia content– Processing powerProcessing power– High-speed data busesHigh-speed data buses– Large memoriesLarge memories– Powerful graphics boardsPowerful graphics boards

Content preparation Content preparation – Substantial more storage than finished Substantial more storage than finished

productproductE.g. image layers up to 100E.g. image layers up to 100High quality audioHigh quality audioCompression only for final deliveryCompression only for final delivery

Speed of data transfer to and from disks is a serious liSpeed of data transfer to and from disks is a serious limiting factormiting factorDigital video: 5Mbytes to 30 MbytesDigital video: 5Mbytes to 30 MbytesIDE, SCSI-1 cannot cope withIDE, SCSI-1 cannot cope withSCSI-2 supports 40MbytesSCSI-2 supports 40MbytesUltra SCSI-2: 80MbytesUltra SCSI-2: 80MbytesFirewire, IEEE 1934: 50 Mbytes, DV camerasFirewire, IEEE 1934: 50 Mbytes, DV camerasCurrent disk speed for a dedicated computer: adequatCurrent disk speed for a dedicated computer: adequate performancee performanceHigher performance of disk for central serverHigher performance of disk for central server– RAID array: Redundant array of inexpensive disksRAID array: Redundant array of inexpensive disks– Improved performance in parallelImproved performance in parallel

AV diskAV disk

RAIDRAID

8 levels for different degrees of 8 levels for different degrees of performance and fault-toleranceperformance and fault-tolerance

RAID 0: no protection against disk RAID 0: no protection against disk failurefailure– If one disk fails, whole array failsIf one disk fails, whole array fails

RAID 1: mirrors diskRAID 1: mirrors disk

Specialized PeripheralsSpecialized PeripheralsGraphics tablet with a pressure-sensitive penGraphics tablet with a pressure-sensitive penLarge high-resolution monitorLarge high-resolution monitorTwo monitors: preparing images and display iTwo monitors: preparing images and display imagesmagesScannersScannersDigital camerasDigital camerasVideo camera and sound recording Video camera and sound recording – The equipment may itself be digital and produce diThe equipment may itself be digital and produce di

gital output that can be sent to computer over Firegital output that can be sent to computer over Firewire.wire.

– Video and audio capture from analogue equipmentVideo and audio capture from analogue equipment

SoftwareSoftwareThe essence of multimedia is the combination of The essence of multimedia is the combination of different media elements through computer different media elements through computer programs.programs.Different applications for different mediaDifferent applications for different media– Graphics: Image editing, painting, drawingGraphics: Image editing, painting, drawing– Text: editors, layoutText: editors, layout– Video: capture, editing, postproductionVideo: capture, editing, postproduction– Motion graphics and animationMotion graphics and animation– Sound: recording, editing, effectsSound: recording, editing, effects– Music synthesizer and sequencersMusic synthesizer and sequencers

Writing program for combination: advanced Writing program for combination: advanced programming skillsprogramming skillsAn authoring systemAn authoring systema software that permits a user to combine and a software that permits a user to combine and organize these media elements (example, Director)organize these media elements (example, Director)

Authoring SystemsAuthoring Systems

May be based onMay be based on– Layout model with markup languageLayout model with markup language– TimelineTimeline

Scripting languageScripting language

Multimedia ProductionMultimedia Production

A host of software tools and skills A host of software tools and skills

Require a teamRequire a team

Low end of multimedia software, web Low end of multimedia software, web page designpage design– Non-specialistsNon-specialists– Hide technical details such as HTML tags Hide technical details such as HTML tags

and JavaScript codeand JavaScript code– Wizards, assistants guide the production Wizards, assistants guide the production

processprocess

Multimedia PresentationMultimedia PresentationA unified interface and manipulated a single programA unified interface and manipulated a single program– Production: different software tools and team inputProduction: different software tools and team input

WWWWWW– Define a format (in markup language, HTML, XML) that can Define a format (in markup language, HTML, XML) that can

accommodate different media and view it using a dedicated accommodate different media and view it using a dedicated browserbrowser

Define an architecture (format) and API provides a rich set Define an architecture (format) and API provides a rich set of functions to manipulate data in that formatof functions to manipulate data in that format– QuickTimeQuickTime

Deliver in a stand alone form, needs no additional softwareDeliver in a stand alone form, needs no additional software– DirectorDirector

Three approaches are not necessarily separate or Three approaches are not necessarily separate or incompatibleincompatible– QuickTime movies in WWW pagesQuickTime movies in WWW pages– Director movies in Shockwave and embedded in WWW pagesDirector movies in Shockwave and embedded in WWW pages

Fig. 2.11Fig. 2.11

NetworksNetworks

TCP/IPTCP/IP

ISP, Internet Service ProvidersISP, Internet Service Providers

Dial-up connection, 56 kbpsDial-up connection, 56 kbps– V90 standard: downstream=56k, 33.6k in V90 standard: downstream=56k, 33.6k in

opposite directionopposite direction– 34-48 kbps are more realistic34-48 kbps are more realistic– Maximum bandwidth using analogue connectionsMaximum bandwidth using analogue connections

Digital telephone, ISDNDigital telephone, ISDN– A digital channel=64 kbpsA digital channel=64 kbps– Basic rate= two connections=128kbpsBasic rate= two connections=128kbps– Mid-1980Mid-1980

ADSLADSL– Downstream: up to 6.1 MbpsDownstream: up to 6.1 Mbps– Upstream: up to 640 kbpsUpstream: up to 640 kbps

Cable modemCable modem– 500 kbps to 10 Mbps500 kbps to 10 Mbps

T1=1.544 Mbps, T3=44.736 MbpsT1=1.544 Mbps, T3=44.736 Mbps

LANLAN– 10 base T Ethernet, 10 Mbps10 base T Ethernet, 10 Mbps– 100 base T, 100Mbps100 base T, 100Mbps

Table 2.1Table 2.1

NetworksNetworks

Clients and ServersClients and Servers

Server listen on a communication channel for Server listen on a communication channel for requests from clients.requests from clients.Server receives a request, it sends a response.Server receives a request, it sends a response.The requests and responses conform to a protThe requests and responses conform to a protocol.ocol.HTTP: HyperText Transfer ProtocolHTTP: HyperText Transfer ProtocolHTML: markup languageHTML: markup languageBrowser: Netscape, IEBrowser: Netscape, IEFTPFTPReal-time data streaming for audio and video Real-time data streaming for audio and video

Dynamically Generate Web PagesDynamically Generate Web Pages

Dynamically generate Web pages Dynamically generate Web pages from databasefrom database

CGI, common gateway interfaceCGI, common gateway interface

Microsoft, Active Server Pages, ASPMicrosoft, Active Server Pages, ASP

Apple, WebObjectsApple, WebObjects

MIME TypesMIME Types

OSOS– Extension of file’s nameExtension of file’s nameMultipurpose Internet Mail Extension Multipurpose Internet Mail Extension – Content-type: type/subtypeContent-type: type/subtype

Type: broad indication of the sort of dataType: broad indication of the sort of data– text, image, audio, videotext, image, audio, video– model for 3D model data such as VRMLmodel for 3D model data such as VRML– message: email message: email – application: binary data, e.g. application/postscriptapplication: binary data, e.g. application/postscript

Subtype: precise formatSubtype: precise format– HTML, GIF, AIFFHTML, GIF, AIFF– x-, experimental, video/x-msvideox-, experimental, video/x-msvideo

StandardsStandards

Define interfaces, file formats, markup languaDefine interfaces, file formats, markup languages, network protocols, and so onges, network protocols, and so onThree organizationThree organization– ISOISO– International Electrotechnical Commission (IEC)International Electrotechnical Commission (IEC)– International Telecommunication Union (ITU)International Telecommunication Union (ITU)

ISO, IEC: commercial companiesISO, IEC: commercial companies

CountriesCountries– ANSI: USAANSI: USA– BSI: UKBSI: UK– DIN: GermanyDIN: Germany

Rapidly changing environment of computers, Rapidly changing environment of computers, networks, multimedianetworks, multimedia– Standard often obsolete before they have passed tStandard often obsolete before they have passed t

hrough all the stages.hrough all the stages.Internet standardInternet standard– Semi-formalSemi-formal– Arpanet + NSFNET = TCP/IPArpanet + NSFNET = TCP/IP

Company productionCompany production– PostscriptPostscript– QuicktimeQuicktime– SVG (WSVG (W33C) < Flash C) < Flash