Video and audio â€“ summary

Video and audio – summary

We will take a look at:• Video• Audio• Digitalization• TV-formats and standards• Human shortcomings• Compression• Coding• What features are interesting?

•

DD2471 (Lecture 04) Modern database systems & their applications Spring 2012 1 / 30

Video and audio

Video is the most challenging data type partly because it “contains” all other datatypes and partly because video data objects are big.Even highly compressed a one hour video (with good quality) occupies severalGB.

Also, it is important to distinguish between analogue and digital video/audio.Our senses work in an analogue fashion and thus we have to either showanalogue video/audio or trick the eye/ear into finding the data analogue, which infact is not all that hard.

Video may contain images or analogue pixel streams and also audio and text sowe may have to apply a number of techniques for a good result.

•


Video and audio




•


Video and audio




•


Video and audio . . .

To turn analogue video/audio into digital video/audio we must convert the signaland at play back we must re-convert the signal (or fool the eye/ear).If we record analogue signals, we have no need for AD/DA-conversion.

Still, the normal case is to convert ???

There is not only the AD/DA-conversion. In a media database we want to extractinteresting features for indexing and querying.And, in addition, storage media is not at all as good for analogue storage as it isfor digital storage.

•






•






•



Digital media has a universal representation and all digital media may be treatedthe same way (even if encoding is not the same and conversion may benecessary)

Digital signals are more compact both when storing them and when transmittingthem and they may easily be managed with computers

In the production stage, analogue signal quality suffer at each step in themanagement while digital signals have one initial quality loss (when converting todigital signals) and one final when converting back to analogue (which as hasbeen noted, not always must be performed)

The final total quality loss is much lower with digital than with analogue signals•




















Against digital media

The coding introduces distorsion

Complicated decoding

Sampled media has to be digitized in two steps

Each step introduces distorsion

More advanced techniques demand more storage

More bits per pixel demands more storage

It can never match the original

Partial semantic loss in the digitizing step•








































































For digital media

Technical development is rapid – giving better and better results

Compression techniques evolve faster than digitizing techniques. Thus theincrease is far from linear.

Vector graphics make digitizing more exact and less bulky

Symbolic representation (like vector graphics) preserve semantics

Symbolic representation is not as distorsion prone as pixel based (thus betterinitial digitalization)

May be compressed to a higher degree than the pixel based counterpart•


For digital media








For digital media








For digital media








For digital media








For digital media








New standards

XML-based standards, like Scalable Vector Graphics (SVG) give more compactimages that scale (up and down) with maintained quality.

May be modified by software . . .

XML is not that compact so compression is needed anyway.•


New standards





New standards





Technologies and formats

There are many formats and technologies, each having advantages anddisadvantages

PAL EuropeSECAM FranceNTSC USA/Canada/Japan

NTSC was first.

SECAM was originally NTSC with better resolution

PAL is NTSC + feedback for colour compensation by correlation (delay and coloursmoothing)

•





NTSC was first.



•





NTSC was first.



•





NTSC was first.



•





NTSC was first.



•


N T S C

National Television System Committee

Lines/Field 525/60Horizontal Frequency 15.734 kHzVertical Frequency 60 HzColour Sub carrier Frequency 3.579545 MHzVideo Bandwidth 4.2 MHzSound Carrier 4.5 MHz

•


P A L

Phase Alternating Line

SYSTEM PAL PAL N PAL MLine/Field 625/50 625/50 525/60Horizontal Freq. 15.625 kHz 15.625 kHz 15.750 kHzVertical Freq. 50 Hz 50 Hz 60 HzColour Sub Carrier 4.433618 MHz 3.582056 MHz 3.575611 MHzVideo Bandwidth 5.0 MHz 4.2 MHz 4.2 MHzSound Carrier 5.5 MHz 4.5 MHz 4.5 MHz

•


S E C A M

Sequential Couleur Avec Memoire

Sequential Colour with Memory

SYSTEM SECAM B,G,H SECAM D,K,K1,LLine/Field 625/50 625/50Horizontal Frequency 15.625 kHz 15.625 kHzVertical Frequency 50 Hz 50 HzVideo Bandwidth 5.0 MHz 6.0 MHzSound Carrier 5.5 MHz 6.5 MHz

•


Digital formats

DV-band miniDV, . . .CDROM QT, AVI, MPEG (I), . . .DVD MPEG II, MPEG N, . . .Internet server QT, AVI, . . .Stream server QT(stream), RA, ASF

•


Comparison

In TV you see only part of the image (edges are cut off)

In a computer screen everything is shown

In (PAL) TV the image is drawn in two steps, first all odd lines and then all evenlines. No flicker at 25 images per sec

The computer screen images is drawn in one sweep (progressive scan)

In TV there are no alternatives

In a computer connection bandwidth must be taken into consideration•


Comparison








Comparison








Comparison








Human shortcomings

The brain is the most sensitive for changes in contrast at about 10 Hz

At 25 Hz we don’t note single small changes. It takes big changes . . . . . .

Spatial changes are noted at up to 12 Hz per angular degree (≈ 0.00875 radiansper image in a PAL setting)

Sensitivity increases with lower frequency and gets really irritating at 4 HZ /angular degree

“From far away we dont notice the stripes”. “At dawn all cats are grey”. . . . . . .

Huge benefits (in terms of coding) if we are just outside of the limits. Contrastperception gives us image rate and spatial perception resolution. At normal (?)viewing distance PAL is more than just adequate.

•


Human shortcomings







•


Human shortcomings







•


Human shortcomings







•


Human shortcomings







•


Human shortcomings







•


Human shortcomings . . .

NTSC is a smart invention. We see colour distortion only if it is considerable (?)and under special circumstances.

SECAM is a french overkill . . .

As we notice changes in intensity easier than colour distortion it is better to usefewer bits for colour and more bits for intensity information

•






•






•


Classification

HDTV - DTV - TV - VCR - Video conference

HDTV give us high resolution (to what benefit??)

TV = today’s mainstream technology (more than adequate)

DTV = tomorrows technology, same resolution as today, less noise

VCR = TV / 2 – in terms of quality

Video conference = TV / 8 + fewer images per sec (we have seen Steve Jobswalking like a robot with plastic facial expressions on Apples live broadcasts overthe internet)

But the human brain uses far more resources trying to classify “news” as “old” andrecognizable than to understand that it is in fact news.

•


Classification








•


Classification








•


Classification








•


Classification








•


Classification








•


Classification








•


Compression

MPEG = Moving Picture Experts Group

MPEG(1) – OK for VCR quality (≈ 1,5 Mbps)

MPEG(2) – DVD / HDTV but requires ≈ 40 Mbps so data is buffered and startis delayed

MPEG3 – never saw the light of day

MPEG4 – video format for limited bandwidths

MPEG7 – XML standard for MPEG (many different parts)

•


Compression







•


Compression







•


Compression







•


Compression







•


Compression







•


Video/audio→ digital media

Digitizing by sampling(earlier RGB coding – now YUV more common)

⇓

Compression over time

⇓

Compression in the image plane

⇓

Coding (entropy-coding)

•




⇓


⇓


⇓


•




⇓


⇓


⇓


•




⇓


⇓


⇓


•


Coding principles

RGB = red – green – blue

old standard

YUV = Y (luminance) + UV (chrominance in two dimensions)

UV can be calculated from RGB and we need the luminance in any case

YUV coding is more compact than RGB and faster to decode.•


Coding principles


old standard





Coding principles


old standard





Coding principles


old standard





Compression principles

Compression algorithms are smart.

Compression over time is based on the fact that not much differ from one image toanother so you don’t resend the whole image but rather information aboutchanges and advanced predictions and error compensations.

In the image plane you compress using Fourier transforms. As discretization hasalready been performed in the digitizing step you only apply a cosine transform toget a “frequency plane image”.

Then you cut off frequencies that the human eye cannot perceive•














Entropy codingClaude Shannon defined in 1948 the entropy of an information source generatingsymbols based on the probability that a symbol occurs as

H = −∑

i

pi log pi

It was defined out of the need to understand how you have to dimension thecapacity of a communication channel (roughly its bandwidth) and is based onstochastic probabilities.It gives a good measure on how many bits that will be necessary for the codingand combined with Huffman coding it is possible to minimize coding length forinformation storage and transfer.

It also makes it possible to differentiate between requirements for differentscenarios. A quite day on a summer meadows will require much less bandwidththan an action movie

•



H = −∑

i

pi log pi



•



H = −∑

i

pi log pi



•



H = −∑

i

pi log pi



•


Interesting features

In media databases we index even meta-data. It is more common to query basedon meta-data than on the data itself.

Features that are interesting for one type of media may be interesting for othermedia as well. Indexes may cover data of several types

In addition we are interested in recurring patterns in the features/attributes.Is the background red(-dish) on many pictures?How many pictures in a sequence had a mostly red background?

Sound volume – a persons tone of voice – background noise

Voice recognition is complicated and challenging both in video and audio•






























Interesting features in speech

Number of identifiable words

Number of speakers

Vocabulary size

Grammar

From the flow of words (sometimes more than one flow) it is interesting to extractnames, places, recognizable idioms, dates (including time specs), organizations(if people can be identified as belonging to an . . . )

There is (according to rumours) speech and video QBE where you may stateelements of a story and get “similar” stories fetched from a database

•




Number of speakers

Vocabulary size

Grammar



•




Number of speakers

Vocabulary size

Grammar



•


Other interesting features

I already mentioned that media data is spatial and temporal in essence, meaningthat knowledge of such systems is an advantage (GIS e.g.)

Temporal aspects deal with time. Not only with exact time specifications but alsotime in relation to events or facts.Was there a house before the road was built?Was the place excavated for archaeological remains prior to erecting the newbuildings?Did house A disappear before house B?. . .

Spacial aspects deal with absolute location as well as relative location.Are the houses along-side road E18 within noise limits for the railroad and / or thefree-way?What areas must be evacuated in case of a tsunami alert?Which rescue stations are within 15 minutes from an accident?

◦






◦






◦


Some notes on persistence

Persistence may be achieved in several ways,

• as an extension to an existing language, not necessarily object oriented,• by using a specially dedicated database programming language or• by extending traditional database management systems by including a

programming language. •




• as an extension to an existing language, not necessarily object oriented,

• by using a specially dedicated database programming language or• by extending traditional database management systems by including a





• as an extension to an existing language, not necessarily object oriented,• by using a specially dedicated database programming language or

• by extending traditional database management systems by including aprogramming language. •




• as an extension to an existing language, not necessarily object oriented,• by using a specially dedicated database programming language or• by extending traditional database management systems by including a



Some notes on persistence . . .

The purpose of persistence is to guarantee secure data storage regardless of thelifespan of the data. The following is a subjective classification:

• Transient data, data that need to be temporarily stored during the evaluation ofexpressions

• Local variables that occur when activating procedures and functions

• Variables that are declared in procedure and function bodies

• Global and/or dynamic variables (data that exist during program runtime)

which are the traditional persistence classes and

• Data that exist as long as a computer program exists

• Data that survive several program versions

• Data that survive “for ever”

which are the new persistence classes, in which data lifespan no longer is determined by asingle program activation. •



The purpose of persistence is to guarantee secure data storage regardless of thelifespan of the data. The following is a subjective classification:• Transient data, data that need to be temporarily stored during the evaluation of

expressions












expressions












expressions












expressions












expressions




which are the traditional persistence classes

and








expressions












expressions












expressions












expressions










Some notes on persistence – requirements and principles

• Orthogonal persistence No data “discrimination”.

• Program must execute in the same “way” regardless of the lifespan of its data. . . •










Some notes on persistence – benefits

• Simpler semantics→ increased productivity

• Without persistence people often use “ad hoc”-solutions to cover the need for valuesthat persist and for transactions.

• Type checks may be performed globally instead of just within an application.

• Simpler to uphold referential integrity as there is a relation to values outside of theapplication.

• Conceptually simpler with one mapping instead of three.

db connection

3GL−world mapping

3GL−world mapping

application

application

DB DB−model

the world

the world

•








db connection

3GL−world mapping

3GL−world mapping

application

application

DB DB−model

the world

the world

•








db connection

3GL−world mapping

3GL−world mapping

application

application

DB DB−model

the world

the world

•








db connection

3GL−world mapping

3GL−world mapping

application

application

DB DB−model

the world

the world

•








db connection

3GL−world mapping

3GL−world mapping

application

application

DB DB−model

the world

the world

•








db connection

3GL−world mapping

3GL−world mapping

application

application

DB DB−model

the world

the world

•


Some notes on persistent object memoryObjects that are to persist may be identified in 3 different ways:

• All data are persistent. Simplest. A specific data area contains all variables (possibly the wholeprogram memory) and is saved at program termination.At restart it is read into primary memory.Known as “core dumping”.Drawbacks: The data area (or the whole program) is managed as one unit.

• Only explicitly marked data are persistent. All transfers between program variables andpersistent memory are explicit.Persistent variables are declared as “dynamic”, “persistent”, . . .Data is stored with commands like “export” and fetched with “import”.Exported values become persistent when “checkpoints” are executed.Drawbacks: You work with copies of persistent data.

• Data that can be reached from a “persistent roots are persistent. All data is organised as objectscontaining references to other persistent objects.Objects become persistent when changes are discovered or a “commit” is executed.Data is retrieved when referenced.

Transaction system necessary to guarantee correct updates of persistent data.Object identity is mandatory (naming or by reference).

Object complexity may vary from simple values to complex data structures. •



























Transaction system necessary to guarantee correct updates of persistent data.

Object identity is mandatory (naming or by reference).

















Languages for persistent programmingmostly historical

PS-Algol:I Orthogonal persistenceI Both static and dynamic bindingI Procedures are first class valuesI Reflection

Napier:I Built on PS-Algol (has all PS-Algol features)+I Polymorphism and abstract data typesI environments are first class values!!

O++:I Object oriented, built on C++I All values are first class values

Pjama:I Object oriented, extension to JavaI Orthogonal persistence

•







•







•







•


Video and audio â€“ summary

Documents