Video Basics ---major ref. From Ch.5 of textbook 2lad.dsc.ufcg.edu.br/multimidia/intro.pdf · Video Basics---major ref. From Ch.5 of textbook 2 ... Many TV cameras and all CRT-based

1

1@NTUEE DSP/IC Lab

Video Basics ---major ref. From Ch.5 of textbook 2

■ Introduction ---- video industry

■ Video Imaging ---- video scan, aspect ratio

■ Color and Composite & component systems

■ From Analog To Digital Video

■ Spatial Conversions ---- video formats

■ Temporal Conversions

■ Mixing And Keying

2@NTUEE DSP/IC Lab

Cable Network

DVB-S downstream(max 90 Mbps)DSS

DVB-C downstream(max 40 Mbps)OpenCable

1. Satellite( fast PSTN/ ISDN2. Cable Modem ( QPSK, TCP / IP � for PSTN/ ISDN modem3. SDSL / ADSL / VDSL …..

IEEE 1394 / USBEthernet 10 Mbps…..

Satellite

Interaction Channel

Home Connection

Terrestrial (Plug&Play , high-data-rate)

(1-way / 2-way)

DTV set

DVB-T/ ATSC

DSTB

Cable Modem

DirecPC/DirecDuo

Video Environments

2

3@NTUEE DSP/IC Lab

Service Provision

TCP / IP

DVB-TDVB-S

(high speed BB)DVB-C

(Cable Modem)

HFC(Full Service

Network)

POTS(xDSL access)

Residential LAN(IR, RF, Wired)

Hybrid Services

DSTB

Wireless Cable(MMDS)

Video Service Environments

4@NTUEE DSP/IC Lab

�F���ãñìµ@�ûì��

�> �r

Gï=.1�� *<��ÎP�U½ÿ½��CD��*¶��1n�ñ�G�*ÐÍV PC;�^éu�u�*ñ<�uïÚí��� Internetw7�Home Spopping�Home Banking�….*PPV2âS�a�DO��"�H2<�_G.�(VOD)ÛÚí

�ö^éGï=.� *ö7Gï��A2Uf÷�� ß�[1n��Яr��n�1<t�*>�1ʺ=.²ÁÞ+Gï STB�w¯Gï=.1�Æ*"�Gï=.²2�òGï STBÞ�1äh����¼oZÐõ1"�2¤���Gï STB�

aöÞ^éGï=.�>� *� õ1n�<tñ)Ëà����Á�réï=é�C�4�*Gï�>h��Úü¶Êº=.�AÓ-�I FM�Medium Wave¤µÚí1�/R�

*Gï�>�Á��ä÷1 transmitters���Çt1ä÷*Gï�>�r�1�ñ²�

ô�<�Gï=.�>� *Î� BBCaGï=.Ú7ÂbÍÈzéú�r�p¾�câ> DTT�>*BDB2�£�<� 30! DTTn�Úí*­�~£��U�>;HÞr> HDTV/SDTV�>*BSkyB�~£ 6´[uSr> 200!n�Gïá#Úí*TCI�ComcastÛUÀMSOb� 1997£¦¬[�àGï CableÚí*Flextech$} BBC>Ë�1Gïn�(å� UKFM)

ö´^éGï=.�>� *1994£¦­�Gï DBSI 1996£¦[JGïá#r>*1997£¦Gï CableÚír>*1998£­�Î DTTr>�ʺ=.²ñhk��¶ 12ß 15£1´t

N�g�ñ·ë��æ�JUJT7�02::9

3

5@NTUEE DSP/IC Lab

¸®ñ

*Î�]]�XÇæ�

*ñ��'$7�Åg

*pÊTÊ�Åg

*PPV(Pay-TV)

*ÌFû1Dñ VOD/NVODìmultimedia- on-demandìKODìÌF �ìÌFm�ì5Fm� ��ì~7J�.....)

*Internetÿ�ñ e-mailì�½WWW...

*��³yìhome banking ......

x+�

*�ò�@(Video Telephone)

*x��ò�

*ò¦éÙ *......

Set-Top BoxRC

Applications of Digital Video

6@NTUEE DSP/IC Lab

Benefits of Digital Video

i) Open Architecture video systems, meaning the existence ofvideo at multiple spatial, temporal, and SNR resolutionswithin a single scalable bitstream

ii) Interactivityiii) Variable-rate transmission on demandiv) Easy software conversion from one standard to anotherv) Integration of various video applicationsvi) Editing capabilities, such as cutting and pasting, zooming

removal of noise and blurvii) Robustness to channel noise and ease of encryptionviii) …...

4

7@NTUEE DSP/IC Lab

N�g�ñ·ë��æ�JUJT7�02::9

�F�ò

5ï

5J9

ÌFû

8z(Satellite) ÝVm�(Terrestrial) 5Y(MMDS,MVDS,LMDS)

MPEG-2 MP@MLÿDigiCipher IÿII

MPEG-2 MP@MLîMPEG-2 MP@HL(HDTV)

MPEG-2 MP@MLÿDigiCipher II

QPSK

ë f

�FVSB(>�)ðOFDM(²)

�ïo�@�¸return pathY R���Òñ�

�ò��®æ�gû( EPG)ìVBIÕÃìVideo Info.2Ì��Nò1D

DirecTV/USSB,PrimeStar,EchoStar,

AlphaStar,ASkyB,PerfecTV’JSkyB,DirecTV(),²(Canal+, BSkyB,DT,FilmNet....),KoreaSat,M-Net(Ñ)....

Cross Country Cable of Riverside,

Tele-TV(Bell Atlantic,Nynex,Pacific

Telesis)......

QAM ÿ QPSKëÃo� Ý�ÍÅÕÃ

•�f‘98úäDTTÉ�

•%f‘98ú10��É�DTT

1D

Wireless Broadcast Video

8@NTUEE DSP/IC Lab

Descrambler OSD

Transport

NTSC/PALEncoder

CPU

MPEG-2Video

TrafficController

MPEG-2Audio

Digital Decoder Chip

Tuner

Tuner

Tuner

Satellite

Cable

Terrestrial

QPSKDemod

QAMDemod

COFDMDemod

FEC

FEC

FEC

SDRAM

Flash/EPROM

SDRAM VCXO

P1394 IR Buffer& Amp RS232 IrDA Port Modem

V-SeriesSmart CardBuffer P1284 SCSI

I2C

RFModulator

SECAMEncoder(optional)

DACs+ Filters

L

R

AV

Switch

RGB or Y/Cand CVBS

Graphics

AND/OR

ADC

AND/OR

IF

IF

QPSKMod

FEC

ADC

DualADC

Future Video Systems

5

9@NTUEE DSP/IC Lab

ÑfÝ���F��¨�

�n Gïá#(DST) Gï Cable(DCT) �ÿ�>(DTT)[Jz� DVB-S DVB-C DVB-T/COFDM­� DSS�DVB-S SCTE-031 ATSC/8-VSBã�U DVB-S --- ATSC/8-VSB²! DVB-S DVB-C DVB/COFDM

(6MHz)�� DVB-S --- ATSC/8-VSB¾J DVB-S --- DVB-Tm� DVB-S DVB-C ATSC/8-VSBU� DVB-S DVB-C ATSC/8-VSB

10@NTUEE DSP/IC Lab

EUWÇ�ÍS`

1998£ 2000 2002 2004 2006 2008 2010

rh

�>

�30!U��>

¯�Ê

ºn�

rh

�>

O�÷�50%́À��¬s¯�

ʺn�´Z

¯�ʺn�ß��

ôr>

d�

r>Uþ���

�ôr>

¾­�bÍ‘98£11́ 1²%¤r>DTV¾Î�bÍ‘98£11́ 15²%¤r>DTV (Ondigital/BDB)¾²!2Í2003£r>ISDB(6Audio�Video�Data)

¾­�bÍ‘98£11́ 1²%¤r>DTV¾Î�bÍ‘98£11́ 15²%¤r>DTV (Ondigital/BDB)¾²!2Í2003£r>ISDB(6Audio�Video�Data)

Cà2009£¯�NTSC

÷±�-�`�Ã=�¬IT ISà½�1998.11

6

11@NTUEE DSP/IC Lab

ÑfEUWÇ�Í

5 ë 1997 1998 1999 2000 2001 2002 2006

;�

*Y«`Ó1÷ *Y«dô�>(1´)

*^é2�`Ó�â

*â>�â>(6´) * ¯� NTSC

*rhçm¤ô��>(12´) n�

�u

*GsÓ-Q7(ATSC)

*â>�n�u½�n��»

*Y«�>=.�%%

Úí

*â>Úí *�þï�>

*DTTÚí VʺÚíu¡�> Úí

*ÑD=.Úí

*�}G�Úí /uïÚí

1997 1998 1999 2000 2001 2002 2006

÷±�-�`�Ã=�¬IT ISà½�1998.11

12@NTUEE DSP/IC Lab

Ñf�ò·g)TUC*�ͺÅ

�FCable STB �FSatellite STBC¹½YnO�¬Ì,»Ñ, C�...

CnO��·ì²�ì?¡ì ]òìI�ì 7µìâeìf2ì�æì»ÑìC�ì�¿ìeW

N�g�:·ë��æ�ITISYgJ�,1998/10

Understanding & Solutions/1998.8

02000

400060008000

100001200014000

1600018000

1997 1998 1999 2000 2001

0%

5%

10%

15%

20%

25%z�à�f

dú�ñf

z�p�ñþ¶÷ ¾‘98£DST STBþz�à�f

p93.4%

¾‘99£2¶�fDTT STBáâ

¾‘98£DST STBþz�à�f

p93.4%

¾‘99£2¶�fDTT STBáâ

Cr�7Û�7µìI�ì]òìâeì�·

C¹â�/�â�nO�¬Ìì²�ì]òì�æì?¡ì¦�ì6ØìyAìA�

�FTerrestrial STB

Aë�9F(CCL)SÏ� � ATSCÕÃtf

O�

37302465

167058090

7

13@NTUEE DSP/IC Lab

Microsoft; GI; S-A Group( IBM,Pioneer,PowerTV,Sun,Toshiba);Intel Group(Cisco,Netscape,NCI,Oracle,Thomson CE);Intel/NCI; Oracle/NCI/Netscape; Thomson CE/NCI; Cisco; Sony;Thomson Sun(OpenTV); Samsung; Pioneer Digital Technologies;ACTV/Sarnoff; IBM; Criterion Software; Lucent Technologies;SCM Microsystems; Zenith Electronics; Worldgate Communications;Texas Instruments; Wink Communications; Panasonic

OpenCable

MCNS-DOCSIS

PacketCable* the specs will establish a data packet network for interactive services(carry IP Internet traffic, telephony,video,teleconferencing)

*broadband Cable Modem (interoperability)

*next-generation digital STB(include MCNS CM)*Open architecture(RTOSs / APIs / microprocessors)*exiting open Internet specifications*'98.3 specified -- IEEE1394(1úÚ)

are moving Cable forward in the race to

provide consumer broadband services

3Com; 8x8; Ascend; Bay Networks; Cisco; eFusion; Ericsson;Fore Systems; IBM; Intel/Lucent Team; Inter-Tel; Linkon; MediatrixPeripherals; Motorola; NEC; Netspeak; Nortel; S-A; Siemens/Com21Team; Sony; Sun; Vienna

Vendors ParticipationTasks / ObjectsScheduleItem'96.7

'97.8�¨�ë�

'97.8�RFI'97.10�23 proposals'98.5�NCTAÍÄ ºø?7'98ä�±ÿY� Y�

'97.10 �es±Þ9-12�Ú?ÔnO�ÍIP voice & video over Cable

Follow�ADC,Bay Networks,3Com,Com21,Cisco,GI,Motorola,NEC,NetGame,Panasonic,Phasecom,S-A, Samsung,Sony, Terayon,Thomson CE,Toshiba……..

full 2-way ,IP capable networks

by the year 2000

÷±�-�CableLabs�`�Ã=�¬ITISà½/1998

�fDbcmfg�×F�¬¨�

14@NTUEE DSP/IC Lab

Video Imaging

Imaging: normal photography, X-rays, electronic documents,electronic still pictures, motion pictures, TV

Video: a sequence of still pictures of a scene taken atvarious subsequent intervals in time

Scanning: a form of sampling of a continuously varyingtwo-dimensional signal

8

15@NTUEE DSP/IC Lab

Video Resolution

Viewing Ratio (VR):VR = d /PH d= viewer’s distance from the screen

PH= picture height

Visible pixels = 3440 /VR Pixel (Pel) : the smallest detail that can be reproduced in a

picture is about the size of a picture element

Minimum Viewing Rates

System Line or pixels/PH VRNTSC television 483 7.2HDTV 1080 3.2Computer display 768 <4.5

16@NTUEE DSP/IC Lab

Refresh Rate and Flicker

Persistence of vision, causes the visual system responds slowly to rapid changes of illumination

Flicker: if the illumination varies cyclically at a low frequency, the eye may perceive an annoying effect

Refresh Rate:the light output of display devices decays after a short time, and thereforemust be refreshed periodically to maintain the effect of steady illumination

Display Refresh Rates

System Refresh rate (Hz) Environment VRNTSC television 60(59.94) Living room 7PAL television 50 Living room 7Computer display 72 Bright office 1-2Motion picture theater48 Dark room 5-10

9

17@NTUEE DSP/IC Lab

Video ImagingRaster Scan ----(Progressive)

18@NTUEE DSP/IC Lab

Video Imaging

Interlace Raster Scan---- it is the tradeoff of bandwidth, flicker, and resolution----- may cause flicker problems for text and graphics

10

19@NTUEE DSP/IC Lab

Image Aspect Ratio

IAR is defined as the ratio of picture width to heightIAR = w / h

PAR (pel aspect ratio) = IAR * vertical_size / horizontal_size : full screen display

where horizontal_size is the width of the image in terms of pelsand vertical_size is the height of the image in lines

ex. Display IAR = 4:3 Image size = 720 pels * 486 lines

PAR = 4/3 x 486/720 = 0.9

or IAR * display_vertical_size / display_horizontal_size : portion display

20@NTUEE DSP/IC Lab

11

21@NTUEE DSP/IC Lab

Gamma Correction

Many TV cameras and all CRT-based displays have a nonlinear relationship between signal voltage and light intensity. B = c v r + b B: light intensity c: gain factor

b: cutoff (camera) or black level (CRT) light intensityr: 1 ~ 3.0

To avoid gamma correction circuitry inside millions of TV receivers, gamma correction is doneprior to transmission. Ex. Assuming the gamma of camera to be 1 and that of the display to be2.2, then the camera voltage is raised to a power of 1/2.2 = 0.45.

22@NTUEE DSP/IC Lab

Human visual system

■ The retina consists of receptors sensitive to light calledphotoreceptors connected by nerve cells. Photoreceptors: Rodsand Cones, rods are responsible for low light vision, while conesare responsible for details and color under normal conditions, asin daylight.

12

23@NTUEE DSP/IC Lab

Primary Colors

Trichromatic theory of color:17 century, the perception of any color can be represented by just three variables

Additive color system (video system): RGB“additive” means they reproduce colors by adding colored light sources

Subtractive color system (painting and printing):magenta, cyan, and yellow (incorrectly) by RBYmodify the light reflected from white paper by absorbing (subtracting) certain colors

24@NTUEE DSP/IC Lab

Component Color Video SystemComposite System

In order to transmit or broadcast the analog video over a signal channel, there is aneed for composite color system that combine the three color components into asingle signal

Component SystemIn color video system, the devices must deliver three components to control the lightsources of display.

13

25@NTUEE DSP/IC Lab

Composite Color System

1. Luminance and Color-difference ComponentsY= 0.299R + 0.587G + 0.114BCr=R-YCb=B-Y

2. NTSC (59.94hz, 525 lines)

Y= 0.299R + 0.587G + 0.114B R= 1.0Y + 0.956I + 0.621QI= 0.596R - 0.274G - 0.322B G= 1.0Y - 0.272I - 0.649QQ = 0.211R - 0.523G + 0.311B B= 1.0Y -1.106I + 1.703Q

3. PAL (50hz, 625 lines)Y= 0.299R + 0.587G + 0.114BU = 0.492 ( B - Y )V = 0.877 ( R - Y )

4. SECAM (50hz, 625 lines)Y= 0.299R + 0.587G + 0.114BDb = -0.450R - 0.833G + 1.333BDr = -1.333R + 1.116G - 0.217B

*all RGB are gamma corrected.

26@NTUEE DSP/IC Lab

Summary of parameters used in worldwide color TV standards

Parameter NTSC PAL SECAMField rate (Hz) 59.94 50 50Lines per frame 525 625 625Gamma 2.2 2.8 2.8Audio Carrier (MHz) 4.5 QAM FMColor Subcarrier (MHz) 3.57 4.43 4.25(U)4.4(V)Color modulation method QAM QAM FMLuminance bandwidth(MHz) 4.2 5.0,5.5 6.0Chrominance bandwidth(Mhz) 1.3(I)0.6(Q) 1.3 1.0

14

27@NTUEE DSP/IC Lab

From Analog To Digital Video

■ Prefiltering (call prefilter prior to sampling)➤ Reduce unwanted excessive frequency and noise in the signal by

region filtering

■ Sampling➤ Sampling theorem (sampling frequency for NTSC, PAL,

CCIR601??)

■ Quantization

■ PCM (Pulse Code Modulation) Coding

28@NTUEE DSP/IC Lab

Video Formats

Picture format Luminance pixels Luminance linesCCIR601(525) 480 720CCIR601(625) 576 720SQCIF 128 96QCIF 176 144CIF1 352 2884CIF 704 57616CIF 1408 1152HDTV2 720 1280HDTV3 1080 1920

1: Common Intermediate Format2: at frame rates of 24, 30, 60 Hz progressive3: at frame rates of 24, 30 Hz progressive and 30Hz interlaced

15

29@NTUEE DSP/IC Lab

Other Commercial Image Structures

30@NTUEE DSP/IC Lab

Digital Video System

CCIR 601:Consultative Committee for International Radio (CCIR)--> International Telecommunications Union (ITU)Recommendation BT.601

4:4:4 Full-bandwidth sampling of R, G, B components4:2:2 Full-bandwidth Y, 2:1 horizontally-subsampled R-Y(U), B-Y(V)

4:1:1 Full-bandwidth Y, 4:1 horizontally-subsampled R-Y(U), B-Y(V)

4:2:0 Full-bandwidth Y, 2:1 horizontally-subsampled and vertically-subsampled R-Y(U), B-Y(V)

16

31@NTUEE DSP/IC Lab

Choice of video sampling frequency for ITU-R Rec.BT.601

525/59.94 625/50n Frequency n Frequency Dif.842 13248250 848 13250000 1750848 13342656 854 13343750 1094852 13405593 858 13406250 657856 13468530 862 13468750 220858 13499999 864 13500000 1860 13531467 866 13531250 217864 13594404 870 13593750 654868 13657341 874 13656250 1091

32@NTUEE DSP/IC Lab

From Analog To Digital Video(cont.)

■ The 4:2:0 format

■ The 4:2:2 format

■ The 4:4:4 format

MPEG-2

MPEG-1

17

33@NTUEE DSP/IC Lab

luminance and chrominance samples in a 4:4:4 video frame

34@NTUEE DSP/IC Lab

luminance and chrominance samples in a 4:2:2 video frame

18

35@NTUEE DSP/IC Lab

Interlaced and progressive representation of a 4:2:2 frame

Look like 2x1 array

36@NTUEE DSP/IC Lab

Luminance and chrominance samples in a 4:2:0 video frame

19

37@NTUEE DSP/IC Lab

Interlaced and progressive representation of a 4:2:0 frame

38@NTUEE DSP/IC Lab

CCIR-601 4:2:2 video frame

Spatial Conversions

20

39@NTUEE DSP/IC Lab

SIF, SIF-1, HHR and CCIR-601 4:2:0 video formats

576 for 25 frames/sec

SIF:source input format SIF-I: SIF interlaceHHR:half horizontal resolution

288 for 25 frames/sec

40@NTUEE DSP/IC Lab

■ CCIR-601 4:2:2➤ -- CCIR-601 4:2:0

21

41@NTUEE DSP/IC Lab

Spatial Conversion (cont.)

■ CCIR-601 4:2:2 -- SIF➤ 7-tap decimation filter [-29, 0, 88, 138, 88, 0, -29]/256 and➤ 4-tap interpolation filter [-12, 140, 140, -12]256 applied horizontally

-->(352)

42@NTUEE DSP/IC Lab

22

43@NTUEE DSP/IC Lab

Spatial Conversion (cont.)

■ CCIR-601 4:2:2 -- HHR

44@NTUEE DSP/IC Lab

Spatial Conversion (cont.)■ CCIR-601 4:2:0 -- SIF-I

➤ same horizontal fileter plus vertical subsampling with➤ [-4, 23, 109, 109, 23, -4]/256 filter and interpolation filter➤ [-2, 20, 110, 110, 20, -2]256

23

45@NTUEE DSP/IC Lab

Temporal Conversion■ Problems: 60 f/s video --> 72 progressive f/s monitor

24 f/s film --> 60 fields/s TV monitor 50 fields/s PAL --> 60 f/s NTSC ….all via MPEG-2

■ Interlaced to noninterlaced➤ 1. Line interpolation - [1/2, 1/2] interpolation filter applied between each pairs of

consecutive lines in each field➤ 2. Field merging - combines lines of two consecutive fields, not good for motion➤ 3. Frame based line interpolation - as follows

The case is from fieldrate to same frame rate,so called Deinterlacing orline-doubling

46@NTUEE DSP/IC Lab

Temporal Conversion (cont.)

■ Noninterlaced to interlaced: ex. 60 frames/s to 60 fields/s➤ 1. Line subsampling➤ 2. Line weighting with filter [1/4, 1/2, 1/4]

24

47@NTUEE DSP/IC Lab

Temporal Conversion (cont.)Field Rate Conversion

ex. Between NTSC and PAL

1. Convert 60 fields/s to 50 fields/s

i) Discardii) Temporal interpolationiii) Motion vector interpolation

2. Convert 50 fields/s to 60 fields/s

48@NTUEE DSP/IC Lab

Temporal Conversion (cont.)

■ Film And Interlaced Video Conversion➤ 24 frames/s => 60 fields/s -- 3:2 pull down

25

49@NTUEE DSP/IC Lab

Mixing And Keying

Mixing can be defined as the addition of two or more input videosignals, Video = α1Video1 + α2Video2 + … or

Video = α1Video1 + (1-α1 )Video2

Luma Keying (short for Luminance keying), a level of luminance signal is specified for the scene containing the

foreground such that all pels above that values are replaced by the corresponding pels of the scene containing the background.

k = 1 rh<rfg use backgroundk = 0 rfg<rl use foregroundk = (rfg - rl)/(rh - rl) else rl <= rfg <= rh use blended

50@NTUEE DSP/IC Lab

Chroma Keying

Chroma keying consists of substituting a background signal in place of the foreground signal based on some key color identified in the foreground video signal, known as “blue screen”.