Basics of Video Courtesy of Professor Yao Wang Polytechnic University, Brooklyn, NY11201 yao@vision.poly.edu.

Basics of Video

Courtesy of Professor Yao WangPolytechnic University, Brooklyn, NY11201

yao@vision.poly.edu

Adapted from Yao Wang, 2004

Video Basics

Outline

• Video capture– Photometric model– Geometric model

• Analog video– Progressive vs. interlaced rasters in analog TV system– Different color representations: YUV/YIQ

• Digital video– Sampling/quantization– Y’CbCr format

• Video display– Spatial/temporal/bit-depth resolution

Adapted from Yao Wang, 2004

Video Basics 3

Photometric Model of Video Capture

Courtesy of Onur Guleryuz

Adapted from Yao Wang, 2004

Video Basics 4

Geometric Model of Video Capture

Cameracenter

Imageplane

2-Dimage

3-Dpoint

The image of an object is reversed from its 3-D position. The object appears smaller when it is farther away.

Z

YFy

Z

XFx ,

Adapted from Yao Wang, 2004

Video Basics 5

Implication of Models in Analog World

Miniature building Explosion from The Mummy Lighting in Filmmaking

Adapted from Yao Wang, 2004

Video Basics 6

Progressive and Interlaced Raster Scans

Field 1 Field 2

Progressive Frame Interlaced Frame

Interlaced scan is developed to provide a trade-off between temporal and vertical resolution, for a given, fixed data rate (number of line/sec).

Horizontal retrace

Vertical retrace

Adapted from Yao Wang, 2004

Video Basics 7

Color TV Broadcasting and Receiving

Lum ina nc e ,C hro m ina nc e ,Aud io M ultip le xing

M o d ula tio n

De -M o d ula tio n

De -M ultip le xing

YC 1C 2--->RG B

RG B--->

YC 1C 2

Adapted from Yao Wang, 2004

Video Basics 8

Why not using RGB directly?

• R,G,B components are correlated– Transmitting R,G,B components separately is redundant– More efficient use of bandwidth is desired

• RGB->YC1C2 transformation– Decorrelating: Y,C1,C2 are uncorrelated– C1 and C2 require lower bandwidth– Y (luminance) component can be received by B/W TV sets

Color transformation is a compromised solution, but the ultimate one

Adapted from Yao Wang, 2004

Video Basics 9

YIQ in NTSC

• I (in-phase): orange-to-cyan

• Q (quadrature): green-to-purple (human eye is less sensitive)– Q can be further bandlimited than I

• Phase=Arctan(Q/I) = hue, Magnitude=sqrt (I^2+Q^2) = saturation

• Hue is better retained than saturation

Recall: Quadrature amplitude modulation (QAM) in digital communication

Color Image Y image

I image (orange-cyan) Q image (green-purple)

Adapted from Yao Wang, 2004

Video Basics 11

I and Q on the color circle

I: orange-cyan

Q: green-purple

Adapted from Yao Wang, 2004

Video Basics 12

Conversion between RGB and YIQ

Y = 0.299 R + 0.587 G + 0.114 BI = 0.596 R -0.275 G -0.321 BQ = 0.212 R -0.523 G + 0.311 B

R =1.0 Y + 0.956 I + 0.620 Q,G = 1.0 Y - 0.272 I -0.647 Q,B =1.0 Y -1.108 I + 1.700 Q.

• RGB -> YIQ

• YIQ -> RGB

Adapted from Yao Wang, 2004

Video Basics 13

YUV in PAL

Adapted from Yao Wang, 2004

Video Basics 14

YUV/RGB Conversion

Y = (( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16)U = ( ( -38 * R - 74 * G + 112 * B + 128) >> 8) + 128V = ( ( 112 * R - 94 * G - 18 * B + 128) >> 8) + 128

Numerical approximations

Adapted from Yao Wang, 2004

Video Basics 15

YIQ/YUV Comparison

Adapted from Yao Wang, 2004

Video Basics 16

Different Color TV Systems

Parameters NTSC PAL SECAM

Field Rate (Hz) 59.95 (60) 50 50

Line Number/Frame 525 625 625

Line Rate (Line/s) 15,750 15,625 15,625

Color Coordinate YIQ YUV YDbDr

Luminance Bandwidth (MHz) 4.2 5.0/5.5 6.0

Chrominance Bandwidth (MHz) 1.5(I)/0.5(Q) 1.3(U,V) 1.0 (U,V)

Color Subcarrier (MHz) 3.58 4.43 4.25(Db),4.41(Dr)

Color Modulation QAM QAM FM

Audio Subcarrier 4.5 5.5/6.0 6.5

Total Bandwidth (MHz) 6.0 7.0/8.0 8.0

Adapted from Yao Wang, 2004

Video Basics 17

Who uses what?

From http://www.stjarnhimlen.se/tv/tv.html#worldwide_0

Adapted from Yao Wang, 2004

Video Basics 18

Digital Video

Taken from EE465: Image Acquisition

Sampling Quantization

Adapted from Yao Wang, 2004

Video Basics 19

BT.601* Video Format

480

line

s

525

line

s

12 2p e l

16p e l

85 8 p e ls

72 0 p e ls

Ac tiveAre a

52 5/60 : 60 fie ld /s57

6 lin

es

625

line

s

86 4 p e ls

13 2p e l

12p e l

72 0 p e ls

Ac tiveAre a

62 5/50 : 50 fie ld /s

* BT.601 is formerly known as CCIR601

Adapted from Yao Wang, 2004

Video Basics 20

RGB <--> Y’CbCr

Analog video

Digital video

Adapted from Yao Wang, 2004

Video Basics 21

YUV vs. Y’CbCr

Adapted from Yao Wang, 2004

Video Basics 22

Chrominance Subsampling Formats

4 :2 :0F o r ev er y 2x 2 Y P ix els

1 C b & 1 C r P ix el(S u b s am p l in g b y 2 :1 b o thh o r i zo n ta l l y an d v e r t i c al l y )

4:2:2 Fo r ev er y 2x 2 Y P ix e ls

2 C b & 2 C r P ix e l(S u b s am p lin g b y 2:1

h o r izo n tal l y o n ly )

4 :4:4 Fo r ev er y 2x 2 Y P ix e ls

4 C b & 4 C r P ix e l(N o s u b s am p l in g )

Y P ix el C b an d C r P ix el

4:1:1Fo r ev er y 4x 1 Y P ix e ls

1 C b & 1 C r P ix e l(S u b s am p lin g b y 4:1

h o r izo n tal ly o n ly )

Adapted from Yao Wang, 2004

Video Basics 23

Digital Video Formats

Video Format Y Size Color Sampling

Frame Rate (Hz)

Raw Data Rate (Mbps)

HDTV Over air. cable, satellite, MPEG2 video, 20-45 Mbps SMPTE296M 1280x720 4:2:0 24P/30P/60P 265/332/664 SMPTE295M 1920x1080 4:2:0 24P/30P/60I 597/746/746 Video production, MPEG2, 15-50 Mbps BT.601 720x480/576 4:4:4 60I/50I 249 BT.601 720x480/576 4:2:2 60I/50I 166 High quality video distribution (DVD, SDTV), MPEG2, 4-10 Mbps BT.601 720x480/576 4:2:0 60I/50I 124 Intermediate quality video distribution (VCD, WWW), MPEG1, 1.5 Mbps SIF 352x240/288 4:2:0 30P/25P 30 Video conferencing over ISDN/Internet, H.261/H.263, 128-384 Kbps CIF 352x288 4:2:0 30P 37 Video telephony over wired/wireless modem, H.263, 20-64 Kbps QCIF 176x144 4:2:0 30P 9.1

Adapted from Yao Wang, 2004

Video Basics 24

4:2:0 YUV Video

Y: 288-by-352

U: 144-by-176 V: 144-by-176

Adapted from Yao Wang, 2004

Video Basics 25

Tricky Photometric Situations

Shadow causes problem to background extraction

Video enhancement

Adapted from Yao Wang, 2004

Video Basics 26

Geometric Invariance

Adapted from Yao Wang, 2004

Video Basics 27

Video Display

• High-end– If the resolution of display device is higher than that of video

sequence, what can we do?

– Tradeoff between quality and complexity

– Subjective evaluation of video quality

• Low-end– If the resolution of display device is lower than that of video

sequence, what can we do?

– What if the bit-depth resolution is lower? (e.g., display video on PDAs and portable DVDs)

It is the last and the least-researched component in visual communication systems

Adapted from Yao Wang, 2004

Video Basics 28

Resolution, Resolution, Resolution

spatial

temporal

Bit-depth

1M 10M

30fps

300fps

8bpp

32bpp

Adapted from Yao Wang, 2004

Video Basics 29

High Dynamic Range Imaging

Q: Can we generate a HDR image (16bpp) by a standard camera?A: Yes, adjust the exposure and fuse multiple LDR images together

Adapted from Yao Wang, 2004

Video Basics 30

HDR Display (after Toner Mapping)

Note that any commercial display devices we see these days are NOT HDR