Digital Video Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2008 Last updated 2008. 8. 31.

Digital Video

Heejune AHNEmbedded Communications Laboratory

Seoul National Univ. of TechnologyFall 2008

Last updated 2008. 8. 31

Heejune AHN: Image and Video Compression p. 2

1. Video & Image

Video Projection of 3-D scene onto a 2-D Plane Illumination & Color information but No depth info

• Cf) 3-D Video, 2 D Stereo Vision Analog/continuous function: f (x, y, t)

Image Still Image f(x, y, t = t0)

Heejune AHN: Image and Video Compression p. 3

Digital Video Processing

Digital Video System Capture

• Sampling in Time Domain and Spatial Domain

• Photo sensor (e.g. CCD arrays)

• Analog vs Digital Camcoder/Camera Question: Why Digital ?

Capture

digitization

Processing, Storage, Transmission

sensor DSP/ASIC

display

Heejune AHN: Image and Video Compression p. 4

2. Video Sampling

Background Theory Nyquist Theorem

• We loose information over sampling rate (fs/2)

Human Visual System

Spatial Domain Sampling Resolution: N1 X N2

pel: Picture Element

2n2 1N

1 1N

0

0 1

1

2

2

1n 1 2,x n n

Heejune AHN: Image and Video Compression p. 5

Time Domain Sampling Frame rate

• Human Visual System (~30 Hz)

• Flickering Interlaced against Progressive

• Reduce Time Resolution without loss of spatial resolution

• Bad effect on Today’s Computer Displays

t

o

y

x

12

3

t + ts/2 (odd)

t + ts (even)

Heejune AHN: Image and Video Compression p. 6

Typical Examples for Resolution CIF 352x288 = 101376 pixels VHS qualitity SD 704x576 = 405504 pixels Broadcast quality HD 1440x1152 = 1313280 pixels HD quality

Typical Examples for Appearance Below 10 fps: Jerky 10 ~ 20 fps: Slow movement (video phone) is OK 20 ~ 30 fps: Movement is quite smooth 50 ~ 60 fps: Very Smooth/Fantastic

Heejune AHN: Image and Video Compression p. 7

3. Color spaces

RGB (red, green and blue) R, G, B has equal sensitivity for HVS e.g. 8 bits for R/G/B, 6/6/5 bits for R/G/B 24bits/pel

B G R

Heejune AHN: Image and Video Compression p. 8

YCrCb

YCrCb (YUV) HVS is more acute to Luminance (Y) Reduce Data by Reducing Coloring Resolution (e.g. YCrCb420) Color Transform

White

Yellow

Magenta

Black

Blue

Cyan

Red

Green

Heejune AHN: Image and Video Compression p. 9

YCbCr Transform Luminance as a weighted sum of R, G, B

• Y = 0.299R + 0.587G + 0.114B Cr, Cb, and Cg are different from Y.

• Cb = 0.564 (B-Y), Cr = 0.713 (R-Y), Cg = ??

• Note that Cr + Cb + Cg = constant

Heejune AHN: Image and Video Compression p. 10

Digital YCbCr

For easy signal processing• Non-negative Color and 128 center value

• 16<Y<235, 16<U, V< 240, Center = 128

Y U(Cb) V (Cr)

Heejune AHN: Image and Video Compression p. 11

Color Sampling (4:x:y) How many/where to sample Color components. x = the number of color samples in odd number line y = the number of color samples in even number line

주사 선12

3

4

4:4:4 4:2:2

4:1:1 4:2:0Y U, V

Heejune AHN: Image and Video Compression p. 12

Raw Image File Format Packed: pixel base packed

• easy for pixel unit processing

• (eg. display and capture) Planar: merged for each components

• easy for compression processing

rgb rgb rgb rgb rgb rgb rgb rgby y y y y y y y

rgb rgb rgb rgb rgb rgb rgb rgb

rgb rgb rgb rgb rgb rgb rgb rgb

rgb rgb rgb rgb rgb rgb rgb rgb

y y y y y y y yy y y y y y y yy y y y y y y y

u u u uu u u uv v v vv v v v

Heejune AHN: Image and Video Compression p. 13

4. HVS

Human Visual System Note that ultimate performance is evaluated by Human Signal Flow Anatomy

• Eye Lens => Retina => Optical nerve => Brain For HVS details, See Jain’s Fundamental of Image Processing

Heejune AHN: Image and Video Compression p. 14

동공의광학특성

광파 주파수인식특성

휘도인식특성

공간주파수인식특성

) , ,( yxL

빛에너지 휘도빛세기

luminancelightintensity

lightenergy

PerceivedLuminance(contrast)

brightness

인식휘도 ( 물체의 )밝기) , ,( yxI ) ,( yxg ) ,( yxc ) ,( yxb

) ,(1 ofH x

xf

LPF 특성( 망막착상 )

)(V

가시광 대역

violet red파장

c

g

) log( 21 aga

Logarithmic 인식특성( 휘도의 크기에 대한 )

)(H

22yx ff

LPF/BPF 특성( 휘도의 공간측변화에 대

한 )

Heejune AHN: Image and Video Compression p. 15

Important Features of HVS More sensitive to Luminance than Color More sensitive to High contrast than low Contrast (e.g. Edges) More sensitive to Low frequency than High More sensitive to Long duration than short Sampling time is around 20 ~ 30 Hz HVS response varies from individuals to individuals

Heejune AHN: Image and Video Compression p. 16

5. Video Quality

No Measure No Engineering Performance Measure

quality: how good similar to original signal?• Applied for lossy compression only

compression ratio : how much data volume is reduced? • Applied for both lossy and lossless compression

Engineering Trade-off between quality and compression ratio Given bandwidth (compression ratio), and optimize quality Given required quality, and minimize the bandwidth

Heejune AHN: Image and Video Compression p. 17

Subjective Quality Measure

Principle By human vision (like Olympic gymnastics) Individual Difference

• E.g. Professional and non-Professional has different MOS (mean Opinion Score)

DSCQS (Dual Stimulus Continuous Quality Scale) Two Image/Video is given at the same time, score them. Original and Processed are compared

SSCQS (Single Stimulus Continuous Quality Scale)

orignal

processing

display

Heejune AHN: Image and Video Compression p. 18

Objective Quality Measure

By Numerical Value SNR (Signal to Noise Ratio) and MSE (Mean Squared Error)

PSNR (Peak Signal to Noise Ratio)

E.g. HQ: 40 db = 20log10 (255/ESTD) => 255/ESTD = 100 => ESTD = 2.5 LQ : 20 db => 255/ESTD = 10 => ESTD = 25

1 2

1 2

1 12

1 2 1 20 01 2

1ˆMSE= , ,

N N

n n

x n n x n nN N

2

10

2 1PSNR=10log dB

MSE

B

Heejune AHN: Image and Video Compression p. 19

What PSNR for Left Image?

Heejune AHN: Image and Video Compression p. 20

6. Standard Image Format

Standard Resolution size

Heejune AHN: Image and Video Compression p. 21

7. Applications

Broadcasting Digital TV : Sattlite, Terrestrial, Cable Mobile TV: DMB, DVB-H etc

Telecommunication Video Phone Video Conference MMS (multimedia Messaging Service)

Game and Entertainment Video Storage : CD, DVD, Blue-Ray Natural Image Games

ETC E-Learning UCC Remote Medicine Surveillance Monitoring

Heejune AHN: Image and Video Compression p. 22

System Type Dedicated System

• STB, DVD player, PMP Player etc PC Mobile Embedded

• PDA, Handsets

Platform type Dedicated Hardware (ASIC) DSP Powerful General Purpose Processor

Heejune AHN: Image and Video Compression p. 23

PVR

Hard disk drive

MPEG-2 QualitiesBest 7.7 MbpsHigh 5.4 MbpsMedium 3.6 MbpsBasic 2.2 Mbps