Image and Video Coding

Image Processing and Computer Vision: 9 1

Image and Video Coding

Compressing data to a smaller volume without losing (too much) information

Image Processing and Computer Vision: 9 3

Why Code Data? To reduce storage volume To reduce transmission time

One colour image 760 by 580 pixels 3 channels, each 8 bits 1.3 Mbyte

Video data Same resolution 25 frames per second 33 Mbyte/second

Image Processing and Computer Vision: 9 4

Redundancy Spatial

Correlation between adjacent pixels Chromatic

Correlation between colour channels Temporal

Correlation between adjacent frames Perceptual

Unnoticed losses

Image Processing and Computer Vision: 9 5

Example

Image Processing and Computer Vision: 9 6

Redundancy - Consequences

Data exceeds information More data than content justifies

Can lose data without losing information

Image Processing and Computer Vision: 9 7

Compression Ratio

image compressed of volume

image eduncompress of volumeC

Image Processing and Computer Vision: 9 8

Lossy vs Lossless Compression Can lose data without losing

information Lossily compressed images can

look similar to the original Lossy compression has greater C

Image Processing and Computer Vision: 9 9

Quality of Decoded Images Measure differences between

Original Coded/decoded images

Options Maximum difference Average difference Subjective scales

Image Processing and Computer Vision: 9 10

Example

Image Processing and Computer Vision: 9 11

Subjective Quality Measurement Scales - Example

0 Unusable1 Annoying degradation2 Adequate images3 Barely perceptible degradation4 No observable degradation

Image Processing and Computer Vision: 9 12

Difference coding Adjacent pixels are similar

Difference is small

Uncompressed:

Compressed:

,,,,,43210 sssss

,,,,,3 42 3210 10 sssssssss

Image Processing and Computer Vision: 9 13

Coding Assign 4 bits to a difference Can code –7, … , +8

Overflow? Use –7 and +8 to show larger differences Code –6, … , +7 directly Use overflow codes to indicate shift of

codes

Image Processing and Computer Vision: 9 14

Image Processing and Computer Vision: 9 15

Predictive Coding If signals are well sampled

Adjacent samples are correlated They have similar values Differences are small Can guess next sample from value of

current

Image Processing and Computer Vision: 9 16

Constants are correlation coefficient and mean grey value

Difference between real and predicted values are smaller

Code as for difference coding

II ttˆ 1

Image Processing and Computer Vision: 9 17

Run Length Coding Replace runs of equal brightness values

by (length of run, value)

1 2 2 3 3 4 4 4 5 6 5(1 1) (2 2) (2 3) (3 4) (1 5) (1 6) (1 5)

More use when few brightness values e.g. fax

Image Processing and Computer Vision: 9 18

Huffman Coding Uses variable length codes Most frequently occurring grey

value has shortest code Least frequently occurring values

have longest codes

Image Processing and Computer Vision: 9 19

ExampleSymbol Probability 21 43

ABCDEF

0.40.30.10.10.060.04

0.40.30.10.10.1

0.40.30.20.1

0.40.30.3

0.60.4

Image Processing and Computer Vision: 9 20

SymbolProbability 21 43

ABCDEF

0.4 10.3 000.1 0110.1 01000.06 010100.04 01011

0.4 10.3 000.1 0110.1 01000.1 0101

0.4 10.3 000.2 0100.1 011

0.4 10.3 000.3 01

0.6 00.4 1

Image Processing and Computer Vision: 9 21

GIF Applicable to images with 256

colours Replace sequences of bytes with

shorter codes Most common sequences use

shortest codes

Image Processing and Computer Vision: 9 22

Wavelet coders Wavelet transform organises

image content efficiently Can easily select data to be

discarded

Image Processing and Computer Vision: 9 23

Wavelet coders

Image Processing and Computer Vision: 9 24

JPEG standard A subcommittee of ISO Optimised for a range of image

subject matter Compression rates can be defined Quality inversely proportional to C

Image Processing and Computer Vision: 9 25

Block Transform Codingoriginal image

decompose

transform

quantise

entropy code

Sequence of 8 by 8 blocks - different planes treated separately (RGB, YUV etc.)

Transformed blocks reduce redundancy and concentrate signal energy into a few coefficients discrete cosine transformation (DCT)

Blocks with discarded information - goal is to smooth picture and discard information that will not be missed, e.g. high frequencies

Image Processing and Computer Vision: 9 26

Block Transform Encoding

DCT

Zig-zag

run length code entropy

code

quantise

010111000111…..

Image Processing and Computer Vision: 9 27

Block Encoding

139 144 149 153144 151 153 156150 155 160 163159 161 162 160

1260 -1 -12 -5 -23 -17 -6 -3 -11 -9 -2 2 -7 -2 0 1

DCT

Zig-zag

run length code

Huffman code

quantise

10011011100011….

79 0 -1 0-2 -1 0 0-1 -1 0 0 0 0 0 0

79 0 -2 -1 -1 -1 0 0 -1 0 0 0 0 0 0 0

0 791 -20 -10 -10 -12 -10 0

Original image

Image Processing and Computer Vision: 9 28

Block Transform Decoding

DCT

Zig-zag

run length code entropy

code

quantise

010111000111…..

Image Processing and Computer Vision: 9 29

Result of Coding and Decoding

139 144 149 153144 151 153 156150 155 160 163159 161 162 160

Original block

144 146 149 152148 150 152 154155 156 157 158160 161 161 162

Reconstructed block

-5 -2 0 1-4 1 1 2-5 -1 3 5-1 0 1 -2

errors

Image Processing and Computer Vision: 9 30

Discrete Cosine Transform

F[u,v] =4C(u)C(v)

n2

n-1

j=0

n-1

k=0

f(j,k) cos(2j+1) u

2ncos

(2k+1) v 2n

with

C(w) =

12

For w = 0

1 otherwise

Image Processing and Computer Vision: 9 31

Discussion Where are lossy steps?

Quatisation and subsampling before coding

How is quantisation matrix chosen? Its predefined by the standard after

much experimentation

Image Processing and Computer Vision: 9 32

Video coding Many specific standards

AVS, MOV, QT, … One universal standard

MPEG

Image Processing and Computer Vision: 9 33

MPEG Standards Standard specifies audio, video

and system layers MPEG-1: low data rates, poor

quality: VHS quality at 1.5Mbits-1

MPEG-2: high quality hence high data rates: studio quality, 15Mbits-1

MPEG-4: low data rates, small images, 64 kbits-1

Image Processing and Computer Vision: 9 34

MPEG-1 Audio and video designed to work

at CD ROM speeds: 1.5Mbits-1

Video 1.150Mbits-1

Audio 0.256Mbits-1

System 0.094Mbits-1

Image Processing and Computer Vision: 9 35

MPEG-2 Released in 1994 Aimed at digital TV, ATM. Additions for

Interlaced video Scalable video coding Graceful degradation with noise

Implementation of full standard impractical Varying profiles/levels of conformity

Image Processing and Computer Vision: 9 36

MPEG-4 Coding specifically for multimedia

objects

Image Processing and Computer Vision: 9 37

Coding Algorithms Frame sequence Motion compensation Frame coding

Image Processing and Computer Vision: 9 38

Frame Sequence I frames (Intraframes)

Coded independently of any other frame P frames (Predicted frames)

Derived from previous I frame by motion prediction

B frames (Bidirectionally interpolated) Interpolate motion compensated blocks

between I and P frames

Image Processing and Computer Vision: 9 39

Motion Compensation Image is divided into macroblocks

(16 x 16 pixels) Matching macroblocks are found

by minimising differences Code differences and macroblock

displacement

Image Processing and Computer Vision: 9 40

Frame Coding Use JPEG algorithms

Image Processing and Computer Vision: 9 41

Summary Why code data? Redundancy Assessment of compression Lossy vs. lossless compression Algorithms

JPEG, MPEG

Image Processing and Computer Vision: 9 42

But what … is it good for?Engineer at the Advanced Computing

Systems Division of IBM, commenting on the microchip in 1968