8/13/2019 basic lecture information theory and coding
1/68
Dr. M. Arif Wahla
EE Dept
Military College of Signals
National University of Sciences & Technology (NUST), PakistanClass webpage:
http://learn.mcs.edu.pk/course/view.php?id=544
Information & Coding Theory
Course Outline/ Introduction
8/13/2019 basic lecture information theory and coding
2/68
Founded by Claude E. Shannon (1916-2001)
The Mathematical Theory of Communication, 1948
Study fundamental limits in communications: transmission, storage,
etc
1:31 AM Course Outline 2
Information Theory
8/13/2019 basic lecture information theory and coding
3/68
1:31 AM Course Outline 3
8/13/2019 basic lecture information theory and coding
4/68
Information is uncertainty: modeled as random
variablesInformation is digital: transmission should be 0s
and 1s (bits) with no reference to what they
represent
1:31 AM Course Outline 4
Two Key Concepts
8/13/2019 basic lecture information theory and coding
5/68
Source coding theorem
fundamental limit in data compression (zip,MP3, JPEG, MPEG)
Channel coding theorem
fundamental limit for reliable communicationthrough a noisy channel (telephone, cell phone,
modem, data storage, etc)
1:31 AM Course Outline 5
Two Fundamental Theorems
8/13/2019 basic lecture information theory and coding
6/68
Information & Coding Theory
The fundamental problem of communication is that ofreproducing at one point either exactly or
approximately a message selected at another point.
(Claude Shannon, 1948)
1:31 AM Course Outline 6
8/13/2019 basic lecture information theory and coding
7/68
Course Outline
1:31 AM Course Outline 7
8/13/2019 basic lecture information theory and coding
8/68
This course will provide the students an introduction to
classical information theory and coding theory. The maincourse objective is to introduce the students to well-known information theoretic tools that can be used tosolve engineering problems.
The course will begin by describing basic communicationsystems problems where information theory may be applied.
An explanation of information measurement andcharacterization will be given. Fundamentals of noiseless
source coding and noisy channel coding will be taught next.Finally, some key information theory principles applied tocommunication security systems will be covered.
1:31 AM Course Outline 8
Course Objective (3+0)
8/13/2019 basic lecture information theory and coding
9/68
Information theory is concerned with the fundamental limits of
communication.
What is the ultimate limit to data compression? e.g. how many bits
are required to represent a music source.
What is the ultimate limit of reliable communication over a noisy
channel, e.g. how many bits can be sent in one second over a
telephone line.
1:31 AM Course Outline 9
Course Outline -I
8/13/2019 basic lecture information theory and coding
10/68
Coding theory is concerned with practical techniques to realize the
limits specified by information theory
Source coding converts source output to bits.
Source output can be voice, video, text, sensor output
Channel coding adds extra bits to data transmitted over the channel
This redundancy helps combat the errors introduced in transmitted bits
due to channel noise
1:31 AM Course Outline 10
Course Outline -II
8/13/2019 basic lecture information theory and coding
11/68
Introduction
Communications Model
Information Sources
Source Coding
Channel Coding
Information Measurement
Definition and Properties of Entropy Uniqueness of the Entropy Measure
Joint and Conditional Entropy
Mutual Information and Conditional Mutual Information
Information Divergence Measures
1:31 AM Course Outline 11
Main Topics to be Covered
8/13/2019 basic lecture information theory and coding
12/68
8/13/2019 basic lecture information theory and coding
13/68
8/13/2019 basic lecture information theory and coding
14/68
8/13/2019 basic lecture information theory and coding
15/68
8/13/2019 basic lecture information theory and coding
16/68
Applied Coding and Theory for Engineers
Richard B. Wells, Prentice Hall, 1999.
A Mathematical Theory of Communication,
Claude E. Shannon,
Bell System Technical Journal, 1948
available for free on line
1:31 AM Course Outline 16
Recommended Text Books & Study Material
8/13/2019 basic lecture information theory and coding
17/68
ScheduleClass Meetings
Wednesday (5pm-8pm) 3L
Consultancy Hours
Wednesday (4pm-5pm), (8pm-8:30pm)
Other times by appointment (phone or email)
1:31 AM 17
8/13/2019 basic lecture information theory and coding
18/68
8/13/2019 basic lecture information theory and coding
19/68
8/13/2019 basic lecture information theory and coding
20/68
Introduction to Information Theory
1:31 AM 21Course Outline
8/13/2019 basic lecture information theory and coding
21/68
IT is about asking what is the most efficient path
from one point to another, in terms of some way ofmeasuring things.
1:31 AM 22
What is Information Theory (IT)?
Introduction to Information Theory
h i f i h ( )?
8/13/2019 basic lecture information theory and coding
22/68
Politics
Ask not what your country can do for you, but what you can do for
your country - John F. Kennedy
What makes the this political statements powerful (or at least
famous)?
force is efficiency of expression, there is an interpolationof many feelings,
attitudes and perceptions; there is an efficient encoding of emotional and
mental information.
1:31 AM 23
What is Information Theory (IT)?
Introduction to Information Theory
f i h
8/13/2019 basic lecture information theory and coding
23/68
Two important questions in engineering:
- What to do if information gets corrupted by errors?
- How much memory does it require to store data?
Both questions were asked and to a large degreeanswered by Shannon in his 1948 article:
use error correction and data compression.
1:31 AM 25
Information Theory
Claude Elwood Shannon (19162001), American electrical
engineer and mathematician, has been called the father of
information theory, and was the founder of practical digital
circuit design theory.
Introduction to Information Theory
bl i C i i
8/13/2019 basic lecture information theory and coding
24/68
Speed
Minimise length of transmitted data
Accuracy
Minimise and eliminate noise
Security
Ensure data is not changed or intercepted whilst in transit
1:31 AM 26
Problems in Communications
Introduction to Information Theory
S l i
8/13/2019 basic lecture information theory and coding
25/68
Speed
Minimise length of transmitted data
Use Data Compression
AccuracyMinimise and eliminate noise
Use Error Detection / Correction Codes
Security Ensure data is not changed or intercepted whilst in transit
Use Data Encryption / Authentication
1:31 AM 27
Solutions
Introduction to Information Theory
C i i M d l
8/13/2019 basic lecture information theory and coding
26/68
1:31 AM 28
Communications Model
Source Destination
signal
noise
Transmitter Receiver
received
signal
data data
Evesdropper
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
27/68
E D t ti /C ti C d
8/13/2019 basic lecture information theory and coding
28/68
Error detectionis the ability to detect errors that are
made due to noise or other impairments in the course ofthe transmission from the transmitter to the receiver.
Error correctionhas the additional feature that enables
locating the errors and correcting them.
Examples: Compact Disc, DVD, GSM
Algorithms: Check Digit, Parity Bit, CRC, HammingCode, Reed-Solomon Code, Convolutional Codes, TurboCodes and LDPC Codes
1:31 AM 30
Error Detecting/Correcting Codes
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
29/68
Wh t i i f ti ?
8/13/2019 basic lecture information theory and coding
30/68
information: [m-w.org]
1: the communication or reception of knowledge or
intelligence
1:31 AM 32
What is information?
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
31/68
Wh t i i f ti ?
8/13/2019 basic lecture information theory and coding
32/68
Intuitively, an information source having more symbols
should have more information
For instance, consider a source, say S1, that wants to
communicate its direction to a destination using the
following symbols:
North (N), South (S), East (E), West (W)Another source, say S2, can communicate its direction
using:
North (N), South (S), East (E), West (W), Northwest (NW),
Northeast (NE), Southwest (SW), Southeast (SE)
Intuitively, all other things being equally likely, S2has
more information than S1
1:31 AM 34
What is an information source?
Introduction to Information Theory
Mi i b f bit f
8/13/2019 basic lecture information theory and coding
33/68
Before we formally define information, let us try toanswer the following question:
What is the minimum number of bits/symbolrequired to communicate an information source
having nsymbols?
1:31 AM 35
Minimum number of bits for a source
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
34/68
Minim m n mber of bits for a so rce
8/13/2019 basic lecture information theory and coding
35/68
Let there be a sourceXthat wants to communicate
information of its direction to a destination
i.e., n=4 symbols: North (N), South (S), East (E), West (W)
According to our previous definition, log2(4)=2 bits are
required to represent each symbol
N: 00, S: 01,E: 10, W: 11
If 1000 symbols are generated byX, how many bits are
required to transmit these 1000 symbols?
1:31 AM 37
Minimum number of bits for a source
Introduction to Information Theory
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
36/68
Let there be a sourceXthat wants to communicate informationof its direction to a destination
i.e., n=4 symbols: North (N), South (S), East (E), West (W) According to our previous definition, log2(4)=2 bits are
required to represent each symbol
N: 00, S: 01,E: 10, W: 11
If 1000 symbols are generated byX, how many bits arerequired to transmit these 1000 symbols?
2000 bits are required to communicate 1000 symbols2 bits/symbol
1:31 AM 38
Minimum number of bits for a source
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
37/68
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
38/68
Are 2 bits/symbol the minimum number of bits/symbol
required to communicate an information source having
n=4 symbols?
The correct answer isNO!
Lets see an example to emphasize this point
1:31 AM 40
Minimum number of bits for a source
Introduction to Information Theory
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
39/68
So far in this example, we implicitly assumed that all
symbols are equally likely to occur
Lets now assume that symbols are generated according to
a probability mass functionpX
1:31 AM 41
Minimum number of bits for a source
N
0.6
0.3
S E
0.05
W
pX
X
Introduction to Information Theory
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
40/68
Let us map the symbols to the following bit sequences:N: 0
S: 01
E: 011
W: 0111
1:31 AM 42
Minimum number of bits for a source
N
0.6
0.3
S E
0.05
W
pX
X
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
41/68
8/13/2019 basic lecture information theory and coding
42/68
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
43/68
1:31 AM 45
Minimum number of bits for a source
Now if 1000 symbols are generated byX, how many bits are
required to transmit these 1000 symbols?
600 Ns, 300 Ss, 50 Es and 50 Ws
Total bits=6001+3002+503+504=1550
1550 bitsare required to communicate 1000 symbols
1.55 bits/symbol
N
0.6
0.3
S E
0.05
W
pX
X
Introduction to Information Theory
N: 0
S: 01
E: 011
W: 0111
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
44/68
1:31 AM 46
Minimum number of bits for a source
1550 bitsare required to communicate 1000 symbols
1.55 bits/symbol
N: 0
S: 01
E: 011
W: 0111
N
0.6
0.3
S E
0.05
W
pX
X
The bit mapping defined in this example is generally called a code
And the process of defining this code is called source coding or
source compression
The mapped symbols (0, 01, 011 and 0111) are called codewords
Introduction to Information Theory
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
45/68
Coming back to our original question:
Are 1.55 bits/symbol the minimum number of bits/symbol
required to communicate an information source having
n=4 symbols?
1:31 AM 47
Minimum number of bits for a source
Introduction to Information Theory
Minimum number of bits for a source
8/13/2019 basic lecture information theory and coding
46/68
Are 1.55 bits/symbol the minimum number of bits/symbol
required to communicate an information source having
n=4 symbols?
The correct answer is I dont know!
To answer this question, we first need to know the
minimum number of bits/symbol for a source with 4symbols
1:31 AM 48
Minimum number of bits for a source
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
47/68
8/13/2019 basic lecture information theory and coding
48/68
8/13/2019 basic lecture information theory and coding
49/68
8/13/2019 basic lecture information theory and coding
50/68
Information content of a source
8/13/2019 basic lecture information theory and coding
51/68
If we assume equally-likely symbols, we will always be
able to communicate all the symbols of the source using
log2(n) bits/symbol
In other words, this is the maximum number of bitsrequired to communicate any discrete source
But if a sources symbols arein fact equally likely, what is
the minimum number of bits required to communicate this
source?
1:31 AM 53
Information content of a source
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
52/68
Information content of uniform sources
8/13/2019 basic lecture information theory and coding
53/68
The minimum number of bits required to represent a
discrete uniform source is log2(n) bits/symbol
For any discrete source where all symbols are not equally-
likely (i.e., non-uniform source), log2(n) represents themaximum number of bits/symbol
Among all discrete sources producing a given number of
n symbols, a uniform source has the highest information
content
1:31 AM 55
Information content of uniform sources
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
54/68
Information content of uniform sources
8/13/2019 basic lecture information theory and coding
55/68
Two uniform sources S1and S1
n1and n2respectively represent the total number of
symbols for the two sources with n1> n2
Which source has higher information content?
1:31 AM 57
Information content of uniform sources
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
56/68
Information content of uniform sources
8/13/2019 basic lecture information theory and coding
57/68
Thus if there are multiple sources with equally-likely
symbols, the source with the maximum number of
symbols has the maximum information content
In other words, for equally likely sources, a functionH(.)that quantifies information content of a source should be
an increasing function of the number of symbols
Lets call this functionH(n)
Any ideas whatH(n) should be?
1:31 AM 59
Information content of uniform sources
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
58/68
8/13/2019 basic lecture information theory and coding
59/68
8/13/2019 basic lecture information theory and coding
60/68
8/13/2019 basic lecture information theory and coding
61/68
8/13/2019 basic lecture information theory and coding
62/68
8/13/2019 basic lecture information theory and coding
63/68
Information content of a non-uniform source
8/13/2019 basic lecture information theory and coding
64/68
For a given symbol i, the information content of that
symbol is given by:
H(pX=i)=log2(1/pX=i)
So what is the expected or average value of the informationcontent of all the symbols ofpX?
1:31 AM 66Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
65/68
Information content of a non-uniform source
8/13/2019 basic lecture information theory and coding
66/68
The information content of a discrete source with symboldistributionpXis:
1:31 AM 68
This is called the entropyof the source
and represents the minimum expected number of
bits/symbol required to communicate this source
Introduction to Information Theory
Information content of a non-uniform source
8/13/2019 basic lecture information theory and coding
67/68
Before finishing our discussion on information sources,
apply the formula for entropy on a uniform source:
1:31 AM 69
1 2
1/n
n
pX
X
Introduction to Information Theory
8/13/2019 basic lecture information theory and coding
68/68