Transcript
Introduction to Information theorychannel capacity and models
A.J. Han VinckUniversity of Essen
May 2011
This lecture
Some models Channel capacity
Shannon channel coding theorem converse
some channel models
Input X P(y|x) output Y
transition probabilities
memoryless:
- output at time i depends only on input at time i
- input and output alphabet finite
Example: binary symmetric channel (BSC)
Error Source
+
E
X
OutputInput
EXY
E is the binary error sequence s.t. P(1) = 1-P(0) = p
X is the binary information sequence
Y is the binary output sequence
1-p
0 0
p
1 1
1-p
from AWGN to BSC
Homework: calculate the capacity as a function of A and 2
p
Other models
0
1
0 (light on)
1 (light off)
p
1-p
X Y
P(X=0) = P0
0
1
0
E
1
1-e
e
e
1-e
P(X=0) = P0
Z-channel (optical) Erasure channel (MAC)
Erasure with errors
0
1
0
E
1p
pe
e
1-p-e
1-p-e
burst error model (Gilbert-Elliot)
Error Source
RandomRandom error channel; outputs independent P(0) = 1- P(1);
BurstBurst error channel; outputs dependent
Error SourceP(0 | state = bad ) = P(1|state = bad ) = 1/2;
P(0 | state = good ) = 1 - P(1|state = good ) = 0.999
State info: good or bad
good bad
transition probability Pgb
Pbg
PggPbb
channel capacity:
I(X;Y) = H(X) - H(X|Y) = H(Y) – H(Y|X) (Shannon 1948)
H(X) H(X|Y)
notes:capacity depends on input probabilities
because the transition probabilites are fixed
channelX Y
capacity)Y;X(Imax)x(P
Practical communication system design
message estimate
channel decoder
n
Code word in
receive
There are 2k code words of length nk is the number of information bits transmitted in n channel uses
2k
Code book
Code bookwith errors
Channel capacity
Definition:
The rate R of a code is the ratio k/n, where
k is the number of information bits transmitted in n channel uses
Shannon showed that: :
for R C
encoding methods exist
with decoding error probability 0
Encoding and decoding according to Shannon
Code: 2k binary codewords where p(0) = P(1) = ½
Channel errors: P(0 1) = P(1 0) = p
i.e. # error sequences 2nh(p)
Decoder: search around received sequence for codeword
with np differences
space of 2n binary sequences
decoding error probability
1. For t errors: |t/n-p|> Є
0 for n (law of large numbers)
2. > 1 code word in region (codewords random)
nand
)p(h1n
kRfor
022
2
2)12()1(P
)RBSCC(n)R)p(h1(n
n
)p(nhk
channel capacity: the BSC
1-p
0 0
p
1 1
1-p
X Y
I(X;Y) = H(Y) – H(Y|X)
the maximum of H(Y) = 1
since Y is binary
H(Y|X) = h(p)
= P(X=0)h(p) + P(X=1)h(p)
Conclusion: the capacity for the BSC CBSC = 1- h(p)Homework: draw CBSC , what happens for p > ½
channel capacity: the BSC
0.5 1.0
1.0
Bit error p
Cha
nnel
cap
acit
y
Explain the behaviour!
channel capacity: the Z-channel
Application in optical communications
0
1
0 (light on)
1 (light off)
p
1-p
X Y
H(Y) = h(P0 +p(1- P0 ) )
H(Y|X) = (1 - P0 ) h(p)
For capacity, maximize I(X;Y) over P0
P(X=0) = P0
channel capacity: the erasure channel
Application: cdma detection
0
1
0
E
1
1-e
e
e
1-e
X Y
I(X;Y) = H(X) – H(X|Y)
H(X) = h(P0 )
H(X|Y) = e h(P0)
Thus Cerasure = 1 – e
(check!, draw and compare with BSC and Z)P(X=0) = P0
Erasure with errors: calculate the capacity!
0
1
0
E
1p
pe
e
1-p-e
1-p-e
example
Consider the following example
1/3
1/3
0
1
2
0
1
2
For P(0) = P(2) = p, P(1) = 1-2p
H(Y) = h(1/3 – 2p/3) + (2/3 + 2p/3); H(Y|X) = (1-2p)log23
Q: maximize H(Y) – H(Y|X) as a function of pQ: is this the capacity?
hint use the following: log2x = lnx / ln 2; d lnx / dx = 1/x
channel models: general diagram
x1
x2
xn
y1
y2
ym
:
:
:
:
:
:
P1|1
P2|1P1|2P2|2
Pm|n
Input alphabet X = {x1, x2, …, xn}
Output alphabet Y = {y1, y2, …, ym}
Pj|i = PY|X(yj|xi)
In general:
calculating capacity needs more theory
The statistical behavior of the channel is completely defined by the channel transition probabilities Pj|i = PY|X(yj|xi)
* clue:
I(X;Y)
is convex in the input probabilities
i.e. finding a maximum is simple
Channel capacity: converse
For R > C the decoding error probability > 0
k/n
C
Pe
Converse: For a discrete memory less channel
1 1 1 1
( ; ) ( ) ( | ) ( ) ( | ) ( ; )n n n n
n n n
i i i i i i ii i i i
I X Y H Y H Y X H Y H Y X I X Y nC
Xi Yi
m Xn Yn m‘encoder channel
channel
Source generates one
out of 2k equiprobable
messages
decoder
Let Pe = probability that m‘ m
source
converse R := k/n
Pe 1 – C/R - 1/nR
Hence: for large n, and R > C,
the probability of error Pe > 0
k = H(M) = I(M;Yn)+H(M|Yn)
Xn is a function of M Fano
I(Xn;Yn) + 1 + k Pe
nC + 1 + k Pe
1 – C n/k - 1/k Pe
We used the data processing theorem Cascading of Channels
I(X;Y)X Y
I(Y;Z)Z
I(X;Z)
The overall transmission rate I(X;Z) for the cascade cannot be larger than I(Y;Z), that is:
)Z;Y(I)Z;X(I
Appendix:
Assume:
binary sequence P(0) = 1 – P(1) = 1-p
t is the # of 1‘s in the sequence
Then n , > 0
Weak law of large numbers
Probability ( |t/n –p| > ) 0
i.e. we expect with high probability pn 1‘s
Appendix:
Consequence:
n(p- ) < t < n(p + ) with high probability
1.
2.
3.
)p(nh)p(n
)p(n2n2
pn
nn2
t
n
)p(hpn
nn2loglim 2n
1
n
)p1(log)p1(plogp)p(h 22
Homework: prove the approximation using ln N! ~ N lnN for N large.
Or use the Stirling approximation: ! 2 N NN NN e
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
p
h
Binary Entropy: h(p) = -plog2p – (1-p) log2 (1-p)
Note:
h(p) = h(1-p)
Input X Output Y
Noise
)]Noise(H)Y(H[sup:Cap
W2/S2x
)x(p
Input X is Gaussian with power spectral density (psd) ≤S/2W;
Noise is Gaussian with psd = 2noise
Output Y is Gaussian with psd = y2 = S/2W + 2
noise
Capacity for Additive White Gaussian Noise
W is (single sided) bandwidth
For Gaussian Channels: y2 =x
2 +noise2
X Y X Y
Noise
.sec/bits)W2/S
(logWCap
.trans/bits)(log
.trans/bits)e2(log))(e2(logCap
2noise
2noise
2
2noise
2x
2noise
221
2noise22
12noise
2x22
1
bits)e2(log)Z(H;e2
1)z(p 2
z2212/z
2z
2z
2
Middleton type of burst channel model
Select channel k with probability Q(k)
Transition probability P(0)
0
1
0
1
…
channel 1
channel 2
channel k has transition probability p(k)
Fritzman model:
multiple states G and only one state B
Closer to an actual real-world channel
Gn B1-p
Error probability 0 Error probability h
…G1
Interleaving: from bursty to random
Message interleaver channel interleaver -1 message
encoder decoder
bursty
„random error“
Note: interleaving brings encoding and decoding delay
Homework: compare the block and convolutional interleaving w.r.t. delay
Interleaving: block
Channel models are difficult to derive:
- burst definition ?
- random and burst errors ?
for practical reasons: convert burst into random error
read in row wise
transmit
column wise
1
0
0
1
1
0
1
0
0
1
1
0
0
0
0
0
0
1
1
0
1
0
0
1
1
De-Interleaving: block
read in column wisethis row contains 1 error
1
0
0
1
1
0
1
0
0
1
1
e
e
e
e
e
e
1
1
0
1
0
0
1
1
read out
row wise
Interleaving: convolutional
input sequence 0
input sequence 1 delay of b elements
input sequence m-1 delay of (m-1)b elements
Example: b = 5, m = 3
in
out
top related