Channel coding

Introduction to Information theorychannel capacity and models

A.J. Han VinckUniversity of Essen

May 2011

This lecture

Some models Channel capacity

Shannon channel coding theorem converse

some channel models

Input X P(y|x) output Y

transition probabilities

memoryless:

- output at time i depends only on input at time i

- input and output alphabet finite

Example: binary symmetric channel (BSC)

Error Source

+

E

X

OutputInput

EXY

E is the binary error sequence s.t. P(1) = 1-P(0) = p

X is the binary information sequence

Y is the binary output sequence

1-p

0 0

p

1 1

1-p

from AWGN to BSC

Homework: calculate the capacity as a function of A and 2

p

Other models

0

1

0 (light on)

1 (light off)

p

1-p

X Y

P(X=0) = P0

0

1

0

E

1

1-e

e

e

1-e

P(X=0) = P0

Z-channel (optical) Erasure channel (MAC)

Erasure with errors

0

1

0

E

1p

pe

e

1-p-e

1-p-e

burst error model (Gilbert-Elliot)

Error Source

RandomRandom error channel; outputs independent P(0) = 1- P(1);

BurstBurst error channel; outputs dependent

Error SourceP(0 | state = bad ) = P(1|state = bad ) = 1/2;

P(0 | state = good ) = 1 - P(1|state = good ) = 0.999

State info: good or bad

good bad

transition probability Pgb

Pbg

PggPbb

channel capacity:

I(X;Y) = H(X) - H(X|Y) = H(Y) – H(Y|X) (Shannon 1948)

H(X) H(X|Y)

notes:capacity depends on input probabilities

because the transition probabilites are fixed

channelX Y

capacity)Y;X(Imax)x(P

Practical communication system design

message estimate

channel decoder

n

Code word in

receive

There are 2k code words of length nk is the number of information bits transmitted in n channel uses

2k

Code book

Code bookwith errors

Channel capacity

Definition:

The rate R of a code is the ratio k/n, where

k is the number of information bits transmitted in n channel uses

Shannon showed that: :

for R C

encoding methods exist

with decoding error probability 0

Encoding and decoding according to Shannon

Code: 2k binary codewords where p(0) = P(1) = ½

Channel errors: P(0 1) = P(1 0) = p

i.e. # error sequences 2nh(p)

Decoder: search around received sequence for codeword

with np differences

space of 2n binary sequences

decoding error probability

1. For t errors: |t/n-p|> Є

0 for n (law of large numbers)

2. > 1 code word in region (codewords random)

nand

)p(h1n

kRfor

022

2

2)12()1(P

)RBSCC(n)R)p(h1(n

n

)p(nhk

channel capacity: the BSC

1-p

0 0

p

1 1

1-p

X Y

I(X;Y) = H(Y) – H(Y|X)

the maximum of H(Y) = 1

since Y is binary

H(Y|X) = h(p)

= P(X=0)h(p) + P(X=1)h(p)

Conclusion: the capacity for the BSC CBSC = 1- h(p)Homework: draw CBSC , what happens for p > ½

channel capacity: the BSC

0.5 1.0

1.0

Bit error p

Cha

nnel

cap

acit

y

Explain the behaviour!

channel capacity: the Z-channel

Application in optical communications

0

1

0 (light on)

1 (light off)

p

1-p

X Y

H(Y) = h(P0 +p(1- P0 ) )

H(Y|X) = (1 - P0 ) h(p)

For capacity, maximize I(X;Y) over P0

P(X=0) = P0

channel capacity: the erasure channel

Application: cdma detection

0

1

0

E

1

1-e

e

e

1-e

X Y

I(X;Y) = H(X) – H(X|Y)

H(X) = h(P0 )

H(X|Y) = e h(P0)

Thus Cerasure = 1 – e

(check!, draw and compare with BSC and Z)P(X=0) = P0

Erasure with errors: calculate the capacity!

0

1

0

E

1p

pe

e

1-p-e

1-p-e

example

Consider the following example

1/3

1/3

0

1

2

0

1

2

For P(0) = P(2) = p, P(1) = 1-2p

H(Y) = h(1/3 – 2p/3) + (2/3 + 2p/3); H(Y|X) = (1-2p)log23

Q: maximize H(Y) – H(Y|X) as a function of pQ: is this the capacity?

hint use the following: log2x = lnx / ln 2; d lnx / dx = 1/x

channel models: general diagram

x1

x2

xn

y1

y2

ym

:

:

:

:

:

:

P1|1

P2|1P1|2P2|2

Pm|n

Input alphabet X = {x1, x2, …, xn}

Output alphabet Y = {y1, y2, …, ym}

Pj|i = PY|X(yj|xi)

In general:

calculating capacity needs more theory

The statistical behavior of the channel is completely defined by the channel transition probabilities Pj|i = PY|X(yj|xi)

* clue:

I(X;Y)

is convex in the input probabilities

i.e. finding a maximum is simple

Channel capacity: converse

For R > C the decoding error probability > 0

k/n

C

Pe

Converse: For a discrete memory less channel

1 1 1 1

( ; ) ( ) ( | ) ( ) ( | ) ( ; )n n n n

n n n

i i i i i i ii i i i

I X Y H Y H Y X H Y H Y X I X Y nC

Xi Yi

m Xn Yn m‘encoder channel

channel

Source generates one

out of 2k equiprobable

messages

decoder

Let Pe = probability that m‘ m

source

converse R := k/n

Pe 1 – C/R - 1/nR

Hence: for large n, and R > C,

the probability of error Pe > 0

k = H(M) = I(M;Yn)+H(M|Yn)

Xn is a function of M Fano

I(Xn;Yn) + 1 + k Pe

nC + 1 + k Pe

1 – C n/k - 1/k Pe

We used the data processing theorem Cascading of Channels

I(X;Y)X Y

I(Y;Z)Z

I(X;Z)

The overall transmission rate I(X;Z) for the cascade cannot be larger than I(Y;Z), that is:

)Z;Y(I)Z;X(I

Appendix:

Assume:

binary sequence P(0) = 1 – P(1) = 1-p

t is the # of 1‘s in the sequence

Then n , > 0

Weak law of large numbers

Probability ( |t/n –p| > ) 0

i.e. we expect with high probability pn 1‘s

Appendix:

Consequence:

n(p- ) < t < n(p + ) with high probability

1.

2.

3.

)p(nh)p(n

)p(n2n2

pn

nn2

t

n

)p(hpn

nn2loglim 2n

1

n

)p1(log)p1(plogp)p(h 22

Homework: prove the approximation using ln N! ~ N lnN for N large.

Or use the Stirling approximation: ! 2 N NN NN e

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

p

h

Binary Entropy: h(p) = -plog2p – (1-p) log2 (1-p)

Note:

h(p) = h(1-p)

Input X Output Y

Noise

)]Noise(H)Y(H[sup:Cap

W2/S2x

)x(p

Input X is Gaussian with power spectral density (psd) ≤S/2W;

Noise is Gaussian with psd = 2noise

Output Y is Gaussian with psd = y2 = S/2W + 2

noise

Capacity for Additive White Gaussian Noise

W is (single sided) bandwidth

For Gaussian Channels: y2 =x

2 +noise2

X Y X Y

Noise

.sec/bits)W2/S

(logWCap

.trans/bits)(log

.trans/bits)e2(log))(e2(logCap

2noise

2noise

2

2noise

2x

2noise

221

2noise22

12noise

2x22

1

bits)e2(log)Z(H;e2

1)z(p 2

z2212/z

2z

2z

2

Middleton type of burst channel model

Select channel k with probability Q(k)

Transition probability P(0)

0

1

0

1

…

channel 1

channel 2

channel k has transition probability p(k)

Fritzman model:

multiple states G and only one state B

Closer to an actual real-world channel

Gn B1-p

Error probability 0 Error probability h

…G1

Interleaving: from bursty to random

Message interleaver channel interleaver -1 message

encoder decoder

bursty

„random error“

Note: interleaving brings encoding and decoding delay

Homework: compare the block and convolutional interleaving w.r.t. delay

Interleaving: block

Channel models are difficult to derive:

- burst definition ?

- random and burst errors ?

for practical reasons: convert burst into random error

read in row wise

transmit

column wise

1

0

0

1

1

0

1

0

0

1

1

0

0

0

0

0

0

1

1

0

1

0

0

1

1

De-Interleaving: block

read in column wisethis row contains 1 error

1

0

0

1

1

0

1

0

0

1

1

e

e

e

e

e

e

1

1

0

1

0

0

1

1

read out

row wise

Interleaving: convolutional

input sequence 0

input sequence 1 delay of b elements

input sequence m-1 delay of (m-1)b elements

Example: b = 5, m = 3

in

out