Replica method: a statistical mechanics approach to ...

bg=black!2 IntroductionLimiting eigenvalue distribution of random matrices

Digital communicationBibliography

Replica method: a statistical mechanics approachto probability-based information processing

Toshiyuki [email protected]

Graduate School of Informatics, Kyoto University, Kyoto, Japan

17th International Symposium on Mathematical Theory ofNetworks and Systems, Kyoto, Japan, July 25, 2006

Toshiyuki Tanaka MTNS2006: Replica method



Introduction

Replica method

Developed in studies of spin glasses (=magnetic materialswith random spin-spin interactions)

Recently applied to problems in information sciences:

Neural networksStatistical learning theoryCombinatorial optimization problemsError-correcting codesCDMA (digital wireless communication)Eigenvalue distribution of random matrices

Still lacks rigorous mathematical justification




Introduction

Objectives

To give a review of the replica method, as well as itsmathematically questionable point.

To demonstrate its applications.

Eigenvalue distribution of random matrices.Analysis of digital communication systems.




Basic resultsApproachesAnalysis via replica method

Problem

Basic defs.

A: N × N real symmetric random matrixλ1, . . . , λN : Eigenvalues of A

Empirical eigenvalue distribution

ρA(x) =1

N

N∑i=1

δ(x − λi )

Problem

To evaluateρ(x) = lim

N→∞ EA

[ρA(x)

]





Basic results

Wigner’s semicircle law (Wigner, 1951)

A = (aij): N × N matrix, aij (i ≤ j): i.i.d., mean 0, variance 1/N.

Marc̆enko-Pastur law (Marc̆enko & Pastur, 1967)

A = ΞTΞ, Ξ = (ξμi ): p × N matrix;ξμi i.i.d., mean 0, variance 1/N.

(Girko’s) full-circle law (Girko, 1985)

A = (aij), aij : i.i.d., mean 0, variance 1/N. (A not symmetric)





Wigner’s semicircle law


A = (aij), aij (i ≤ j): i.i.d, mean 0, variance 1/N.

⇒ ρ(x) =

⎧⎨⎩

1

2π

√4 − x2 (|x | < 2)

0 (|x | > 2)






-2 -1 0 1 2

Histogram of eigenvalues of a 6000 × 6000 random symmetricmatrix with entries following Gaussian distribution.





Marc̆enko-Pastur law


A = ΞTΞ, Ξ = (ξμi ): p × N matrix;ξμi : i.i.d., mean 0, variance 1/N.

ρ(x) =

⎧⎪⎪⎨⎪⎪⎩

√4α − (x − 1 − α)2

2πxχα(x) (α ≥ 1)

(1 − α)δ(x) +

√4α − (x − 1 − α)2

2πxχα(x) (0 < α < 1)

α ≡ p/Nχα(x): Characteristic function of interval [(1 −√

α)2, (1 +√

α)2].






0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 1 2 3 4 5 6

ρ(x)

x

α = 0.3, 0.6, 2; Terms proportional to δ(x) not shown.





Full-circle

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

Im(λ

)

Re(λ)

Eigenvalue distribution of a real 6000 × 6000 random matrix.





Eigenvalue distribution of random matrices

Useful for what?

Wide applications in mathematical physics

Applications in Information Processing

Statistical learning theoryDigital communication(kernel) PCA (bioinformatics, mathematical finance, etc.)





Eigenvalue distribution of random matrix

Approaches

Marginalization of joint eigenvalue distribution (ex. Mehta,

1967)

Evaluation of moments (ex. Brody et al., 1981)

“Locator” expansion (ex. Bray & Moore, 1979; Hertz et al., 1989)

Cavity method

Free probability theory (ex. Voiculescu, 1985; Hiai & Petz, 2000)

Replica method (ex. Edwards & Jones, 1976)





Reformulation

ρA(x) =1

N

N∑i=1

δ(x − λi)

mA(z) =1

Ntr(A − zI )−1

=2

N

d

dzlog ZA(z)

ZA(z) = (−2πi)N/2|A − zI |−1/2

=

∫R

N

exp[− i

2uT (A − zI )u

]du

mA(z) =

∫R

ρA(x)

x − zdx

ρA(x) = limε→+0

1

π�[

mA(x + iε)]

Stieltjes trans.

tr∗−1 = (log det ∗)′

Gaussian integ.rep. of det.





Averaging over A

ρ(x) = EA

[1

N

N∑i=1

δ(x − λi)

]

m(z) = EA

[1

Ntr(A − zI )−1

]

= 2d

dzEA

[1

Nlog ZA(z)

]

ZA(z) =

∫R

N

exp[− i

2uT (A − zI )u

]du

mA(z) =

∫R

ρA(x)

x − zdx

ρA(x) = limε→+0

1

π�[

mA(x + iε)]

Stieltjes trans.

tr∗−1 = (log det ∗)′





Outline of the approach

1 Evaluate

f (z) = limN→∞ EA

[1

Nlog ZA(z)

]

where (ZA(z) =

∫R

N

exp[− i

2uT (A − zI )u

]du

)

2 Calculate Stieltjes transform m(z) of ρ(x) with

m(z) = 2d

dzf (z).

3 Evaluate the inverse Stieltjes transform to obtain ρ(x):

ρ(x) = limε→+0

1

π�[

m(x + iε)]





Replica method

Rewriting of formulas

f (z) = limN→∞ EA

[1

Nlog ZA(z)

](

∂

∂nlog EA

(Z n

)=

EA(Z n log Z )

EA(Z n)

)

= limN→∞

1

Nlimn→0

∂

∂nlog EA

{[ZA(z)

]n}(

Exchange order of limn→0 ∂/∂nand limN→∞.

)

= limn→0

∂

∂nlim

N→∞1

Nlog EA

{[ZA(z)

]n}





Replica method

Goal

To evaluate limN→∞

1

Nlog EA

{[ZA(z)

]n}.

The replica “trick”

Evaluate it by assuming n to be a positive integer.

Believe the result to be valid for real n.

No mathematically rigorous justification.





Random matrix ensemble

Covariance matrix of random samples

A = ΞTΞ, Ξ = (ξμi ){ξμi ; μ = 1, . . . , p; i = 1, . . . , N} i.i.d.,

E(ξ) = 0, E(ξ2) = O(1/N), E(ξm) = o(1/N) (m ≥ 3)⇒ Marc̆enko-Pastur





Replica method

Introduction of “Replicated” systems

ZA(z) =

∫R

N

exp[− i

2uT (A − zI )u

]du

⇒ For n = 1, 2, . . .,

[ZA(z)

]n=

∫R

Nn

exp[− i

2

n∑a=1

uTa (A − zI )ua

] n∏a=1

dua

EA

{[ZA(z)

]n}=

∫R

NnEA

[exp

(− i

2

n∑a=1

uTa Aua

)]

× exp( iz

2

n∑a=1

∣∣ua

∣∣2) n∏a=1

dua





Calculation of EA(· · · )Factor to be evaluated

EA

[exp

(− i

2

n∑a=1

uTa Aua

)]

Assumptions on ramdom mtx. ensemble

A = ΞTΞ, Ξ = (ξμi ) (size: p × N)

{ξμi ; μ = 1, . . . , p; i = 1, . . . , N}: i.i.d.

vμa ≡N∑

i=1

ξμiuai ⇒ uTa Aua =

p∑μ=1

(vμa

)2

n∑a=1

uTa Aua =

p∑μ=1

n∑a=1

(vμa

)2=

p∑μ=1

∣∣vμ

∣∣2, (vμ = (vμ1, . . . , vμn)

T)





Calculation of EA(· · · )Average over A = Average over v

vμa ≡N∑

i=1

ξμiuai ⇒ uTa Aua =

p∑μ=1

(vμa

)2

n∑a=1

uTa Aua =

p∑μ=1

n∑a=1

(vμa

)2=

p∑μ=1

∣∣vμ

∣∣2(vμ = (vμ1, . . . , vμn)

T)

⇒ EA

[exp

(− i

2

n∑a=1

uTa Aua

)]=

{Ev

[exp

(− i

2

∣∣v∣∣2)]}p

(v = (ξTu1, . . . , ξTun)

T)





Assumptions on ramdom mtx. ensemble

E(ξ) = 0, E(ξ2) = 1/N, E(ξm) = o(1/N) (m ≥ 3)

Statistical properties of v = (ξTu1, . . . , ξTun)T

v ∼ N(0, Q) for fixed {ua} (⇐ Central limit theorem)

Order parameters

Q = (qab), qab ≡ Eξ(vvT ) = N−1

N∑i=1

uaiubi

⇒ Ev

[exp

(− i

2

∣∣v∣∣2)]=

∣∣I + iQ∣∣−1/2

(Gaussian integral)

exp( iz

2

n∑a=1

∣∣ua

∣∣2) = exp(Niz

2trQ

)





Integral with {ua} = Integral with Q

Ev

[exp

(− i

2

∣∣v∣∣2)]=

∣∣I + iQ∣∣−1/2

(Gaussian integral)

exp( iz

2

n∑a=1

∣∣ua

∣∣2) = exp(Niz

2trQ

)

⇒ EA

{[ZA(z)

]n}=

∫eNG(Q) μ(Q) dQ

G(Q) ≡ −α

2log

∣∣I + iQ∣∣ +

iz

2trQ, α ≡ p/N

μ(Q) ≡∫ ∏

a≤b

δ

(qab − 1

N

N∑i=1

uaiubi

) n∏a=1

dua

(Subshell volume)





Varadhan’s theorem (Large deviation theory)

limN→∞

1

Nlog

∫eNG(Q) μ(Q) dQ = sup

Q

[G(Q) − I(Q)]

Rate function I(Q)

The heuristic formula μ(Q) = e−NI(Q) holds for large N with

I(Q) = −1

2log |Q| + n

2

[1 + log(−2π)

].

Stationary condition (saddle-point equation)

∂

∂Q

[G(Q) − I(Q)]

= O ⇒ izI + Q−1 − iα(I + iQ)−1 = O





m(z) = limn→0

∂

∂nitrQ, izI + Q−1 − iα(I + iQ)−1 = O

Q uniquely determined by requiringintegrals not to diverge.

⎧⎪⎪⎨⎪⎪⎩

Q = qI

iz +1

q− iα

1 + iq= 0

m(z) = iq

⇒ m(z) = −[z − α

1 + m(z)

]−1

ρ(x) =

⎧⎪⎪⎨⎪⎪⎩

√4α − (x − 1 − α)2

2πxχα(x) (α ≥ 1)

(1 − α)δ(x) +

√4α − (x − 1 − α)2

2πxχα(x) (0 < α < 1)

(Marc̆enko-Pastur)




Basic settingUse of randomness in communicationReplica method

CommunicationBasic setting

X

Input

Channel Y

Output

p(x) p(y |x)

Output Y · · · What we observe.

Input X · · · What we want to know!

Problem

How much the output Y conveys information about the input X?





Communication

Problem

How much the output Y conveys information about the input X?

Mutual information

I (X ; Y ) =

∫p(x , y) log

p(x , y)

p(x)p(y)dx dy

= H(Y ) − H(Y |X )

H(Y |X ) = −∫ [∫

p(y |x) log p(y |x) dy

]p(x) dx

H(Y ) = −∫

p(y) log p(y) dy





Use of randomness in communication

Basic diagram

X

Input

Channel? Y

Output

Error-correcting code: ? =Encoder

Modulation: ? =Modulator






Examples

“Turbo” codes (Berrou et al., 1993): Two convolutionalcodes interlinked with a random interleaver.

Low-density parity-check codes (Gallager, 1962): Randomensemble of low-density parity-check matrices.

Code-division multiple-access (CDMA): Spreadingmodulation with random spreading sequences.





Mobile communication

�

�

��

�

�

�

�

�





Multiple access and CDMA

Multiple-access: Multipleusers simultaneously commu-nicate with the same basestation.





2-user CDMA system

×

⇑

×

⇑

Noise

×

⇓

⇓

Alice

Information x1

Spreading code{sμ1}

Bob

Information x2

Spreading code{sμ2}

Base st.

{yμ} Rcvd. signal

{sμ1}

h1 ⇒ x̂1





K -user CDMA system

x1

x2

xK

�

�

�

Information

{sμ1}{sμ2} {sμK}� � �

Spreading codes

××

×

�

�

�

Channel

++

Noise{nμ}

Rcvd. signal{yμ}

yμ =1√N

K∑k=1

sμkxk + nμ

(μ = 1, . . . , N)






Basic diagram

X

Input

ChannelS Y

Output

p(x) p(y |x , s)

Modeling of randomness

p(y |x , s): Channel input-output characteristics depending onauxiliary random variable S (e.g., spreading codes).






Randomness-averaged mutual information

One wants to evaluate:

ES

[I (X ; Y |S)

]= ES

[H(Y |S)

] − ES

[H(Y |X , S)

]

Difficulty

ES

[H(Y |S)

]= −ES

[∫p(y |S) log p(y |S) dy

]

p(y |S) =

∫p(y |x , S) p(x) dx





Replica method

Evaluation of randomness-averaged entropy

ES

[H(Y |S)

]= −ES

[∫p(y |S) log p(y |S) dy

]= − lim

n→0

∂

∂nlog Ξn

Ξn ≡ ES

[∫ [p(y |S)

]n+1dy

]=

∫∫ [p(y |s)]n+1

p(s) dy ds

Replica method provides a powerful approach to evaluaterandomness-averaged mutual information.





Example of results

10-710-610-510-410-310-210-1100

0 2 4 6 8 10 12

Bit-

Err

or R

ate

Pb

Signal-to-Noise Ratio Eb /N0 [dB]

System load β = 1, 1.2, 1.4, 1.6, 1.8, 2Single-user limit





Discussion

S-shaped performance curve

Multiple solutions for performance.

Essentially the same as magnetization curve of ferromagnets:{Stable, Metastable, Unstable} solutions

-1

-0.5

0

0.5

1

-0.4 -0.2 0 0.2 0.4

Mag

netiz

atio

n

External magnetic field

“Hysteresis” · · · Affects behavior of estimation algorithms.





Replica method: Applications

Digital communication

Turbo codes (Montanari & Sourlas, 2000)

Low-density parity-check codes (Murayama et al., 2000)

Code-division multiple-access (Tanaka, 2002)

· · ·

Other fields

Associative memory of neural network, Hopfield model (Amitet al., 1985)

Perceptron learning (Gardner & Derrida, 1989)

Random K -SAT problem (Monasson & Zecchina, 1997)

· · ·





Replica method

Mathematics

Validity of replica solutions (not replica method) (Talagrand, 2003)

“It is difficult to see (in the replica method) more than away to guess the correct formula.” — Talagrand, 2003.

Current status

Empirically gives the correct results to various problems.

Heuristics: Validity unknown.

Justification (or counterexample) needed.




Bibliography

D. J. Amit et al., Phys. Rev. Lett., 55(14), 1530–1533, 1985.

Berrou et al., Proc. IEEE Int. Conf. Commun., 1064–1070, 1993.

A. Bray & Moore, J. Phys. C: Solid State Phys., 12(11), L441–L448, 1979.

T. A. Brody et al., Rev. Mod. Phys., 53(3), 385–479, 1981.

S. F. Edwards & R. C. Jones, J. Phys. A: Math. Gen., 9(10), 1595–1603, 1976.

R. G. Gallager, Trans. IRE Info. Theory, 8, 21–28, 1962.

E. Gardner & B. Derrida, J. Phys. A: Math. Gen., 22(12), 1983–1994, 1989.

V. L. Girko, Theory of Prob. Its Appl. (USSR), 29(4), 694–706, 1985.

J. A. Hertz et al., J. Phys. A: Math. Gen., 22(12), 2133–2150, 1989.

F. Hiai & D. Petz, The Semicircle Law, Free Random Variables, and Entropy, Amer. Math. Soc., 2000.

V. A. Marc̆enko & L. A. Pastur, Math. USSR-Sb., 1, 457–483, 1967.

M. L. Mehta, Random Matrices, Academic Press, 1967.

R. Monasson & R. Zecchina, Phys. Rev. E, 56(2), 1357–1370, 1997.

A. Montanari & N. Sourlas, Eur. Phys. J. B, 18(1) 107–119, 2000.

T. Murayama et al., Phys. Rev. E, 62(2), 1577–1591, 2000.

M. Talagrand, Spin Glasses: A Challenge for Mathematicians, Springer, 2003.

T. Tanaka, IEEE Trans. Info. Theory, 48(11), 2888–2910, 2002.

D. V. Voiculescu, in Operator Algebras and Their Connection with Topology and Ergodic Theory,Lecture Notes in Math., 1131, Springer, 1985, 556–588.

E. P. Wigner, Proc Cambridge Phil. Soc., 47, 790–798, 1951.


Replica method: a statistical mechanics approach to ...

Documents