Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 6th Graphical model and physical model Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University [email protected]http://www.smapip.is.tohoku.ac.jp/~kazu/
45
Embed
Physical Fluctuomatics Applied Stochastic Process 6th Graphical model and physical model
Physical Fluctuomatics Applied Stochastic Process 6th Graphical model and physical model. Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University [email protected] http://www.smapip.is.tohoku.ac.jp/~kazu/. - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1
Physical FluctuomaticsApplied Stochastic Process
6th Graphical model and physical model
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5.
ReferencesH. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011.M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 3
Main InterestsInformation Processing:
DataPhysics:
Material, Natural Phenomena
System of a lot of elements with mutual relationCommon Concept between Information Sciences and Physics
MaterialMolecule
Materials are constructed from a lot of molecules.
A sequence is formed by deciding the arrangement of bits.
A lot of elements have mutual relation of each otherSome physical concepts
in Physical models are useful for the design of computational models in probabilistic information processing.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 4
Why is a physical viewpoint effective in probabilistic information processing?
Matrials are constructed from a lot of molecules.(1023 molecules exist in 1 mol.)
Molecules have intermolecular forces of each other
1 2
,,, 21x x x
N
N
xxxf
Theoretical physicists always have to treat such multiple summation.
Development of Approximate MethodsProbabilistic information processing is also usually reduced to multiple summations or integrations.
Application of physical approximate methods to probabilistic information processing
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 5
Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for
Ferromagnetic Materials
p p
p p
)1,1()1,1()1.1()1.1( PPPP
pPP )1.1()1,1(
11 a
1
12 a
1
11
1 1
p
PP
2
1
)1.1()1,1(
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 6
Probabilistic Model for Ferromagnetic MaterialsProbabilistic Model for
Ferromagnetic Materials
Prior probability prefers to the configuration with the least number of red lines.
> >=
Lines Red of #Lines Blue of # )2
1()( ppaP
p p
11 a 112 a 111 1 1
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 7
More is different in Probabilistic Model for Ferromagnetic Materials
Disordered State
Ordered State
Sampling by Markov Chain Monte Carlo method
p p
Small p Large p
p p
More is different.
p2
1p
2
1
Critical Point(Large fluctuation)
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 8
Fundamental Probabilistic Models for Magnetic Materials
Since h is positive, the probablity of up spin is larger than the one of down spin .
1
)exp(
)exp()(
a
ha
haaP
1a
+1 1
he he
)tanh()(1
haaPma
h : External Field
)(tanh1)()( 2
1
2 haPmaaVa
Variance
Average
0h
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 9
Fundamental Probabilistic Models for Magnetic Materials
Since J is positive, (a1,a2)=(+1,+1) and (1,1) have the largest probability .
1 121
2121
1 2
)exp(
)exp(),(
a a
aJa
aJaaaP
11 a
0),(1 1
2111
1 2
a a
aaPam
J : Interaction
1),()(1 1
212
111
1 2
a a
aaPmaaVVariance
Average
0J
Je Je
+1 +1 1 1
+1 +1 11
12 aJe Je
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 10
Fundamental Probabilistic Models for Magnetic Materials
a
aEZ
))(exp(
Eji
jiVi
i aaJahaE},{
)(
Translational Symmetry
),( EVJ
J
h h
)(exp1
)( aEZ
aP
),,,( 21 Naaaa
E : Set of All the neighbouring Pairs of Nodes
1ia 1ia
N
i ai aPa
Nm
1
)(1
Problem: Compute
)'()()'()( aPaPaEaE
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 11
Fundamental Probabilistic Models for Magnetic Materials
Eji
jiVi
i aaJahaE},{
)(
N
i ai
NhaPa
Nm
10)(
1limlim
)(exp1
)( aEZ
aP
),,,( 21 Naaaa
1ia
Problem: Compute
Translational Symmetry
),( EV
J
J
h h
Spontaneous Magnetization
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 12
Mean Field Approximation for Ising Model
)},{( 0))(( Ejimama ji We assume that the probability for configurations satisfying
Vi
iaJmhaE )4()(
2mmamaaa ijji
Eji
jiVi
i aaJahaE},{
iJm
Jm
JmJm
h
are large.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 13
Mean Field Approximation for Ising Model
)4tanh()(1
1
JmhaPaN
mN
i ai
Vi
ii aPaEZ
aP )())(exp(1
)(
Fixed Point Equation of m)(mm
We assume that all random variables ai are independent of each other, approximately.
Vi
iaJmhaE )4()(
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 14
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 15
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 16
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
0M0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 17
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
01 MM
0M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 18
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
12
01
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 19
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
12
01
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 20
Fixed Point Equation and Iterative Method
•Fixed Point Equation ** MM •Iterative Method
23
12
01
MM
MM
MM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 21
Marginal Probability Distribution in Mean Field Approximation
))4exp((1
)()(
1 2 1 1
ii
a a a a aii
aJmhZ
aPaP
i i N
i
JmJm
JmJm
h
1
)(
iaiii aPam
))4tanh(( mJhm Jm: Mean Field
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 22
Advanced Mean Field Method
))4exp((1
)( ii
ii ahZ
aP
)))(3exp((1
),( jijii
jiij aJaaahZ
aaP
h
h
h
1
),()(
jajiijii aaPaP
))3tanh()(tanh(arctanh hJ
Bethe Approximation
Kikuchi Method (Cluster Variation Meth)
: Effective Field
Fixed Point Equation for
J
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 23
Average of Ising Model on Square Grid Graph
(a) Mean Field Approximation(b) Bethe Approximation(c) Kikuchi Method (Cluster Variation Method)(d) Exact Solution ( L. Onsager , C.N.Yang )
J/1
a
iNh
aPa
)(limlim
0
Ejiji
Vii aaJah
ZaP
},{
exp1 ),( EVJ
J
h h
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 24
Model Representation in Statistical Physics
),,,(},,,Pr{ 212211 NNN aaaPaAaAaA
a
aEZ
))(exp(
)(}Pr{ aPaA
))(exp(1
)( aEZ
aP
),,,( 21 NAAAA
Gibbs Distribution Partition Function
)))(exp(ln(ln a
aEZF
Free Energy
Energy Function
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 25
Gibbs Distribution and Free Energy
Gibbs Distribution
ZPFaQQFaQ
ln][}1)(|][{min
))(exp(1
)( aEZ
aP
)(ln)()()(][ aQaQaQaEQFaa
Variational Principle of Free Energy Functional Variational Principle of Free Energy Functional FF[[QQ] under Normalization Condition for ] under Normalization Condition for QQ((aa))
Free Energy Functional of Trial Probability Distribution Q(a)
a
aEZ
))(exp(lnlnFree Energy
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 26
Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional
ZPFaQQFaQ
ln][}1)(|][{min
)(
)(exp
)(exp)(ˆ aP
aE
aEaQ
a
1)()())(ln)((1)(
aaa
aQaQaQaEaQQFQL
01)(ln)(
)(
aQaEaQ
QL
1)(exp)(ˆ aEaQ
Normalization Condition
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 27
Kullback-Leibler Divergence and Free Energy
0)(
)(ln)(
aP
aQaQPQD
a
a
aQaQ
1)( ,0)(
ZQF
ZaQaQaEaQPQD
QF
aa
ln][
ln)(ln)()()(]|[
][
0)()( PQDaPaQ
))(exp(1
)( aEZ
aP
}1)(|]|[{minarg}1)(|][{minarg aQaQ
aQPQDaQQF
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 28
Interpretation of Mean Field Approximation as Information Theory
Vi
ii aQaQ )()(
)(
)(ln)(
aP
aQaQPQD
a
))(exp(1
aEZ
aP
and
Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P]
1 2 1 1 2
)()()(\ a a a a a aaa
ii
i i i Ni
aQaQaQ
Minimization of Kullback-Leibler Divergence between
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 29
Interpretation of Mean Field Approximation as Information Theory
Eji
jiVi
i aaJahaE},{
)(
Vi a
iiVi a
i
i
aPaV
aPaV
m1
)(||
1)(
||
1
)(exp1
)( aEZ
aP
),,,( ||21 Vaaaa
1ia
Problem: Compute
Translational Symmetry
),( EV
J
J
h h
Magnetization
1 2 1 1 2
)()()(\ a a a a a aaa
ii
i i i Ni
aPaPaP
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 30
Kullback-Leibler Divergence in Mean Field Approximation for Ising Model
Vi
ii aQaQ )()(
ZViQFPQD i ln|MF
)(
)(ln)(
aP
aQaQPQD
a
Viii
Ejiji
Viii
QQQQJ
QhViQF
1},{ 11
1MF
ln))()()((
)(}]|[{
1 2 1 1 2
)(
)()(\
a a a a a a
aaii
i i i N
i
aQ
aQaQ
Eji
jiVi
i aaJahaE},{
)(
)(exp1
)( aEZ
aP
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 31
Minimization of Kullback-Leibler Divergence and Mean Field Equation
)( ))(ˆ(exp1ˆ
1
ViQJhZ
Qij
ji
i
} ,1)(|]|[{minarg)}(ˆ{}{
ViQPQDQ iQ
ii
Fixed Point Equations for {Qi|iV}
Variation
1 1
))(ˆ(exp
ij
ji QJhZ
i
Set of all the neighbouring nodes of the node i
Ejiji },{
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 32
Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model
iiii amaQ2
1
2
1)(
1
)(
iaiii
aii aQaaQam
),,,( 21 Naaaa
1ia
ia
iiia
iia
iii
aii
ai
aii
iiii
maQadddacaaQa
aQccdacaQ
adacaQ
iii
iii
2
1)(
2
12)()(
2
1)(
2
12)()(
1)( )(
111
111
2
1 2 1 1
)()()(\ a a a a aaa
ii
i i Ni
aQaQaQ
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 33
Conventional Mean Field Equation in Ising Model
)4tanh( Jmhm
iiiiiaa
ii maamaQaPaPi
2
1
2
1
2
1
2
1)(ˆ)()(
\
maPaN
N
i ai
1
)(1
Fixed Point Equation
mmmm N 21
Eji
jiVi
i aaJahaE},{
)(
))4exp((1
))(ˆ(exp1
)(ˆ1
ii
iij
ji
ii aJmhZ
aQJhZ
aQ
VJ
J
Translational Symmetry
h h
)( 4|| Vii
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 34
It corresponds to Bethe approximation in the statistical mechanics.
43Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
Interpretation of Bethe Approximation (11)
1 1 \
1 \
,
,
jikikij
jikikij
ji M
M
M
44Physics Fluctuomatics / Applied
Stochastic Process (Tohoku University)
))tanh()(tanh(arctanh\
jik
ikji hJ
jijiM exp
))3tanh()(tanh(arctanh hJ
ji Translational Symmetry
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 45
Summary
Statistical Physics and Information TheoryProbabilistic Model of FerromagnetismMean Field TheoryGibbs Distribution and Free EnergyFree Energy and Kullback-Leibler DivergenceInterpretation of Mean Field Approximation as Information TheoryInterpretation of Bethe Approximation as Information Theory