1 October, 2007 1 October, 2007 ALT&DS2007 (Sendai, Japan ALT&DS2007 (Sendai, Japan) 1 Introduction to Introduction to Probabilistic Image Processing Probabilistic Image Processing and Bayesian Networks and Bayesian Networks Kazuyuki Tanaka Kazuyuki Tanaka Graduate School of Information Sciences, Graduate School of Information Sciences, Tohoku University, Sendai, Japan Tohoku University, Sendai, Japan [email protected][email protected]http://www.smapip.is.tohoku.ac.jp/~kazu/ http://www.smapip.is.tohoku.ac.jp/~kazu/
51
Embed
Introduction to Probabilistic Image Processing and Bayesian … › ... › 08 › papers08 › bayes1.pdf · 2018-02-11 · Probabilistic Image Processing and Bayesian Networks Kazuyuki
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
S. S. GemanGeman and D. and D. GemanGeman (1986): IEEE Transactions on PAMI (1986): IEEE Transactions on PAMIImage Processing for Image Processing for Markov Random Fields (MRF)Markov Random Fields (MRF)(Simulated Annealing, Line Fields)(Simulated Annealing, Line Fields)
J. Zhang (1992): IEEE Transactions on Signal ProcessingJ. Zhang (1992): IEEE Transactions on Signal ProcessingImage Processing in EM algorithm for Image Processing in EM algorithm for MarkovMarkovRandom Fields (MRF)Random Fields (MRF) (Mean Field Methods) (Mean Field Methods)
Markov Random Fieldsare One of Probabilistic Methods for Image processing.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 5
Markov Random Fields for Image Processing
In Markov Random Fields, we have to consider notonly the states with high probabilities but also oneswith low probabilities.In Markov Random Fields, wehave to estimate not only theimage but also hyperparametersin the probabilistic model.We have to perform thecalculations of statisticalquantities repeatedly.
HyperparameterEstimation
Statistical Quantities
Estimation of Image
We need a deterministic algorithm for calculatingstatistical quantities.Belief Propagation
1 October, 2007 ALT&DS2007 (Sendai, Japan) 6
Belief Propagation
Belief Propagation has been proposed in order toachieve probabilistic inference systems (Pearl,1988).It has been suggested that Belief Propagation has aclosed relationship to Mean Field Methods in thestatistical mechanics (Kabashima and Saad 1998).Generalized Belief Propagation has been proposedbased on Advanced Mean Field Methods (Yedidia,Freeman and Weiss, 2000).Interpretation of Generalized Belief Propagationalso has been presented in terms of InformationGeometry (Ikeda, T. Tanaka and Amari, 2004).
1 October, 2007 ALT&DS2007 (Sendai, Japan) 7
Probabilistic Model and Belief Propagation
!!!!!1 2 3 4 5
),(),(),(),( 25242321
x x x x x
xxCxxCxxBxxA x3
x1 x2 x4
x5
!!!1 2 3
),(),(),( 133221
x x x
xxCxxBxxA
Treex3x1
x2
Cycle
Function consisting of a product of functions with twovariables can be assigned to a graph representation.
Examples
Belief Propagation can give us an exact result for thecalculations of statistical quantities of probabilistic modelswith tree graph representations.Generally, Belief Propagation cannot give us an exactresult for the calculations of statistical quantities ofprobabilistic models with cycle graph representations.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 8
Application of Belief Propagation
Turbo and LDPC codes in Error Correcting Codes (Berrou andGlavieux: IEEE Trans. Comm., 1996; Kabashima and Saad: J. Phys.A, 2004, Topical Review).CDMA Multiuser Detection in Mobile Phone Communication(Kabashima: J. Phys. A, 2003).Satisfability (SAT) Problems in Computation Theory (Mezard,Parisi, Zecchina: Science, 2002).Image Processing (Tanaka: J. Phys. A, 2002, Topical Review;Willsky: Proceedings of IEEE, 2002).Probabilistic Inference in AI (Kappen and Wiegerinck, NIPS, 2002).
Applications of belief propagation to many problems whichare formulated as probabilistic models with cycle graphrepresentations have caused to many successful results.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 9
Purpose of My Talk
Review of formulation of probabilistic modelfor image processing by means ofconventional statistical schemes.Review of probabilistic image processing byusing Gaussian graphical model (GaussianMarkov Random Fields) as the most basicexample.Review of how to construct a beliefpropagation algorithm for imageprocessing.
Digital image is defined on the set of points arranged ona square lattice.The elements of such a digital array are called pixels.We have to treat more than 100,000 pixels even in thedigital cameras and the mobile phones.
xx
y y
)1,1( )1,2( )1,3(
)2,1( )2,2( )2,3(
)3,1( )3,2( )3,3(
),( yxPixels 200,307480640 =!
1 October, 2007 ALT&DS2007 (Sendai, Japan) 12
Image Representation in Computer Vision
Pixels 65536256256 =!
x
y
At each point, the intensity of light is represented as aninteger number or a real number in the digital imagedata.A monochrome digital image is then expressed as atwo-dimensional light intensity function and the valueis proportional to the brightness of the image at thepixel.
( )yxfyx ,),( !
( ) 0, =yxf ( ) 255, =yxf
1 October, 2007 ALT&DS2007 (Sendai, Japan) 13
Noise Reduction byConventional Filters
173
110218100
120219202
190202192
Average =
!"
!#
$
!%
!&
'
192 202 190
202 219 120
100 218 110
192 202 190
202 173 120
100 218 110
It is expected that probabilistic algorithms forimage processing can be constructed from suchaspects in the conventional signal processing.
Markov Random Fields Probabilistic Image ProcessingAlgorithm
Smoothing Filters
The function of a linearfilter is to take the sumof the product of themask coefficients and theintensities of the pixels.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 14
Bayes Formula and Bayesian Network
Posterior Probability
}Pr{
}Pr{}|Pr{}|Pr{
B
AABBA =
Bayes Rule
PriorProbability
Event A is given as the observed data.Event B corresponds to the originalinformation to estimate.Thus the Bayes formula can be applied to theestimation of the original information fromthe given data.
Assumption 1: The degradedimage is randomly generatedfrom the original image byaccording to the degradationprocess.Assumption 2: The originalimage is randomly generatedby according to the priorprobability.
Bayes Formula
1 October, 2007 ALT&DS2007 (Sendai, Japan) 16
Image Restoration by Probabilistic Model
Degraded
Image
i
fi: Light Intensity of Pixel iin Original Image
),( iii yxr =r
Position Vectorof Pixel i
gi: Light Intensity of Pixel iin Degraded Image
i
Original
Image
The original images and degraded images arerepresented by f = {fi} and g = {gi}, respectively.
Assumption 1: A given degraded image is obtained from theoriginal image by changing the state of each pixel to anotherstate by the same probability, independently of the otherpixels.
Assumption 2: The original image is generated according to aprior probability. Prior Probability consists of a product offunctions defined on the neighbouring pixels.
i j
Product over All the Nearest Neighbour Pairs of Pixels
1 October, 2007 ALT&DS2007 (Sendai, Japan) 19
Prior Probability for Binary Image
== >p p p!
2
1p!
2
1i j Probability ofNeigbouringPixel
!"==
NeighbourNearest :
),(}Pr{
ij
ji fffFi j
It is important how we should assume the functionΦ(fi,fj) in the prior probability.
)0,1()1,0()0,0()1,1( !=!>!=!
We assume that every nearest-neighbour pair of pixelstake the same state of each other in the prior probability.
1,0=if
1 October, 2007 ALT&DS2007 (Sendai, Japan) 20
Prior Probability for Binary Image
Prior probability prefers to the configurationwith the least number of red lines.
Which state should the centerpixel be taken when the statesof neighbouring pixels are fixedto the white states?
?
>
== >p p p!
2
1p!
2
1i j Probability ofNearest NeigbourPair of Pixels
1 October, 2007 ALT&DS2007 (Sendai, Japan) 21
Prior Probability for Binary Image
Which state should the centerpixel be taken when the states ofneighbouring pixels are fixed asthis figure?
?-?== >
p p
> >=
Prior probability prefers to the configurationwith the least number of red lines.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 22
What happens for the case oflarge umber of pixels?
p 0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0lnp
Disordered State Critical Point(Large fluctuation)
small p large p
Covariancebetween thenearest neghbourpairs of pixels
Sampling byMarko chainMonte Carlo
Ordered State
Patterns withboth ordered statesand disordered statesare often generatednear the critical point.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 23
Pattern near Critical Pointof Prior Probability
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0ln p
similar
small p large p
Covariancebetween thenearest neghbourpairs of pixels
We regard that patternsgenerated near the criticalpoint are similar to thelocal patterns in real worldimages.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 24
Bayesian Image Analysis
{ }{ } { }
{ }
!!
"
#
$$
%
&'
!!
"
#
$$
%
&()
=
======
**= NeighbourNearest:1
),(),(
Pr
PrPrPr
ij
ji
N
i
ii ffgf
gG
fFfFgGgGfF
fg
{ }fF =Pr { }fFgG ==Pr gOriginalImage
DegradedImage
Prior Probability
Posterior Probability
Degradation Process
Image processing is reduced to calculations ofaverages, variances and co-variances in theposterior probability.
B:Set of allthe nearestneighbourpairs of pixels
Ω:Set of Allthe pixels
1 October, 2007 ALT&DS2007 (Sendai, Japan) 25
Estimation of Original Image
We have some choices to estimate the restored image fromposterior probability.
In each choice, the computational time is generallyexponential order of the number of pixels.
}|Pr{maxargˆ gG === iiz
i zFfi
2
}|Pr{minargˆ!!
"
#
$$
%
&=='= (
z
gGzFiii zfi
))
}|Pr{maxargˆ gGzFfz
===
Thresholded Posterior Mean (TPM) estimation
Maximum posterior marginal (MPM) estimation
Maximum A Posteriori (MAP) estimation
! =====
if
ii fF
\
}|Pr{}|Pr{
f
gGfFgG
(1)
(2)
(3)
1 October, 2007 ALT&DS2007 (Sendai, Japan) 26
Statistical Estimation of Hyperparameters
! =====
z
zFzFgGgG }|Pr{},|Pr{},|Pr{ "##"
( )},|Pr{max arg)ˆ,ˆ(
,
!"!"!"
gG ==
f
Marginalized with respect to F
}|Pr{ !fF = },|Pr{ !fFgG == gOriginal Image
Marginal Likelihood
Degraded Image!y
x
},|Pr{ !"gG =
Hyperparameters α, β are determined so as tomaximize the marginal likelihood Pr{G=g|α,β} withrespect to α, β.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 27
Maximization of Marginal Likelihood byEM Algorithm
! =====
z
zFzFgGgG }|Pr{},|Pr{},|Pr{ "##"MarginalLikelihood
( ) },|,Pr{ln}',',|Pr{
,',',
! =====
z
gGzFgGzF
g
"#"#
"#"#Q
( ) ( )( )
( ) ( )( )( )
( ) ( )( ).,,maxarg1,1 :Step-M
},|,Pr{ln)}(),(,|Pr{
,,
:Step-E
,ttQtt
tt
ttQ
!"!"!"
!"!"
!"!"
!"#++
====#$z
gGzFgGzF
E-step and M-Step are iterated until convergence:EM (Expectation Maximization) Algorithm
It is possible to calculate eachsummation independently.
It is hard to calculate eachsummation independently.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 36
Belief Propagationfor Tree Graphical Model
3
1 2
5
4
3
1 2
5
4
23!M
24!M
25!M
After taking thesummations over rednodes 3,4 and 5,the function of nodes 1and 2 can be expressedin terms of somemessages. 44 344 2144 344 2144 344 21
)(
52
)(
42
)(
3221
52423221
225
8
224
7
223
3
3 4 5
),(),(),(),(
),(),(),(),(
fM
f
fM
f
fM
f
f f f
ffDffCffBffA
ffDffCffBffA
!!!
""
#
$
%%
&
'
""
#
$
%%
&
'
""
#
$
%%
&
'= (((
(((
1 October, 2007 ALT&DS2007 (Sendai, Japan) 37
Belief Propagationfor Tree Graphical Model
3
1 2
5
4
By taking the summation over all the nodesexcept node 1, message from node 2 to node 1can be expressed in terms of all the messagesincoming to node 2 except the own message.
3
1 2
5
4
23!M
24!M
25!M
!=2z
212!M
1
Summation over allthe nodes except 1
1 October, 2007 ALT&DS2007 (Sendai, Japan) 38
Loopy Belief Propagation forGraphical Model in Image Processing
Graphical model for image processing is representedin terms of the square lattice.Square lattice includes a lot of cycles.Belief propagation are applied to the calculation ofstatistical quantities as an approximate algorithm.
1 2 4
5
31 2 4
5
Every graph consisting of apixel and its fourneighbouring pixels can beregarded as a tree graph.
Loopy Belief Propagation
3
1 2
5
4!"
2z
21
3
1 October, 2007 ALT&DS2007 (Sendai, Japan) 39
Loopy Belief Propagation forGraphical Model in Image Processing
( )MM
rrr!"
( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )!!
!
"""
"""
"#
#
$
1 2
1
1151141132112
1151141132112
221,
,
z z
z
zMzMzMzz
zMzMzMfz
fM
42
3
1
5
Message Passing Rulein Loopy Belief Propagation
Averages, variances and covariances ofthe graphical model are expressed interms of messages.
3
1 2
5
4!"
2z
21
1 October, 2007 ALT&DS2007 (Sendai, Japan) 40
Loopy Belief Propagationfor Graphical Model in Image Processing
We have four kinds of message passing rules for each pixel.
Each massage passing rule includes3 incoming messages and 1 outgoing message
Visualizations ofPassing Messages
1 October, 2007 ALT&DS2007 (Sendai, Japan) 41
EM algorithm by means of Belief Propagation
Input
Output
LoopyBP EM Update Rule of Loopy BP
3
1 2
5
423!M
24!M
25!M
!=2z
212!M
1
EM Algorithm for Hyperparameter Estimation
( ) ( )( )
( )( ) ( )( ).,,,maxarg
1,1
,gttQ
tt
!"!"
!"
!"#
++
1 October, 2007 ALT&DS2007 (Sendai, Japan) 42
Probabilistic Image Processing byEM Algorithm and Loopy BP for GaussianGraphical Model( ) ( )( )
Formulation of probabilistic model for imageprocessing by means of conventional statisticalschemes has been summarized.Probabilistic image processing by usingGaussian graphical model has been shown as themost basic example.It has been explained how to construct a beliefpropagation algorithm for image processing.
S. S. GemanGeman and D. and D. GemanGeman (1986): IEEE Transactions on PAMI (1986): IEEE Transactions on PAMIImage Processing for Markov Random Fields (MRF)Image Processing for Markov Random Fields (MRF)((Simulated AnnealingSimulated Annealing, Line Fields), Line Fields)
J. Zhang (1992): IEEE Transactions on Signal ProcessingJ. Zhang (1992): IEEE Transactions on Signal ProcessingImage Processing in EM algorithm for Markov RandomImage Processing in EM algorithm for Markov RandomFields (MRF) (Fields (MRF) (Mean Field MethodsMean Field Methods))
K. Tanaka and T. Morita (1995): Physics Letters AK. Tanaka and T. Morita (1995): Physics Letters ACluster Variation MethodCluster Variation Method for MRF in Image Processing for MRF in Image Processing
Original ideas of some techniques, Simulated Annealing,Mean Field Methods and Belief Propagation, is oftenbased on the statistical mechanics.
Mathematical structure of Belief Propagation is equivalent toBethe Approximation and Cluster Variation Method (KikuchiMethod) which are ones of advanced mean field methods in thestatistical mechanics.
1 October, 2007 ALT&DS2007 (Sendai, Japan) 48
Statistical Mechanical Informatics forProbabilistic Information Processing
It has been suggested that statistical performanceestimations for probabilistic information processing areclosed to the spin glass theory.The computational techniques of spin glass theory has beenapplied to many problems in computer sciences.
Error Correcting Codes (Y. Kabashima and D. Saad: J. Phys.A, 2004, Topical Review).CDMA Multiuser Detection in Mobile Phone Communication(T. Tanaka: IEEE Information Theory, 2002).SAT Problems (Mezard, Parisi, Zecchina: Science, 2002).Image Processing (K. Tanaka: J. Phys. A, 2002, TopicalReview).