7/21/2019 1 http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 1/24
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 1/24
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 2/24
1. EM Algorithm
2. Mixture Model
3. EM Algorithm for Normal Mixture Model4. Simulation
5. Results
6. Conclusions
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 3/24
EM Algorithm
Expectation-Maximization (EM) algorithm is an iterative method for
maximum likelihood estimates of parameters in statistical models, w
model depends on unobserved latent variables.
Latent variable : not directly observed but are rather inferred from o
variables that are observed.
EM algorithm is useful in incomplete-data problems : Missing data, T
distributions, Censored observations, Random effects, Mixtures, etc
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 4/24
EM Algorithm
= (1, 2, ⋯ , ) : observed incomplete data
= (1, 2, ⋯ , ) : missing data (latent variable)
= (1, 2, ⋯ , ) : unknown parameters.
) = ) : incomplete likelihood of .
, ) = , ) : complete likelihood of and .
, ) : conditional distribution of given .
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 5/24
EM Algorithm
1. Expectation Step (E - Step) : Calculate
(−1), ) = (|,())[log , ].
2. Maximization Step (M – Step) : Choose
() = argmax
(−1), ).
3. Return to the E-Step unless a stopping criterion has been met such
() (−1) < .
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 6/24
, = ,
log = log , log , )↔
= , )
, )↔
(|,()) log = (|,())[log , ] (|,())[lo↔
log = (|,())[log , ] (|,())[log , ]↔
log = (−1), ) (−1), )↔
log = (−1), ) (−1), )↔
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 7/24
EM Algorithm
1. () (−1), ) ≥ −1 −1 , )
(−1), ) is increasing.
2. () (−1), ) ≤ (−1) (−1), )
(−1), ) is decreasing.
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 8/24
Mixture Model
Mixture model is a probabilistic model for representing the presence
subpopulations within an overall population, without requiring that
observed data set should identify the sub-population to which an in
observation belongs.
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 9/24
Mixture Model
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 10/24
Mixture Model
Basic definition of finite mixture model
() = ∑=1
()
,
1, 2, ⋯ , : mixing proportions or weights
1 , 2 , ⋯ , () : component densities
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 11/24
Basic definition of finite normal mixture model
, ) = ∑=1
, 2)
,
1, 2, ⋯ , : mixing proportions or weights
| , 2 : normal component densities
Normal Mixture Model
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 12/24
Normal mixture model
, = ∑=1
, 2)
,
= (1, 2, ⋯ , ) : observed data
= (1, ⋯ , , 12, ⋯ ,
2) : unknown parameters
= (1, ⋯ , ) : unknown mixing proportion
EM Algorithm for Normal Mixture Model
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 13/24
Normal mixture model (hierarchical)
, = ∑=1
, 2)
= 12
⋯
, = 1, ⋯ , ~ (1, 1, ⋯ , )
= (1 , 2
, ⋯ , ) : missing data (latent variable)
EM Algorithm for Normal Mixture Model
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 14/24
Normal mixture model
, , = , ) ) = ∑=1
, 2)
Complete likelihood of and
, , ) = ∏=1 , , = ∏=1 ∑=1 , 2)
Complete log-likelihood of and
log , , ) = ∑= ∑
=1
log [ , 2)]
EM Algorithm for Normal Mixture Model
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 15/24
Simulation
Define the measures for the evaluation of the asymptotic properties a
1. Bias of for the th component
= 1
∑=1
(
() )
2. Mean square error (MSE) of for the th component
= 1
∑=1
(
)2
,
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 16/24
Simulation
Simulate 2-5 components of normal data with the combination of ,
= 5, 10, 20, 50,100
2 = 1, 2, 5, 10, 20, 50,100
= 0.1, 0.2, ⋯ , 0.9
Fix the total size = 25, 50, 100, 200,500, 1000, 2000, 5000, 10000.
Repeat 1000 times.
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 17/24
Simulation
Since there are a large number of combination, combine these into
= ∑=1
[ 2 + 2 �2 2
]
, = 11 + ⋯ + , �2 = 112 + ⋯ +
2
Classify into small ( = 0~100), medium ( = 100~1000), large ( =
Small : mixture of normal data that overlap largely
Medium : mixture of normal data that have some overlap
Large : mixture of normal data that show only slight overlap
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 18/24
Simulation
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 19/24
Results
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 20/24
Results
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 21/24
Results
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 22/24
Results
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 23/24
Conclusions
EM algorithm gives reasonable solutions in an asymptotic unbiased
EM algorithm estimate seem to provide reasonable estimates of the
values.
7/21/2019 1
http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 24/24