Markov Chain Sampling Methods for Dirichlet Process Mixture Models R.M. Neal Summarized by Joon Shik Kim 12.03.15.(Thu) Computational Models of Intelligence
Mar 30, 2015
Markov Chain Sampling Methods for Dirichlet Process Mixture Models
R.M. Neal
Summarized by Joon Shik Kim12.03.15.(Thu)
Computational Models of Intelligence
Abstract
• This article reviews Markov chain methods for sampling from the posterior distribu-tion of a Dirichlet process mixture model and presents two new classes of methods.
• One new approach is to make Metropolis-Hastings updates of the indicators specify-ing which mixture component is associ-ated with each observation, perhaps sup-plemented with a partial form of Gibbs sampling.
Chinese Restaurant Process (1/2)
Chinese Restaurant Process (2/2)
Introduction (1/2)
• Modeling a distribution as a mixture of simpler distribution is useful both as a nonparametric density estimation method and as a way of iden-tifying latent classes that can explain the depen-dencies observed between variables.
• Use of Dirichlet process mixture models has be-come computationally feasible with the devel-opment of Markov chain methods for sampling from the posterior distribution of the parameters of the component distribution and/or of the asso-ciations of mixture components with observa-tions.
Introduction (2/2)
• In this article, I present two new ap-proaches to Markov chain sampling.
• A very simple method for handling non-conjugate priors is to use Metropolis-Hast-ings updates with the conditional prior as the proposal distribution.
• A variation of this method may some-times sample more efficiently, particularly when combined with a partial form of Gibbs sampling.
Dirichlet Process Mixture Models (1/5)
• The basic model applies to data y1,…,yn which we regard as part of an indefinite exchangeable sequence, or equivalent, as being independently drawn from some unknown distribu-tion.
Dirichlet Process Mixture Models (2/5)
• We model the distribution from which the yi are drawn as a mixture of dis-tributions of the form F(θ), with the mixing distribution over θ given G. We let the prior for this mixing distri-bution be a Dirichlet process, with concentration parameter α and base distribution G0.
Dirichlet Process Mixture Models (3/5)
Dirichlet Process Mixture Models (4/5)
Dirichlet Process Mixture Models (5/5)
• If we let K go to infinity, the condi-tional probabilities reach the follow-ing limits:
Gibbs Sampling when Conjugate Priors are used (3/4)
Nested CRP
Day 1 Day 2 Day 3