Chinese Restaurants and Stick- Breaking: An Introduction to the Dirichlet Process Teg Grenager NLP Group Lunch February 24, 2005
Mar 30, 2015
Chinese Restaurants and Stick-Breaking: An Introduction to the Dirichlet Process
Teg GrenagerNLP Group LunchFebruary 24, 2005
Agenda
Motivation Mixture Models Dirichlet Process Gibbs Sampling Applications
Clustering
Goal: learn a partition of the data, such that: Data within classes are “similar” Classes are “different” from each other
Two very different approaches: Agglomerative: build up clusters by
iteratively sticking similar things together Mixture Model: learn a generative model
over the data, treating the classes as hidden variables
Agglomerative Clustering
Pros: Doesn’t need generative model (number of clusters, parametric distribution)
Cons: Ad-hoc, no probabilistic foundation, intractable for large data sets
Num Clusters Max Distance
19 518 517 516 815 814 813 812 811 910 99 98 107 106 105 104 123 122 151 16
20 0
Mixture Model Clustering
Examples: K-means, mixture of Gaussians, Naïve Bayes Pros: Sound probabilistic foundation, efficient even for
large data sets Cons: Requires generative model, including number of
clusters (mixture components)
Problem
1 2 3 4 5 6 7 8
Number of Clusters
Distance/Likelihood
Big Idea
Want to use a generative model, but don’t want to decide number of clusters in advance
Suggestion: put each datum in its own cluster Problem: probability of 2 clusters colliding is
zero under any density function, no “stickiness”
Solution: instead of a density function, use a statistical process where the probability of two clusters falling together is non-negative
Best of both worlds: stickiness with variable number of clusters
Finite Mixture Model
Gaussian
Naïve Bayes
c
x
c
x1 x2 xM
ci
xi N
p
ci
xij NM
p
Dirichlet Priors (Review)
A distribution over possible parameter vectors of the multinomial distribution
Thus values must lie in the k-dimensional simplex Beta distribution is the 2-parameter special case Expectation
A conjugate prior to the multinomial
Explicit formulation is ugly! xi
N
Infinite Mixture Model
ci
xi N
p
G0
Chinese Restaurant Process
DP Mixture Model
xi N
G
i
G0
ci
xi N
p
G0
Stick-breaking Process
G0
0 0.4 0.4
0.6 0.5 0.3
0.3 0.8 0.24
Properties of the DP
Let (,) be a measurable space, G0 be a probability measure on the space, and be a positive real number
A Dirichlet process is any distribution of a random probability measure G over (,) such that, for all finite partitions (A1,…,Ar) of ,
Draws G from DP are generally not distinct The number of distinct values grows with O(log n)
Infinite Exchangeability
In general, an infinite set of random variables is said to be infinitely exchangeable if for every finite subset {xi,…,xn} and for any permutation we have
Note that infinite exchangeability is not the same as being independent and identically distributed (i.i.d.)!
Using DeFinetti’s theorem, it is possible to show that our draws are infinitely exchangeable
Thus the mixture components may be sampled in any order
Mixture Model Inference
We want to find a clustering of the data: an assignment of values to the hidden class variable
Sometimes we also want the component parameters
In most finite mixture models, this can be found with EM
The Dirichlet process is a non-parametric prior, and doesn’t permit EM
We use Gibbs sampling instead
Gibbs Sampling 1
Algorithm 1: integrate out G, and sample the i
directly, conditioned on everything else This is inefficient, because we update cluster
information for one datum at a time
xi N
G
i
G0
xi N
i
G0
Gibbs Sampling 2
Algorithm 2: Reintroduce a cluster variable ci which takes on values that
are the names c of the clusters Store the parameters that are shared by all data in class c
in a new variable c
xi N
G
i
G0
xi N
ci
G0
c
Gibbs Sampling 2 (cont.)
Algorithm 2: For i = 1,…,N sample ci from
where H-i,c is the posterior distribution of c based on the prior G0 and all observations for which ji and cj=c
Repeat Works well Note: can also use variational methods (other than EM)
NLP Applications
Clustering Document clustering for topic, genre, sentiment,… Word clustering for POS, WSD, synonymy,… Topic clustering across documents (see Blei et. al.,
2004 and Teh et. al., 2004) Noun coreference: don’t know how many entities
there are Other identity uncertainty problems: deduping, etc. Grammar induction
Sequence modeling: the “infinite HMM” Topic segmentation (see Grenager et. al., 2005) Sequence models for POS tagging
Others?
Nested CRP
Day 1 Day 2 Day 3
Nested CRP (cont.)
To generate a document given a tree with L levels Choose a path from the root of the tree to a
leaf Draw a vector of topic mixing proportions
from an L-dimensional Dirichlet Generate the words in the document from a
mixture of the topics along the path, with mixing proportions
Nested CRP (cont.)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
References Seminal:
T.S. Ferguson. A Bayesian analysis of some nonparametric problems. Annals of Statistics 1:209-230, 1973.
C.E. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Annals of Statistics 2:1152-1174, 1974.
Foundational: M.D. Escobar and M. West. Bayesian density estimation and inference using
mixtures. Journal of the American Statistical Association, 90:577-588, 1995.
S.N. MacEachern and P. Muller. Estimating mixture of Dirichlet process models. Journal of Computational and Graphical Statistics, 7:223-238, 1998.
R.M. Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9:249-265, 2000.
C.E. Rasmussen. The Infinite Gaussian Mixture Model. NIPS, 2000. H. Ishwaran and L. James. Gibbs sampling methods for stick-breaking priors.
Journal of the American Statistical Association, 96:161-173, 2001. NLP:
D.M. Blei, T.L. Griffiths, M.I. Jordan, and J.B. Tenenbaum. Hierarchical topic models and the nested Chinese restaurant process. NIPS, 2004.
Y.W. Teh, M.I. Jordan, M.J. Beal, and D.M. Blei. Hierarchical Dirichlet processes. NIPS, 2004.