Course: Neural Networks, Instructor: Professor L.Behera. -Joy Bhattacharjee, Department of ChE, IIT Kanpur. Johann Peter Gustav Lejeune Dirichlet.

Course: Neural Networks, Instructor: Professor L.Behera.

-Joy Bhattacharjee,Department of ChE,

IIT Kanpur.Johann Peter Gustav Lejeune Dirichlet

DIRICHLET PROCESS

What is Dirichlet Process ?The Dirichlet process is a stochastic process used in Bayesian nonparametricmodels of data, particularly in Dirichlet process mixture models (also known as infinite mixture models). It is a distribution over distributions, i.e. each draw from a Dirichlet process is itself a distribution. It is called a Dirichlet process because it has Dirichlet distributed finite dimensional marginal distributions.

What is Dirichlet Process ?The Dirichlet process is a stochastic process used in Bayesian nonparametricmodels of data, particularly in Dirichlet process mixture models (also known as infinite mixture models). It is a distribution over distributions, i.e. each draw from a Dirichlet process is itself a distribution. It is called a Dirichlet process because it has Dirichlet distributed finite dimensional marginal distributions.

What is Dirichlet Process ?The Dirichlet process is a stochastic process used in Bayesian nonparametricmodels of data, particularly in Dirichlet process mixture models (also known as infinite mixture models). It is a distribution over distributions, i.e. each draw from a Dirichlet process is itself a distribution. It is called a Dirichlet process because it has Dirichlet distributed finite dimensional marginal distributions.

Dirichlet Priors

• A distribution over possible parameter vectors of the multinomial distribution

• Thus values must lie in the k-dimensional simplex• Beta distribution is the 2-parameter special case• Expectation

• A conjugate prior to the multinomial

xi

N

What is Dirichlet Distribution ?

Methods to generate Dirichlet distribution :1. Polya’s Urn2. Stick Breaking3. Chinese Restaurant Problem

Samples from a DP

Dirichlet Distribution

Polya’s Urn scheme:

Suppose we want to generate a realization of Q Dir(α). To start, put i balls of color i for i = 1; 2; : : : ; k; in an urn. Note that i > 0 is not necessarily an integer, so wemay have a fractional or even an irrational number of balls of color i in our urn! At each iteration, draw one ball uniformly at random from the urn, and then place it back into the urn along with an additional ball of the same color. As we iterate this procedure more and more times, the proportions of balls of each color will converge to a pmf that is a sample from the distribution Dir(α).

Mathematical form:

Stick Breaking ProcessThe stick-breaking approach to generating a random vector with a Dir(α) distribution involves iteratively breaking a stick of length 1 into k pieces in such a way that the lengths of the k pieces follow a Dir(α) distribution. Following figure illustrates this process with simulation results.

Stick Breaking Process

G0

0 0.4 0.4

0.6 0.5 0.3

0.3 0.8 0.24

Chinese Restaurant Process

Chinese Restaurant Process

Nested CRP

• To generate a document given a tree with L levels– Choose a path from the root of the tree to a leaf– Draw a vector of topic mixing proportions from

an L-dimensional Dirichlet– Generate the words in the document from a

mixture of the topics along the path, with mixing proportions

Nested CRP

Day 1 Day 2 Day 3

Properties of the DP

• Let (,) be a measurable space, G0 be a probability measure on the space, and be a positive real number

• A Dirichlet process is any distribution of a random probability measure G over (,) such that, for all finite partitions (A1,…,Ar) of ,

• Draws G from DP are generally not distinct• The number of distinct values grows with O(log n)

•In general, an infinite set of random variables is said to be infinitely exchangeable if for every finite subset {xi,…,xn} and for any permutation we have

•Note that infinite exchangeability is not the same as being independent and identically distributed (i.i.d.)!•Using DeFinetti’s theorem, it is possible to show that our draws are infinitely exchangeable•Thus the mixture components may be sampled in any order.

Mixture Model Inference

• We want to find a clustering of the data: an assignment of values to the hidden class variable

• Sometimes we also want the component parameters

• In most finite mixture models, this can be found with EM

• The Dirichlet process is a non-parametric prior, and doesn’t permit EM

• We use Gibbs sampling instead

Finite mixture model

Infinite mixture model

DP Mixture model

Agglomerative Clustering

• Pros: Doesn’t need generative model (number of clusters, parametric distribution)

• Cons: Ad-hoc, no probabilistic foundation, intractable for large data sets

Num Clusters Max Distance

19 518 517 516 815 814 813 812 811 910 99 98 107 106 105 104 123 122 151 16

20 0

Mixture Model Clustering

• Examples: K-means, mixture of Gaussians, Naïve Bayes • Pros: Sound probabilistic foundation, efficient even for large

data sets• Cons: Requires generative model, including number of clusters

(mixture components)

Applications• Clustering in Natural Language Processing

– Document clustering for topic, genre, sentiment…– Word clustering for Part of Speech(POS), Word sense

disambiguation(WSD), synonymy…– Topic clustering across documents– Noun coreference: don’t know how many entities are there– Other identity uncertainty problems: deduping, etc.– Grammar induction

• Sequence modeling: the “infinite HMM”– Topic segmentation)– Sequence models for POS tagging

• Society modeling in public places• Unsupervised machine learning

References:•Bela A. Frigyik, Amol Kapila, and Maya R. Gupta , University of Washington,Seattle, UWEE Technical report : Introduction to Dirichlet distribution and related processes, report number UWEETR2010-0006.•Yee Whye Teh, University College London : Dirichlet Process•Khalid-El-Arini, Select Lab meeting, October 2006.•Teg Granager, Natural Language Processing, Stanford University : Introduction to Chinese Restaurant problem and Stick breaking scheme.•Wikipedia

Questions ?•Suggest some distributions that can use Dirichlet process to find classes.•What are the applications in finite mixture model?•Comment on: The DP of a cluster is also a Dirichlet distribution.