STICK-BREAKING CONSTRUCTIONS Patrick Dallaire June 10 th, 2011.

STICK-BREAKING CONSTRUCTIONSPatrick Dallaire

June 10th, 2011

Outline

Introduction of the Stick-Breaking process

Outline

Introduction of the Stick-Breaking process

Presentation of fundamental representation

Outline

Introduction of the Stick-Breaking process

Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Outline

Introduction of the Stick-Breaking process

Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Definition of the Beta process

Outline

Introduction of the Stick-Breaking process

Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Definition of the Beta process A Stick-Breaking construction of Beta

process

Outline

Introduction of the Stick-Breaking process

Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Definition of the Beta process A Stick-Breaking construction of Beta

process Conclusion and current work

The Stick-Breaking process

The Stick-Breaking process

Assume a stick of unit length

The Stick-Breaking process

Assume a stick of unit length

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining stick

is broken by sampling the proportion to cut How should we sample these proportions?

Beta random proportions

Let be the proportion to cut at iteration

Beta random proportions

Let be the proportion to cut at iteration

The remaining length can be expressed as

Beta random proportions

Let be the proportion to cut at iteration

The remaining length can be expressed as

Thus, the broken part is defined by

Beta random proportions

Let be the proportion to cut at iteration

The remaining length can be expressed as

Thus, the broken part is defined by

We first consider the case where

Beta distribution

The Beta distribution is a density function on

Parameters and control its shape

The Dirichlet process

The Dirichlet process

Dirichlet processes are often used to produce infinite mixture models

The Dirichlet process

Dirichlet processes are often used to produce infinite mixture models

Each observation belongs to one of the infinitely many components

The Dirichlet process

Dirichlet processes are often used to produce infinite mixture models

Each observation belongs to one of the infinitely many components

The model ensures that only a finite number of components have appreciable weight

The Dirichlet process

A Dirichlet process, , can be constructed according to a Stick-Breaking process

Where is the base distribution and is a unit mass at

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

The Pitman-Yor process

The Pitman-Yor process

A Pitman-Yor process, , can be constructed according to a Stick-Breaking process

Where and

Evolution of the Beta cuts

The parameter controls the speed at which the Beta distribution changes

Evolution of the Beta cuts

The parameter controls the speed at which the Beta distribution changes

The parameter determines initial shapes of the Beta distribution

Evolution of the Beta cuts

The parameter controls the speed at which the Beta distribution changes

The parameter determines initial shapes of the Beta distribution

When , there is no changes over time and its called a Dirichlet process

Evolution of the Beta cuts

The parameter controls the speed at which the Beta distribution changes

The parameter determines initial shapes of the Beta distribution

When , there is no changes over time and its called a Dirichlet process

MATLAB DEMO

The Indian Buffet process

The Indian Buffet process

The Indian Buffet process was initially used to represent latent features

The Indian Buffet process

The Indian Buffet process was initially used to represent latent features

Observations are generated according to a set of unknown hidden features

The Indian Buffet process

The Indian Buffet process was initially used to represent latent features

Observations are generated according to a set of unknown hidden features

The model ensure that only a finite number of features have appreciable probability

The Indian Buffet process

Recall the basic Stick-Breaking process

The Indian Buffet process

Recall the basic Stick-Breaking process

The Indian Buffet process

Recall the basic Stick-Breaking process Here, we only consider the remaining

parts

The Indian Buffet process

Recall the basic Stick-Breaking process Here, we only consider the remaining

parts

The Indian Buffet process

Recall the basic Stick-Breaking process Here, we only consider the remaining

parts Each value corresponds to a feature

probability of appearance

Summary

Summary

The Dirichlet process induces a probability over infinitely many classes

Summary

The Dirichlet process induces a probability over infinitely many classes

This is the underlying de Finetti mixing distribution of the Chinese restaurant process

De Finetti theorem

It states that the distribution of any infinitely exchangeable sequence can be written

where is the de Finetti mixing distribution

Summary

The Dirichlet process induces a probability over infinitely many classes

This is the underlying de Finetti mixing distribution of the Chinese restaurant process

The Indian Buffet process induces a probability over infinitely many features

Summary

The Dirichlet process induces a probability over infinitely many classes

This is the underlying de Finetti mixing distribution of the Chinese restaurant process

The Indian Buffet process induces a probability over infinitely many features

Its underlying de Finetti mixing distribution is the Beta process

The Beta process

The Beta process

This process

Beta with Stick-Breaking

The Beta distribution has a Stick-Breaking representation which allows to sample from

Beta with Stick-Breaking

The Beta distribution has a Stick-Breaking representation which allows to sample from

The construction is

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

The Beta distribution has a Stick-Breaking representation which allows to sample from

The construction is

The Beta process

A Beta process is defined as

as , and is a Beta process

Stick-Breaking the Beta process

The Stick-Breaking construction of the Beta process is such that

Stick-Breaking the Beta process

Expending the first terms

Conclusion

We briefly described various Stick-Breaking constructions for Bayesian nonparametric priors

These constructions help to understand the properties of each process

It also unveils connections among existing priors

The Stick-Breaking process might help to construct new priors

Current work

Applying a Stick-Breaking process to select the number of support points in a Gaussian process

Defining a stochastic process for unbounded random directed acyclic graph

Finding its underlying Stick-Breaking representation