MLPI Lecture 3: Advanced Sampling Techniques

Lecture'3

Advanced(Sampling(TechniquesDahua%Lin

The$Chinese$University$of$Hong$Kong

1

Overview• Collapsed*Gibbs*Sampling

• Sampling*with*Auxiliary*Variables

• Slice*Sampling

• Simulated*Tempering*&*Parallel*Tempering

• Swendsen?Wang*Algorithm

• Hamiltonian*Monte*Carlo

2

Collapsed)Gibbs)Sampling

3

Mo#va#ng(Example

with

We#want#to#sample#from# .

4

Gibbs%Sampling

Draw% %where% :

with% .

5

Gibbs%Sampling%(cont'd)

Draw% :

• How%well%can%this%sampler%perform%when%?

6

Collapsed)Gibbs)Sampling

• Basic&idea:"replace"the"original"condi0onal"distribu0on"with"a"condi0onal"distribu0on"of"a"marginal(distribu.on,"o7en"called"a"reduced(condi.onal(distribu.on.

• Consider"the"example"above,"we"consider"a"marginal(distribu.on:

7

Collapsed)Gibbs)Sampling)(cont'd)

• Draw& ,&with& &marginalized&out,&as:

• Draw&

• Can&we&exchange&the&order&of&these&two&steps?&Why?

8

Basic&Guidelines

• Order%of%steps%ma-ers!

• Generally,*one*can*move*components*from*"being'sampled"*to*"being'condi0oned'on".

• replacing*outputs*with*intermediates*would*change*the*sta:onary*distribu:on.

• A*variable*can*be*updated*mul:ple*:mes*in*an*itera:on.

9

Why$do$collapsed$samplers$o/en$perform$be3er$than$full6fledged$Gibbs$samplers?

10

Rao$Blackwell+Theorem

Consider)an)example) )and)we)want)to)es1mate) .)Suppose)we)have)two)tractable)ways)to)do)so:

(1)$draw$ ,$and$compute

11

Rao$Blackwell+Theorem+(cont'd)

(2)$draw$ $where$ $is$the$marginal$distribu4on,$and$compute

• Both&are&correct.&By&Strong&LLN,&both& &and& &converge&to& &almost&surely.

• Which&one&is&be<er?&Can&you&jus@fy&your&answer?12

Rao$Blackwell+Theorem+(cont'd)• (Rao%Blackwell,Theorem)"Sample"variance"will"be"reduced"when"some"components"are"marginalized"out."With"the"se:ng"above,"we"have

• Generally,*reducing)sample)variance*would*also*lead*to*the*reduc3on*of*autocorrela2on*of*the*chain,*thus*improving*the*mixing*performance.

13

Sampling)with)Auxiliary)Variables• The%Rao$Blackwell$Theorem%suggests%that%in%order%to%achieve%be3er%performance,%one%should%try%to%marginalize%out%as%many%components%as%possible.

• However,%in%many%cases,%one%may%want%to%do%the%opposite,%that%is,%to%introduce%addi>onal%variables%to%facilitate%the%simula>ons.

• For%example,%when%the%target$distribu6on%is%mul6modal,%one%may%use%an%auxiliary%variable%to%help%the%chain%escape%from%local%traps.

14

Use$Auxiliary$Variables

• Specify)an)auxiliary)variable) )and)the)joint)distribu8on) )such)that)

)for)certain) .

• Design)a)chain)to)update) )using)the)M=H)algorithm)or)the)Gibbs)sampler.

• The)samples)of) )can)then)be)obtained)through)marginaliza)on)or)condi)oning.

15

Slice&Sampling

16

Slice&Sampler

• Sampling* *is*equivalent*to*sampling*uniformly*from*the*area*under* :*

.

• Gibbs*sampling*based*on*the*uniform*distribu;on*over* .*Each*itera;on*consists*of*two*steps:

• Given* ,*

• Given* ,*

17

Slice&Sampler&(Illustra0on)

18

Slice&Sampler&(Discussion)• Slice&sampler"can"mix"very"rapidly,"as"it"will"not"be"locally"trapped.

• Slice&sampler"is"o7en"nontrivial"to"implement"in"prac8ce."Drawing" "is"some8mes"very"difficult.

• For"distribu8ons"of"certain"forms,"which"have"an"easy&way"to"draw" ,"slice&sampling"is"good"strategy.

19

Simulated*Tempering

20

Gibbs%Measure

A"Gibbs%measure"is"a"probability"measure"with"a"density"in"the"following"form:

Here,% %is%called%the%energy&func*on,% %is%called%the%inverse&temperature,%and%the%normalizing%constant% %depends%on% .

21

Gibbs%Measure%(cont'd)

In#literature#of#MCMC#sampling,#we#o5en#parameterize#a#Gibbs#measure#using#the#temperature(parameter# ,#thus#

.

22

Tempered'MCMC

Typical(MCMC(methods(usually(rely(on(local%moves(to(explore(the(state(space.(What(is(the(problem?

23

Tempered'MCMC'(cont'd)

Local&traps&o+en&leads&to&very&poor&mixing.&Can&we&improve&this?

24

Simulated*Tempering

Suppose'we'intend'to'sample'from'

Basic&idea:!Augment!the!target!distribu0on!by!including!a!temperature(index( ,!with!joint!distribu0on!given!by

25

Simulated*Tempering*(cont'd)• We$only$collect$samples$at$the$lowest'temperature,$

.

• The$chain$mixes$much$faster$at$high$temperatures,$but$we$want$to$collect$samples$at$the$lowest$temperature.$So$we$have$to$constantly$switch$between$temperatures.

26

Simulated*Tempering*(Algorithm)

One$itera)on$of$Simulated*Tempering$has$two$steps:

• (Base&transi+on):#update# #at#the#same#temperature,#i.e.#holding# #fixed.

• (Temperature&switching):#with# #fixed,#propose##with# #such#that#

• Accept#the#change#with#probability#.

• Any#drawbacks?27

Simulated*Tempering*(Discussion)

• Set% .%Given% ,%we%should%set% %such%that%uphill%moves%from%( )%should%have%a%considerable%probability%of%being%accepted.

• Build%the%temperature(ladder%step%by%step%un?l%we%have%a%sufficiently%smooth%distribu?on%at%the%top.

• The%?me%spent%on%the%base%level% %is%around%.%If%we%have%too%many%levels,%only%a%very%

small%por?on%of%samples%can%be%used.

28

Simulated*Tempering*(Discussion)• All$temperature$levels$play$an$important$role.$So$it$is$desirable$to$spend$comparable$amount$of$8me$at$

each$level.$Se:ng$ $for$each$ ,$we$have

• The%normalzing%constants% %are%typically%unknown%and%es8ma8ng%them%is%very%difficult%and%expensive.

29

Parallel&Tempering

(Basic'idea)!rather!than!jumping!between!temperatures,!it!simultaneously!simulate!mul3ple!chains,!each!at!a!temperature!level! ,!called!a!replica,!and!constantly!swap!samples!between!replicas.

30

Parallel&Tempering&(Algorithm)

Each%itera*on%consists%of%the%following%steps:

• (Parallel'update):"simulate"each"replica"with"its"own"transi2on"kernel

• (Replica'exchange):"propose"to"swap"states"between"two"replicas"(say"the" 7th"and" 7th,"where"

):

31

Parallel&Tempering&(Algorithm)• The%proposal%is%accepted%with%probability%

,%where

• We$collect$samples$from$the$base$replica$(the$one$with$ ).

• Why$does$this$algorithm$produce$the$desired$distribu;on?

32

Parallel&Tempering&(Jus1fica1on)

Let$ .$We$define

Obviously,+the+step+of+parallel&update+preserves+the+invariant+distribu5on+ .

33

Parallel&Tempering&(Jus1fica1on)

Note%that%the%step%of%replica(exchange%is%symmetric,%i.e.%the%probabili0es%of%going%up%and%down%are%equal,%then%according%to%the%Metropolis(algorithm,%we%have%

%with

34

Parallel&Tempering&(Discussion)• It$is$efficient$and$very$easy$to$implement,$especially$in$a$parallel$compu6ng$environment.

• It$is$o9en$an$art$instead$of$a$technique$to$tune$a$parallel$tempering$system.

• The$parallel-tempering$is$a$special$case$of$a$large$family$of$MCMC$methods$called$Extended-Ensemble-Monte-Carlo,$which$involves$a$collec6on$of$parallel$Markov$chains$and$the$simula6on$switches$between$these$them.

35

Swendsen'Wang+Algorithm

The$Swendsen'Wang$algorithm$(R.-Swendsen$and$J.-Wang,$1987)$is$an$efficient$Gibbs$sampling$algorithm$for$sampling$from$the$extended-Ising-model.

36

Standard'Ising'Model

The$standard$Ising&model$is$defined$as

where% %for%each% %is%called%a%spin,%and%.

• Gibbs&sampling&is&extremely&slow,&especially&when&the&temperature&is&low.

37

Extended'Ising'Model• We$extend$the$model$by$introducing$addi5onal$bond%variables$ ,$each$for$an$edge.$Each$bond$has$two$states:$ $indica5ng$connected$and$ $indica5ng$disconnected.

• We$define$a$joint$distribu5on$that$couples$the$spins$and$bonds,

38

Extended'Ising'Model'(cont'd)

Here,% %is%described%as%below:

• When& ,& &for&every&se.ng&of&

• When& ,&

39

Extended'Ising'Model'(cont'd)

With%this%se(ng,% %can%be%wri1en%as:

where% :

• when& ,& &must&be&

• when& ,& &is&set&to&zero&with&probability&.

40

Swendsen'Wang+Algorithm

!Each!itera*on!consists!of!two!steps:

• (Clustering):"condi(oned"on"the"spins" ,"draw"the"bonds" "independenly."For"an"edge"

:

• If" ,"set"

• If" ,"set" "with"probability""or" "otherwise.

41

Swendsen'Wang+Algorithm

• (Swapping):"condi(oned"on"the"bonds" ,"draw"the"spins" .

• For"each"connected"component,"draw" "or" "with"equal"chance,"and"assign"the"resultant"value"to"all"nodes"in"the"component.

42

Swendsen'Wang+Algorithm+(Illustra7on)In the case of a rectangular grid, this Gibbs sampling algorithm mixes very rapidly.

The following figures illustrate Gibbs sampling. Spin states up and down areshown by filled and empty circles. Bond states 1 and 0 are shown by thick lines andthin dotted lines. We start from a state with five connected components. (Rememberthat isolated spins count as connected components, albeit of size 1.)

First, let’s update the bonds The forbidden bonds are highlighted

Bonds are forbidden from forming wherever the two adjacent spins are in oppositestates. The bonds that are not forbidden are set to the 1 state with probability p.

After updating the bonds Now we update spins Update bonds again

1.2 Other properties of the extended model

We already mentioned that the partition function Z is the same as that of the Isingmodel.

The marginal P (x) is correct, because when we sum the factor gm over dm, we getfm. Summing over dm is easy because it appears in only one factor.

OK, we’ve summed out d and obtained the Ising model. What if we sum out x?The marginal P (d) is called the random cluster model. Summing over x for given

d, all factors are constants. The number of states is 2number of clusters. Thus

P (d) =1

Z̃

!

m

"

pdm(1 − p)1−dm

#

2c(d) (10)

where c(d) is the number of connected components in the state d. Isolated spinswhose neighbouring bonds are all zero count as single connected components.

The random cluster model can be generalized by replacing the number 2 by aparameter q:

P (q)(d) =!

m

"

pdm(1 − p)1−dm

#

qc(d) (11)

The random cluster model can be simulated directly, just as the Ising model canbe simulated directly; but the S–W method, augmenting the bonds with spins, isprobably the most efficient way to simulate the model. For integer values of q, theappropriate spin system is the ‘Potts model’, the generalization of the Ising modelfrom 2 spin states to q.

In the case of a rectangular grid, this Gibbs sampling algorithm mixes very rapidly.The following figures illustrate Gibbs sampling. Spin states up and down are

shown by filled and empty circles. Bond states 1 and 0 are shown by thick lines andthin dotted lines. We start from a state with five connected components. (Rememberthat isolated spins count as connected components, albeit of size 1.)

First, let’s update the bonds The forbidden bonds are highlighted

Bonds are forbidden from forming wherever the two adjacent spins are in oppositestates. The bonds that are not forbidden are set to the 1 state with probability p.

After updating the bonds Now we update spins Update bonds again

1.2 Other properties of the extended model

We already mentioned that the partition function Z is the same as that of the Isingmodel.

The marginal P (x) is correct, because when we sum the factor gm over dm, we getfm. Summing over dm is easy because it appears in only one factor.

OK, we’ve summed out d and obtained the Ising model. What if we sum out x?The marginal P (d) is called the random cluster model. Summing over x for given

d, all factors are constants. The number of states is 2number of clusters. Thus

P (d) =1

Z̃

!

m

"

pdm(1 − p)1−dm

#

2c(d) (10)

where c(d) is the number of connected components in the state d. Isolated spinswhose neighbouring bonds are all zero count as single connected components.

The random cluster model can be generalized by replacing the number 2 by aparameter q:

P (q)(d) =!

m

"

pdm(1 − p)1−dm

#

qc(d) (11)

The random cluster model can be simulated directly, just as the Ising model canbe simulated directly; but the S–W method, augmenting the bonds with spins, isprobably the most efficient way to simulate the model. For integer values of q, theappropriate spin system is the ‘Potts model’, the generalization of the Ising modelfrom 2 spin states to q.

43

Swendsen'Wang+Algorithm+(Discussion)

• When& &is&large,& &has&a&high&probability&of&being&set&to&one,&i.e.& &and& &are&likely&to&be&connected.

• Experiments&show&that&the&Swendsen)Wang&algorithm&mixes&very&rapidly,&especially&for&rectangular&grids.

• Can&you&provide&an&intui?ve&explana?on?

44

Swendsen'Wang+Algorithm+(Discussion)

• The%Swendsen'Wang%algorithm%can%be%generalized%to%Po4s%models%(nodes%can%take%values%from%a%finite%set).

• The%Swendsen'Wang%algorithm%has%been%widely%used%in%image%analysis%applicaAons,%e.g.%image%segmentaAon%(in%this%case,%it%is%called%Swendsen'Wang,cut).

45

Hamiltonian)Monte)Carlo• An$MCMC$method$based$on$Hamiltonian)Dynamics.$It$was$originally$devised$for$molecular)simula1on

• In$1987,$a$seminal$paper$by$Duane$et)al$unifies$MCMC$and$molecular$dynamics.$They$called$it$Hybrid)Monte)Carlo,$which$abbreviates$to$HMC

• In$many$arEcles,$people$call$it$Hamiltonian)Monte)Carlo,$as$this$name$is$considered$to$be$more$specific$and$informaEve,$and$it$retains$the$same$abbreviaEon$"HMC".

46

Mo#va#ng(Example:(Free(Fall

47

Mo#va#ng(Example:(Free(Fall

• The%change%of%momentum% %is%caused%by%the%accumula5on/release%of%the%poten(al+energy:

• The%change%of%loca-on% %is%caused%by%velocity,%the%deriva-ve%of%kinema-c.energy%w.r.t.%the%momentum:

48

Hamiltonian)Dynamics• Hamiltonian)Dynamics"is"a"generalized"theory"of"the"classical)mechanics,"which"provides"a"elegant"and"flexible"abstrac:on"of"a"dynamic"system"in"physics.

• In"Hamiltonian"Dynamics,"a"physical"system"is"described"by" ,"where" "and" "are"respec:vely"the"posi1on"and"momentum"of"the" @th"en:ty.

49

Hamilton's+Equa/ons

The$dynamics$of$the$system$is$characterized$by$the$Hamilton's+Equa/ons:

Here,% %is%called%the%Hamiltonian,%which%can%be%interpreted%as%the%total)energy%of%the%system.

50

Hamilton's+Equa/ons+(cont'd)

• The%Hamiltonian% %is%o)en%formulated%as%the%sum%of%the%poten+al,energy% %and%the%kine+c,energy% :

• With&this&se)ng,&the&Hamilton's+Equa/ons&become:

51

Conserva)on*of*Hamiltonian

The$Hamiltonian$is$conserved,$i.e.,$it$is$invariant$over$,me:

Intui&vely,,this,reflects,the,law$of$energy$conserva/on.

52

Hamiltonian)Reversibility• The%Hamiltonian)dynamics%is%reversible

• Let%the%ini+al%states%be% %and%the%states%at%+me% %be% .%Then,%it%we%reverse%the%process,%star+ng%at% ,%then%the%states%at%+me% %would%be% .

• In%the%context%of%MCMC,%this%leads%to%the%reversibility%of%the%underlying%chain.

53

Simula'on*of*Hamiltonian*Dynamics

A"natural"idea"to"simulate"Hamiltonian)dynamics"is"to"use"Euler's)method"over"discre1zed"1me"steps:

Is#this#a#good#method?

54

Leapfrog)Method

Be#er%results%can%be%obtained%with%leapfrog:

More%importantly,%the%leapfrog%update%is%reversible.55

Leapfrog)Method)(cont'd)

56

Example

Consider)a)Hamiltonian)system:

Write&down&the&Hamilton's+Equa/ons:

Derive&the&solu-on:

57

Example((Simula-on)

58

Hamiltonian)Monte)Carlo

(Basic'idea):!Consider!the!poten&al)energy!as!the!Gibbs)energy,!and!introduce!the!"momentums"!as!auxiliary)variables!to!control!the!dynamics.

59

Hamiltonian)Monte)Carlo)(cont'd)

Suppose'the'target&distribu,on'is'

,'then'we'form'an'augmented&

distribu,on'as

Here,%the%loca%ons% %represent%the%variables%of%interest,%and%the%momentums% %control%the%dynamics%of%simula7on.

60

Hamiltonian)Monte)Carlo)(cont'd)

In#prac(ce,#the#kine%c#energy#is#o2en#formalized#as

61

Hamiltonian)Monte)Carlo)(Algorithm)

Each%itera*on%of%HMC%comprises%two%steps:

• Gibbs%update:#sample#the#momentums# #from#the#Gaussian#prior#given#by

62

Hamiltonian)Monte)Carlo)(Algorithm)• Metropolis*update:#using#Hamiltonian#dynamics#to#propose#a#new#state.#Star8ng#from# ,#simulate#the#dynamic#system#with#the#leapfrog#method#for# #steps#with#step<size# ,#which#yields# .#The#proposed#state#is#accepted#with#probability:

63

HMC$(Discussion)• If$the$simula.on$is$exact,$we$will$have$

,$and$thus$the$proposed$state$should$always$be$accepted.$

• In$prac.ce,$there$can$be$some$devia.on$due$to$discre.za.on,$we$have$to$use$the$Metropolis$rule$to$guarantee$the$correctness.

64

HMC$(Discussion)

• HMC%has%a%high%acceptance%rate%while%allowing%large%moves%along%less6constrained%direc8ons%at%each%itera8on.

• This%is%a%key%advantage%as%compared%to%random'walk%proposals,%which,%in%order%to%maintain%a%reasonably%high%acceptance%rate,%has%to%keep%a%very%small%step%size,%resul8ng%in%substan8al%correla8on%between%consecu8ve%samples.

65

Tuning&HMC• For%efficient%simula1on,%it%is%important%to%choose%appropriate%values%for%both%the%leapfrog%step%size% %and%the%number%of%leapfrog%steps%per%itera1on% .

• Tuning%HMC%(and%actually%many%generic%sampling%methods)%oCen%requires%preliminary*runs%with%different%trial%seGngs%and%different%ini1al%values,%as%well%as%careful%analysis%of%the%energy%trajectories.

66

Tuning&HMC&(cont'd)

• For%most%cases,% %and% %can%be%tuned%independently.

• Too%small%a%stepsize%would%waste%computa8on%8me,%while%large%stepsize%would%cause%unstable%simula8on,%and%thus%low%acceptance%rate.

• One%should%choose% %such%that%the%energy%trajectory%is%stable%and%the%acceptance%rate%is%maintained%at%a%reasonably%high%level.

• One%should%choose% %such%that%back@and@forth%movement%of%the%states%can%be%observed.

67

Generic'Sampling'Systems

A"number"of"so,ware"systems"are"available"for"sampling"from"models"specified"by"the"user

• WinBUGS:*based*on*BUGS*(Bayesian*inference*Using*Gibbs*Sampling).

• provide*a*friendly*language*for*user*to*specify*the*model

• Running*only*on*Windows

• Note:*The*development*has*stopped*since*2007.68

Generic'Sampling'Systems'(cont'd)• JAGS:'"Just'Another'Gibbs'Sampler"

• Cross8pla9orm'support

• Use'a'dialect'of'BUGS

• Extensible:'allow'users'to'write'customized'funcCons,'distribuCons,'and'samplers

69

Generic'Sampling'Systems'(cont'd)• Stan:'"Sampling'Through'Adap5ve'Neighborhoods"

• Core'wri=en'in'C++,'and'ports'available'in'Python,'R,'Matlab,'and'Julia

• A'user'friendly'language'for'model'specifica5on

• Use'Hamiltonian'Monte'Carlo'(HMC)'and'No'ULTurn'Samplers'(NUTS)'as'core'algorithm

• Open'source'(GPLv3'licensed)'and'under'ac5ve'development'on'Github

70

Stan%Exampledata { int<lower=0> N; vector[N] x; vector[N] y;}parameters { real alpha; real beta; real<lower=0> sigma;}model { for (n in 1:N) y[n] ~ normal(alpha + beta * x[n], sigma);}

71

Generic'Sampling'System'vs.'Dedicated'Algorithms

Generic' Dedicated'

Easy%to%use% Require%knowledge%and%experience%

High%produc9vity% Time=consuming%to%develop%

Slow% O@en%remarkably%more%efficient%

Limited%flexibility% Necessary%for%many%new%models%

72