Top Banner
Variational Methods, Belief Propagation, & Graphical Models Erik Sudderth Massachusetts Institute of Technology
36

Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Sep 08, 2018

Download

Documents

dangduong
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Variational Methods, BeliefPropagation, & Graphical Models

Erik SudderthMassachusetts Institute of Technology

Page 2: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 3: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Nearest-Neighbor Grids

Low Level Vision

• Image denoising

• Stereo

• Optical flow

• Shape from shading

• Superresolution

• Segmentation

unobserved or hidden variable local observation of

Page 4: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 5: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Other Graphical Models

Images removed due to copyright considerations.

Pictorial Structures Articulated Models

(Constellation Models)

Page 6: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

OutlineInference in Graphical Models

‹ Pairwise Markov random fields

‹ Belief propagation for trees

Variational Methods

‹ Mean field

‹ Bethe approximation & BP

Extensions of Belief Propagation

‹ Efficient message passing implementation

‹ Generalized BP

‹ Particle filters and nonparametric BP

Page 7: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 8: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 9: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 10: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 11: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 12: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 13: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Belief Propagation for Trees• Dynamic programming algorithm which exactly

computes all marginals

• On Markov chains, BP equivalent to alpha-beta or forward-backward algorithms for HMMs

• Sequential message schedules require each message to be updated only once

• Computational cost:number of nodes

discrete states for each node

Belief Prop:

Brute Force:

Page 14: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Inference for Graphs with Cycles• For graphs with cycles, the dynamic

programming BP derivation breaks

Junction Tree Algorithm

• Cluster nodes to break cycles

• Run BP on the tree of clusters

• Exact, but often intractable

Loopy Belief Propagation

• Iterate local BP message updateson the graph with cycles

• Hope beliefs converge

• Empirically, often very effective…

Page 15: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

A Brief History of Loopy BP• 1993: Turbo codes (and later LDPC codes,

rediscovered from Gallager’s 1963 thesis) revolutionize error correcting codes (Berrou et. al.)

• 1995-1997: Realization that turbo decoding algorithm is equivalent to loopy BP (MacKay & Neal)

• 1997-1999: Promising results in other domains, & theoretical analysis via computation trees (Weiss)

• 2000: Connection between loopy BP & variational approximations, using ideas from statistical physics (Yedidia, Freeman, & Weiss)

• 2001-2005: Many results interpreting, justifying, and extending loopy BP

Page 16: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

OutlineInference in Graphical Models

‹ Pairwise Markov random fields

‹ Belief propagation for trees

Variational Methods

‹ Mean field

‹ Bethe approximation & BP

Extensions of Belief Propagation

‹ Efficient message passing implementation

‹ Generalized BP

‹ Particle filters and nonparametric BP

Page 17: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 18: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 19: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 20: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 21: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 22: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 23: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Structured Mean Field

Structured Original Graph Naïve Mean Field

Mean Field

• Any subgraph for which inference is tractable leads to a mean field style approximation for which the update equations are tractable

Page 24: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 25: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 26: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 27: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 28: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Implications for Loopy BPBethe Free Energy is an Approximation

• BP may have multiple fixed points (non-convex)

• BP is not guaranteed to converge

• Few general guarantees on BP’s accuracy

Characterizations of BP Fixed Points

• All graphical models have at least one BP fixed point

• Stable fixed points are local minima of Bethe

• For graphs with cycles, BP is almost never exact

• As cycles grow long, BP becomes exact (coding)

Page 29: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

OutlineInference in Graphical Models

‹ Pairwise Markov random fields

‹ Belief propagation for trees

Variational Methods

‹ Mean field

‹ Bethe approximation & BP

Extensions of Belief Propagation

‹ Efficient message passing implementation

‹ Generalized BP

‹ Particle filters and nonparametric BP

Page 30: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 31: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 32: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Dynamic Quantization(Coughlan et. al., ECCV 2002 & 2004 CVPR GMBV workshop)

Images removed due to copyright considerations.

• Deformable template: State at each node is discretization of position and orientation of some point along the letter contour

• Rules for pruning unlikely states based on local evidence, and current message estimates, allow efficient, nearly globally optimal matching

Page 33: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Generalized Belief Propagationidi i(Yed a, Freeman, & We ss, NIPS 2000)

• Big idea: cluster nodes to break shortest cycles

• Non-overlapping clusters: exactly equivalent to loopy BP on the graph of cluster nodes

• Overlapping clusters: higher-order Kikuchi free energies ensure that information not over-counted

Page 34: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 35: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation
Page 36: Variational Methods, Belief Propagation, & Graphical Models · Variational Methods, Belief Propagation, & Graphical Models ... Belief Propagation for Trees ... Loopy Belief Propagation

Nonparametric Belief Propagation(Sudderth, Ihler, Freeman, & Willsky)

Belief Propagation Particle Filters• General graphs

ian

• Markov chains

• Discrete or Gauss • General potentials

Nonparametric BP

• General graphs

• General potentials


Related Documents
On the Power of Belief Propagation: A Constraint Propagation Perspective

On the Power of Belief Propagation: A Constraint Propagation...

Category: Documents
Efficient Belief Propagation for Higher Order Cliques ...tai/papers/CVIU.pdf · Efficient Belief Propagation for Higher Order Cliques Using Linear Constraint Nodes ... Loopy belief

Efficient Belief Propagation for Higher Order Cliques...

Category: Documents
Belief Propagation

Belief Propagation

Category: Documents
Bayesian Belief Propagation and Image Interpretation

Bayesian Belief Propagation and Image Interpretation

Category: Documents
Nonparametric Belief Propagation

Nonparametric Belief Propagation

Category: Documents
Nonparametric Belief Propagation and Facial Appearance Estimationpeople.csail.mit.edu/.../Nonparametic_Belief_Propagation.pdf · Nonparametric Belief Propagation and Facial Appearance

Nonparametric Belief Propagation and Facial Appearance...

Category: Documents
Stereo Matching Using Loopy Belief Propagation

Stereo Matching Using Loopy Belief Propagation

Category: Documents
Belief Propagation and its Generalizations Shane Oldenburger.

Belief Propagation and its Generalizations Shane...

Category: Documents
Physical Fluctuomatics 7th~10th Belief propagation

Physical Fluctuomatics 7th~10th Belief propagation

Category: Documents
Compressed Sensing Reconstruction via Belief Propagation

Compressed Sensing Reconstruction via Belief Propagation

Category: Documents
Multiple Frame Motion Inference Using Belief Propagation

Multiple Frame Motion Inference Using Belief Propagation

Category: Documents
Belief Propagation and its Generalizations

Belief Propagation and its Generalizations

Category: Documents