Top Banner
Markov Random Fields Allows rich probabilistic models for images. But built in a local, modular way. Learn local relationships, get global effects out.
91

Markov Random Fields

Dec 31, 2015

Download

Documents

jeremy-boyle

Markov Random Fields. Allows rich probabilistic models for images. But built in a local, modular way. Learn local relationships, get global effects out. MRF nodes as pixels. Winkler, 1995, p. 32. MRF nodes as patches. image patches. scene patches. image. F ( x i , y i ). Y ( x i , x j ). - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Markov Random Fields

Markov Random Fields

• Allows rich probabilistic models for images.• But built in a local, modular way. Learn local

relationships, get global effects out.

Page 2: Markov Random Fields

MRF nodes as pixels

Winkler, 1995, p. 32

Page 3: Markov Random Fields

MRF nodes as patches

image patches

(xi, yi)

(xi, xj)

image

scene

scene patches

Page 4: Markov Random Fields

Network joint probability

scene

image

Scene-scenecompatibility

functionneighboringscene nodes

local observations

Image-scenecompatibility

function

i

iiji

ji yxxxZ

yxP ),(),(1

),(,

Page 5: Markov Random Fields

In order to use MRFs:

• Given observations y, and the parameters of the MRF, how infer the hidden variables, x?

• How learn the parameters of the MRF?

Page 6: Markov Random Fields

Outline of MRF section

• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Variational methods– Belief propagation– Graph cuts

• Vision applications of inference in MRF’s.• Learning MRF parameters.

– Iterative proportional fitting (IPF)

Page 7: Markov Random Fields

Gibbs Sampling and Simulated Annealing

• Gibbs sampling: – A way to generate random samples from a (potentially

very complicated) probability distribution.

• Simulated annealing:– A schedule for modifying the probability distribution so

that, at “zero temperature”, you draw samples only from the MAP solution.

Reference: Geman and Geman, IEEE PAMI 1984.

Page 8: Markov Random Fields

Sampling from a 1-d function

1. Discretize the density function

2. Compute distribution function from density function

)(kf

)(kF

)(xf

)(kf

3. Sampling

draw ~ U(0,1);

for k = 1 to n

if

break;

;

)(kF

kxx 0

Page 9: Markov Random Fields

Gibbs Sampling

x1

x2

),,,|(~ )()(3

)(21

)1(1

tK

ttt xxxxx

),,,|(~ )()(3

)1(12

)1(2

tK

ttt xxxxπx

),,|(~ )1(1

)1(1

)1(

tK

tK

tK xxxx

Slide by Ce Liu

Page 10: Markov Random Fields

Gibbs sampling and simulated annealing

Simulated annealing as you gradually lower the “temperature” of the probability distribution ultimately giving zero probability to all but the MAP estimate.

What’s good about it: finds global MAP solution.

What’s bad about it: takes forever. Gibbs sampling is in the inner loop…

Page 11: Markov Random Fields

Iterated conditional modes

• For each node:– Condition on all the neighbors– Find the mode– Repeat.

Described in: Winkler, 1995. Introduced by Besag in 1986.

Page 12: Markov Random Fields

Winkler, 1995

Page 13: Markov Random Fields

Variational methods

• Reference: Tommi Jaakkola’s tutorial on variational methods, http://www.ai.mit.edu/people/tommi/

• Example: mean field– For each node

• Calculate the expected value of the node, conditioned on the mean values of the neighbors.

Page 14: Markov Random Fields

Outline of MRF section

• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Variational methods– Belief propagation– Graph cuts

• Vision applications of inference in MRF’s.• Learning MRF parameters.

– Iterative proportional fitting (IPF)

Page 15: Markov Random Fields

),,,,,(sumsummean 3213211321

yyyxxxPxxxx

MMSE

y1

Derivation of belief propagation

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

Page 16: Markov Random Fields

),(),(sum

),(),(sum

),(mean

),(),(

),(),(

),(sumsummean

),,,,,(sumsummean

3233

2122

111

3233

2122

111

3213211

3

2

1

321

321

xxyx

xxyx

yxx

xxyx

xxyx

yxx

yyyxxxPx

x

x

xMMSE

xxxMMSE

xxxMMSE

The posterior factorizes

y1

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

Page 17: Markov Random Fields

Propagation rules

y1

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3),(),(sum

),(),(sum

),(mean

),(),(

),(),(

),(sumsummean

),,,,,(sumsummean

3233

2122

111

3233

2122

111

3213211

3

2

1

321

321

xxyx

xxyx

yxx

xxyx

xxyx

yxx

yyyxxxPx

x

x

xMMSE

xxxMMSE

xxxMMSE

Page 18: Markov Random Fields

Propagation rules

y1

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

),(),(sum

),(),(sum

),(mean

3233

2122

111

3

2

1

xxyx

xxyx

yxx

x

x

xMMSE

)( ),( ),(sum)( 23222211

21

2

xMyxxxxMx

Page 19: Markov Random Fields

Belief, and message updates

jii =

ijNk

jkjji

xi

ji xMxxxM

j \)(ij )(),( )(

j

)(

)( )(jNk

jkjjj xMxb

Page 20: Markov Random Fields

Optimal solution in a chain or tree:Belief Propagation

• “Do the right thing” Bayesian algorithm.

• For Gaussian random variables over time: Kalman filter.

• For hidden Markov models: forward/backward algorithm (and MAP variant is Viterbi).

Page 21: Markov Random Fields

No factorization with loops!

y1

x1

y2

x2

y3

x3

),(),(sum

),(),(sum

),(mean

3233

2122

111

3

2

1

xxyx

xxyx

yxx

x

x

xMMSE

31 ),( xx

Page 22: Markov Random Fields

Justification for running belief propagation in networks with loops

• Experimental results:

– Error-correcting codes

– Vision applications

• Theoretical results:

– For Gaussian processes, means are correct.

– Large neighborhood local maximum for MAP.

– Equivalent to Bethe approx. in statistical physics.

– Tree-weighted reparameterization

Weiss and Freeman, 2000

Yedidia, Freeman, and Weiss, 2000

Freeman and Pasztor, 1999;Frey, 2000

Kschischang and Frey, 1998;McEliece et al., 1998

Weiss and Freeman, 1999

Wainwright, Willsky, Jaakkola, 2001

Page 23: Markov Random Fields

Statistical mechanics interpretation

U - TS = Free energy

U = avg. energy =

T = temperature

S = entropy =

,...),(,...),( 2121 xxExxpstates

,...),(ln,...),( 2121 xxpxxpstates

Page 24: Markov Random Fields

Free energy formulation

Defining

then the probability distribution

that minimizes the F.E. is precisely

the true probability of the Markov network, )(),(,...),( 21 i

iiji

ijij xxxxxP

,...),( 21 xxP

TxxEjiij

jiexx /),(),( TxEii

iex /)()(

Page 25: Markov Random Fields

Approximating the Free Energy

Exact: Mean Field Theory: Bethe Approximation : Kikuchi Approximations:

)],...,,([ 21 NxxxpF)]([ ii xbF

)],(),([ jiijii xxbxbF

),....],(),,(),([ , kjiijkjiijii xxxbxxbxbF

Page 26: Markov Random Fields

Bethe Approximation

On tree-like lattices, exact formula:

i

qii

ijjiijN

ixpxxpxxxp 1

)(21 )]([),(),...,,(

)( ,

)),(ln),()(,(),(ij xx

jiijjiijjiijijiBethe

ji

xxbTxxExxbbbF

i x

iiiiiii

i

xbTxExbq ))(ln)()(()1(

Page 27: Markov Random Fields

Gibbs Free Energy

)}(),(){(

}1),({),(

)(

,)(

jjx

jiijjij

ijx

xxjiij

ijijijiBethe

xbxxbx

xxbbbF

ij

ji

Page 28: Markov Random Fields

Gibbs Free Energy

)}(),(){(

}1),({),(

)(

,)(

jjx

jiijjij

ijx

xxjiij

ijijijiBethe

xbxxbx

xxbbbF

ij

ji

Set derivative of Gibbs Free Energy w.r.t. bij, bi terms to zero:

)exp( )( )(

))(

exp( ),( ),(

)()(

Txkxb

T

xxxkxxb

iNjixij

iii

iijjiijjiij

Page 29: Markov Random Fields

Belief Propagation = Bethe

ix

jiijjj xxbxb ),()(

)( jij xLagrange multipliers

enforce the constraints

Bethe stationary conditions = message update rules

ijNk

jkjjij xMTx

\)(

)(ln)(with

Page 30: Markov Random Fields

Region marginal probabilities

)()(),( ),(

)()( )(

\)(\)(

)(

ijNkj

kj

jiNki

kijijiij

iNki

kiiii

xMxMxxkxxb

xMxkxb

i

ji

Page 31: Markov Random Fields

Belief propagation equationsBelief propagation equations come from the

marginalization constraints.

jii

jii =

ijNk

jkjji

xi

ji xMxxxM

j \)(ij )(),( )(

Page 32: Markov Random Fields

Results from Bethe free energy analysis

• Fixed point of belief propagation equations iff. Bethe approximation stationary point.

• Belief propagation always has a fixed point.• Connection with variational methods for inference: both

minimize approximations to Free Energy,– variational: usually use primal variables.

– belief propagation: fixed pt. equs. for dual variables.

• Kikuchi approximations lead to more accurate belief propagation algorithms.

• Other Bethe free energy minimization algorithms—Yuille, Welling, etc.

Page 33: Markov Random Fields

Kikuchi message-update rules

i ji

=ji ji

lk

=

Groups of nodes send messages to other groups of nodes.

Update formessages

Update formessages

Typical choice for Kikuchi cluster.

Page 34: Markov Random Fields

Generalized belief propagationMarginal probabilities for nodes in one row

of a 10x10 spin glass

Page 35: Markov Random Fields

References on BP and GBP

• J. Pearl, 1985– classic

• Y. Weiss, NIPS 1998– Inspires application of BP to vision

• W. Freeman et al learning low-level vision, IJCV 1999– Applications in super-resolution, motion, shading/paint

discrimination• H. Shum et al, ECCV 2002

– Application to stereo• M. Wainwright, T. Jaakkola, A. Willsky

– Reparameterization version• J. Yedidia, AAAI 2000

– The clearest place to read about BP and GBP.

Page 36: Markov Random Fields

Graph cuts

• Algorithm: uses node label swaps or expansions as moves in the algorithm to reduce the energy. Swaps many labels at once, not just one at a time, as with ICM.

• Find which pixel labels to swap using min cut/max flow algorithms from network theory.

• Can offer bounds on optimality.• See Boykov, Veksler, Zabih, IEEE PAMI 23 (11)

Nov. 2001 (available on web).

Page 37: Markov Random Fields

Comparison of graph cuts and belief propagation

Comparison of Graph Cuts with Belief Propagation for Stereo, using IdenticalMRF Parameters, ICCV 2003.Marshall F. Tappen William T. Freeman

Page 38: Markov Random Fields

Ground truth, graph cuts, and belief propagation disparity solution energies

Page 39: Markov Random Fields

Graph cuts versus belief propagation

• Graph cuts consistently gave slightly lower energy solutions for that stereo-problem MRF, although BP ran faster, although there is now a faster graph cuts implementation than what we used…

• However, here’s why I still use Belief Propagation:– Works for any compatibility functions, not a restricted

set like graph cuts.– I find it very intuitive.– Extensions: sum-product algorithm computes MMSE,

and Generalized Belief Propagation gives you very accurate solutions, at a cost of time.

Page 40: Markov Random Fields

MAP versus MMSE

Page 41: Markov Random Fields

Show program comparing some methods on a simple MRF

testMRF.m

Page 42: Markov Random Fields

Outline of MRF section

• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Variational methods– Belief propagation– Graph cuts

• Vision applications of inference in MRF’s.• Learning MRF parameters.

– Iterative proportional fitting (IPF)

Page 43: Markov Random Fields

Vision applications of MRF’s

• Stereo

• Motion estimation

• Labelling shading and reflectance

• Many others…

Page 44: Markov Random Fields

Vision applications of MRF’s

• Stereo

• Motion estimation

• Labelling shading and reflectance

• Many others…

Page 45: Markov Random Fields

Motion applicationimage patches

image

scene

scene patches

Page 46: Markov Random Fields

What behavior should we see in a motion algorithm?

• Aperture problem

• Resolution through propagation of information

• Figure/ground discrimination

Page 47: Markov Random Fields

The aperture problem

Page 48: Markov Random Fields

The aperture problem

Page 49: Markov Random Fields

Program demo

Page 50: Markov Random Fields

Motion analysis: related work

• Markov network– Luettgen, Karl, Willsky and collaborators.

• Neural network or learning-based– Nowlan & T. J. Senjowski; Sereno.

• Optical flow analysis– Weiss & Adelson; Darrell & Pentland; Ju,

Black & Jepson; Simoncelli; Grzywacz & Yuille; Hildreth; Horn & Schunk; etc.

Page 51: Markov Random Fields

Motion estimation results (maxima of scene probability distributions displayed)

Initial guesses only show motion at edges.

Iterations 0 and 1

Inference:

Image data

Page 52: Markov Random Fields

Motion estimation results

Figure/ground still unresolved here.

(maxima of scene probability distributions displayed)

Iterations 2 and 3

Page 53: Markov Random Fields

Motion estimation results

Final result compares well with vector quantized true (uniform) velocities.

(maxima of scene probability distributions displayed)

Iterations 4 and 5

Page 54: Markov Random Fields

Vision applications of MRF’s

• Stereo

• Motion estimation

• Labelling shading and reflectance

• Many others…

Page 55: Markov Random Fields

Forming an Image

Surface (Height Map)

Illuminate the surface to get:

The shading image is the interaction of the shapeof the surface and the illumination

Shading Image

Page 56: Markov Random Fields

Painting the Surface

Scene

Add a reflectance pattern to the surface. Points inside the squares should reflect less light

Image

Page 57: Markov Random Fields

Goal

Image Shading Image Reflectance Image

Page 58: Markov Random Fields

Basic Steps1. Compute the x and y image derivatives2. Classify each derivative as being caused by

either shading or a reflectance change3. Set derivatives with the wrong label to zero. 4. Recover the intrinsic images by finding the least-

squares solution of the derivatives.

Original x derivative image Classify each derivative(White is reflectance)

Page 59: Markov Random Fields

Learning the Classifiers• Combine multiple classifiers into a strong classifier using

AdaBoost (Freund and Schapire)• Choose weak classifiers greedily similar to (Tieu and Viola

2000)• Train on synthetic images• Assume the light direction is from the right

Shading Training Set Reflectance Change Training Set

Page 60: Markov Random Fields

Using Both Color and Gray-Scale Information

Results withoutconsidering gray-scale

Page 61: Markov Random Fields

Some Areas of the Image Are Locally Ambiguous

Input

Shading Reflectance

Is the change here better explained as

or ?

Page 62: Markov Random Fields

Propagating Information• Can disambiguate areas by propagating

information from reliable areas of the image into ambiguous areas of the image

Page 63: Markov Random Fields

• Consider relationship between neighboring derivatives

• Use Generalized Belief Propagation to infer labels

Propagating Information

Page 64: Markov Random Fields

Setting Compatibilities

• Set compatibilities according to image contours– All derivatives along a

contour should have the same label

• Derivatives along an image contour strongly influence each other 0.5 1.0

1

1),( jxx

i

β=

Page 65: Markov Random Fields

Improvements Using Propagation

Input Image Reflectance ImageWith Propagation

Reflectance ImageWithout Propagation

Page 66: Markov Random Fields
Page 67: Markov Random Fields

(More Results)

Input Image Shading Image Reflectance Image

Page 68: Markov Random Fields
Page 69: Markov Random Fields
Page 70: Markov Random Fields

Outline of MRF section

• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated conditional modes (ICM)– Variational methods– Belief propagation– Graph cuts

• Vision applications of inference in MRF’s.• Learning MRF parameters.

– Iterative proportional fitting (IPF)

Page 71: Markov Random Fields

Learning MRF parameters, labeled data

Iterative proportional fitting lets you make a maximum likelihood estimate a joint distribution from observations of various marginal distributions.

Page 72: Markov Random Fields

True joint probability

Observed marginal distributions

Page 73: Markov Random Fields

Initial guess at joint probability

Page 74: Markov Random Fields

IPF update equation

Scale the previous iteration’s estimate for the joint probability by the ratio of the true to the predicted marginals.

Gives gradient ascent in the likelihood of the joint probability, given the observations of the marginals.

See: Michael Jordan’s book on graphical models

Page 75: Markov Random Fields

Convergence of to correct marginals by IPF algorithm

Page 76: Markov Random Fields

Convergence of to correct marginals by IPF algorithm

Page 77: Markov Random Fields

IPF results for this example: comparison of joint probabilities

Initial guess Final maximumentropy estimate

True joint probability

Page 78: Markov Random Fields

Application to MRF parameter estimation

• Can show that for the ML estimate of the clique potentials, c(xc), the empirical marginals equal the model marginals,

• This leads to the IPF update rule for c(xc)

• Performs coordinate ascent in the likelihood of the MRF parameters, given the observed data.

Reference: unpublished notes by Michael Jordan

Page 79: Markov Random Fields

More general graphical models than MRF grids

• In this course, we’ve studied Markov chains, and Markov random fields, but, of course, many other structures of probabilistic models are possible and useful in computer vision.

• For a nice on-line tutorial about Bayes nets, see Kevin Murphy’s tutorial in his web page.

Page 80: Markov Random Fields

“Top-down” information: a representation for image context

Images

80-dimensional representation

Credit: Antonio Torralba

Page 81: Markov Random Fields

“Bottom-up” information: labeled training data for object recognition.

•Hand-annotated 1200 frames of video from a wearable webcam •Trained detectors for 9 types of objects: bookshelf, desk,screen (frontal) , steps, building facade, etc.•100-200 positive patches, > 10,000 negative patches

Page 82: Markov Random Fields

Combining top-down with bottom-up: graphical model showing assumed

statistical relationships between variables

Scene category

Visual “gist” observations

Object class

Particular objects

Local image features

kitchen, office, lab, conference room, open area, corridor, elevator and street.

Page 83: Markov Random Fields

Categorization of new places

frame

Specific location

Location category

Indoor/outdoor

ICCV 2003 posterBy Torralba, Murphy, Freeman, and Rubin

Page 84: Markov Random Fields

Bottom-up detection: ROC curvesICCV 2003 posterBy Torralba, Murphy, Freeman, and Rubin

Page 85: Markov Random Fields

Generative/discriminative hybrids

• CMF’s: conditional Markov random fields.– Used in text analysis community.– Used in a vision application by [name?] and Hebert, from

CMU, as a poster in ICCV 2003.• The idea: an ordinary MRF models P(x, y). But

you may not care about what the distribution of the images, y, is.

• It might be simpler to model P(x|y), with this graphical model. It combines the structured modeling of a generative model with the power of discriminative training.

Page 86: Markov Random Fields

Conditional Markov Random FieldsAnother benefit of CMF’s: you can include long-

range dependencies in the model without it messing up inference by introducing many new loops.

x1 x2 x3

x1 x2 x3

y1 y2 y3

y1 y2 y3

Lots of interdependencies to deal with during inference

Many fewer interdependencies, because everything is conditioned on the image data.

Page 87: Markov Random Fields

2003 ICCV Marr prize winners

• This year, the winners were all in the subject area of vision and learning

• This is an exciting time to be working on these problems; researchers are making progress.

Page 88: Markov Random Fields

Afternoon companion class

• Learning and vision: discriminative methods,

taught by Paul Viola and Chris Bishop.

Page 89: Markov Random Fields

Course web page

For powerpoint slides, references, example code:

www.ai.mit.edu/people/wtf/learningvision

Page 90: Markov Random Fields

end

Page 91: Markov Random Fields

Markov chain Monte Carlo

Image Parsing: Unifying Segmentation, Detection, and Recognition

Zhuowen Tu, Xiangrong Chen, Alan L. Yuille, Song-Chun Zhu

See ICCV 2003 talk: