Lecture12 - Graph-based Segmentation

Graph-based SegmentationComputer VisionCS 543 / ECE 549 University of Illinois

Derek Hoiem02/25/10

Last class

Gestalt cues and principles of organization

Mean-shift segmentationGood general-purpose segmentation method Generally useful clustering, tracking technique

Watershed segmentationGood for hierarchical segmentationUse in combination with boundary prediction

Todays class

Treating the image as a graphNormalized cuts segmentationMRFs Graph cuts segmentation

Recap

Go over HW2 instructions

iImages as graphsFully-connected graphnode for every pixellink between every pair of pixels, p,qsimilarity wij for each linkjwijcSource: Seitz

Similarity matrixIncreasing sigma

Segmentation by Graph CutsBreak Graph into SegmentsDelete links that cross between segmentsEasiest to break links that have low cost (low similarity)similar pixels should be in the same segmentsdissimilar pixels should be in different segments

ABCSource: Seitz

Cuts in a graphLink Cutset of links whose removal makes a graph disconnectedcost of a cut:ABOne idea: Find minimum cutgives you a segmentationfast algorithms exist for doing thisSource: Seitz

But min cut is not always the best cut...

Cuts in a graphABNormalized Cuta cut penalizes large segmentsfix by normalizing for size of segments

volume(A) = sum of costs of all edges that touch ASource: Seitz

Recursive normalized cutsGiven an image or image sequence, set up a weighted graph: G=(V, E)Vertex for each pixelEdge weight for nearby pairs of pixels

Solve for eigenvectors with the smallest eigenvalues: (D W)y = DyUse the eigenvector with the second smallest eigenvalue to bipartition the graphNote: this is an approximation

4. Recursively repartition the segmented parts if necessary

http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdfDetails:

Normalized cuts results

Normalized cuts: Pro and conProsGeneric framework, can be used with many different features and affinity formulationsProvides regular segmentsConsNeed to chose number of segmentsHigh storage requirement and time complexityBias towards partitioning into equal segmentsUsageUse for oversegmentation when you want regular segments

Graph cuts segmentation

Markov Random FieldsNode yi: pixel labelEdge: constrained pairsCost to assign a label to each pixelCost to assign a pair of labels to connected pixels

Markov Random FieldsExample: label smoothing gridUnary potential 0 10 0 K1 K 0Pairwise Potential0: -logP(yi = 0 ; data)1: -logP(yi = 1 ; data)

Solving MRFs with graph cutsSource (Label 0)Sink (Label 1)Cost to assign to 0Cost to assign to 1Cost to split nodes

Solving MRFs with graph cutsSource (Label 0)Sink (Label 1)Cost to assign to 0Cost to assign to 1Cost to split nodes

Grab cuts and graph cutsUser InputResultMagic Wand (198?)Intelligent Scissors Mortensen and Barrett (1995)GrabCutRegionsBoundaryRegions & BoundarySource: Rother

Colour ModelGaussian Mixture Model (typically 5-8 components)Foreground & BackgroundBackgroundForegroundBackgroundGRGRIterated graph cutSource: Rother

Graph cuts Boykov and Jolly (2001)ImageSource: Rother

Graph cuts segmentationDefine graph usually 4-connected or 8-connectedDefine unary potentialsColor histogram or mixture of Gaussians for background and foreground

Define pairwise potentials

Apply graph cutsReturn to 2, using current labels to compute foreground, background models

Moderately straightforward examples GrabCut completes automatically

Difficult ExamplesCamouflage & Low ContrastHarder CaseFine structureInitial RectangleInitial Result

Using graph cuts for recognitionTextonBoost (Shotton et al. 2009 IJCV)

Using graph cuts for recognitionTextonBoost (Shotton et al. 2009 IJCV)Unary PotentialsAlpha Expansion Graph Cuts

Limits of graph cuts

Associative: edge potentials penalize different labels

If not associative, can sometimes clip potentials

Approximate for multilabelAlpha-expansion or alpha-beta swaps

Must satisfy

Graph cuts: Pros and ConsProsVery fast inferenceCan incorporate recognition or high-level priorsApplies to a wide range of problems (stereo, image labeling, recognition)ConsNot always applicable (associative only)Need unary terms (not used for generic segmentation)Use whenever applicable

Further reading and resources

Normalized cuts and image segmentation (Shi and Malik)http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf

N-cut implementation http://www.seas.upenn.edu/~timothee/software/ncut/ncut.html

Graph cutshttp://www.cs.cornell.edu/~rdz/graphcuts.htmlClassic paper: What Energy Functions can be Minimized via Graph Cuts? (Kolmogorov and Zabih, ECCV '02/PAMI '04)

Recap of Grouping and Fitting

Line detection and Hough transformCanny edge detector = smooth derivative thin threshold link

Generalized Hough transform = points vote for shape parameters

Straight line detector = canny + gradient orientations orientation binning linking check for straightness

Robust fitting and registrationKey algorithmRANSAC

ClusteringKey algorithmKmeans

EM and Mixture of GaussiansTutorials: http://www.cs.duke.edu/courses/spring04/cps196.1/.../EM/tomasiEM.pdfhttp://www-clmc.usc.edu/~adsouza/notes/mix_gauss.pdf

SegmentationMean-shift segmentationFlexible clustering method, good segmentation

Watershed segmentationHierarchical segmentation from soft boundaries

Normalized cutsProduces regular regionsSlow but good for oversegmentation

MRFs with Graph CutIncorporates foreground/background/object model and prefers to cut at image boundariesGood for interactive segmentation or recognition

Next section: RecognitionHow to recognizeSpecific object instancesFacesScenesObject categoriesMaterials

**Basic idea is to combine the information which was used in the 2 well known tools: MW and IS.MS select a region of similar colour according to your input. Fat pen and the boundary snapps to high contrast edges. Grabcut combines it. And this allows us to simplify the user interface considerably by *Color enbergy.Iterations have the effect of pulling them away; D is log likelyhood of the GMM.*Optimization engine we use Graph Cuts. By now everybody should know what graph cut is. IN case you missed it here is a very brief introduction.

To image. 3D view. First task to construct a graph. All pixels on a certain scanline. Just a few of them. Next step introduce artificial nodes. fgd and background. Edges contrast. Boundary is quite likely between black and white. Min Cut - Minimum edge strength.

*Moderately straightforward examples- after the user input automnatically*You might wonder when does it fail. 4 cases. Low contrats an edge not good visible