Mean-Field Theory and Its Applications In Computer Vision6 1.

Mean-Field Theory and Its Applications In Computer Vision6

1

Inference In Product Label Space

• Many problem requires jointly estimating labels in product label space

2

Black Box

SolverLeft Camera Image

Right Camera Image

Object Class Segmentation

Dense Stereo Reconstruction

Joint Object-Stereo Labelling• Computation complexity very high

• Graph-cuts based method takes almost 50 secs for 320x200 image size

• We propose mean-field based inference method

• Our method takes 2 secs for the same task

3

Joint stereo-object inference• Introduce two different set of variables

4

disparity variable

object variable

Messages exchanged between the variables

Joint stereo-object formulation

5

Unary Potential

• Weighted sum of object class, depth and joint potential

• Joint unary potential based on histograms of height

Joint stereo-object formulation

6

Pairwise Potential

• Object class and depth edges correlated

• We disregard the joint pairwise terms though

• Dense pairwise connection at both disparity variable and object variables

Joint stereo-object formulation

7

Higher Order Potential

• Use higher order terms only for object variables

Joint stereo-object updation

8

For object variables

Message from disparity variables to object variables

Joint stereo-object updation

9

For object variables

Filtering is done using permutohedral lattice based filtering strategy

Joint stereo-object updation

10

For disparity variables

Message from object variables to disparity variables

Joint stereo-object updation

11

For disparity variables

Filtering is done using domain transform based filtering strategy

Leuven dataset

12

Some of qualitative results

Leuven dataset

13Some of qualitative results

Algorithm Time (s) Object (% correct) Stereo (% correct)

GC + Range (1) 24.6 95.94 76.97

GC + Range (2) 49.9 95.94 77.31

GC + Range (3) 74.4 95.94 77.46

Extended CostVol 4.2 95.20 77.18

Dense + HO(PLBF)

3.1 95.24 78.89

Dense + HO(DTBF)

2.1 95.06 78.21

Dense + HO + CostVol + DTBF

6.3 94.98 79.00