Page 1
A Variational Bayesian Inference Framework for
Multiview Depth Image Enhancement
Pravin Kumar Rana, Jalil Taghia, and Markus Flierl
School of Electrical Engineering
KTH Royal Institute of Technology
Stockholm, Sweden
December 10, 2012
IEEE International Symposium on Multimedia 2012
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 2
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement
Background and motivation
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 3
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 1
Free-viewpoint television
Multiview video imagery
Background Motivation Enhancement framework Experimental results Conclusions Future directions
User Display
Page 4
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 1
Free-viewpoint television
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Multiview video imagery
User Display
Page 5
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 1
Free-viewpoint television
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Multiview video imagery
User Display
Page 6
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 1
Free-viewpoint television
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Multiview video imagery
User Display
Page 7
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 1
Free-viewpoint television
Multiview video imagery
Background Motivation Enhancement framework Experimental results Conclusions Future directions
User Display
Virtual camera Virtual view
Page 8
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 2
Depth image based rendering
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Multiview video imagery
Virtual camera Virtual view
Page 9
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 2
Depth image based rendering
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Depth pixels represent shortest
distance between object points and
the camera plane
• To be estimated from multiview
imagery Depth image
Multiview video imagery
Virtual camera Virtual view
Near
Far
Page 10
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 2
Depth image based rendering
Background Motivation Enhancement framework Experimental results Conclusions Future directions
?
• Depth pixels represent shortest
distance between object points and
the camera plane
• To be estimated from multiview
imagery Depth image
3D warping
Multiview video imagery
Virtual camera Virtual view
Near
Far
Page 11
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 3
Common practice
Estimated depth map
Depth estimation
Depth estimation
MPEG Depth Estimation Reference Software
View (n-2) View (n-1) View (n) View (n+1) View (n+2) View (n+3)
View (n-1) View 3 View 4
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 12
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 3
Common practice
Estimated depth map
Depth estimation
Estimated depth map
Depth estimation
Depth estimation
MPEG Depth Estimation Reference Software
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-2) View (n-1) View (n) View (n+1) View (n+2) View (n+3)
View (n-1) View 3 View 4 View (n+2)
Page 13
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 4
Problem: Inter-view depth inconsistency
View (n-1)
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 14
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 4
Problem: Inter-view depth inconsistency
View (n-1) View (n)
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 15
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 4
Problem: Inter-view depth inconsistency
View (n-1) View (n) View (n+1)
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 16
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 4
Problem: Inter-view depth inconsistency
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-1) View (n) View (n+1)
Page 17
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 4
Problem: Inter-view depth inconsistency
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-1) View (n) View (n+1)
Page 18
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 4
Problem: Inter-view depth inconsistency
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-1) View (n) View (n+1)
Page 19
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 4
Problem: Inter-view depth inconsistency
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-1) View (n) View (n+1)
Note: we assume a 1D-parallel camera arrangement
Page 20
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 5
Prior work on depth enhancement
1. Existing methods warp depth images from multiple viewpoints to a common viewpoint for
spatial alignment ([2], [3])
2. Warping errors due to the discrete values in depth maps affects enhancement algorithms
negatively
Spatial alignment
3D warping
[2] P. K. Rana and M. Flierl, “Depth consistency testing for improved view interpolation,” IEEE Int. Workshop MMSP, 2010.
[3] E. Ekmekcioglu, V. Velisavljevic, and S. Worrall, “Content adaptive enhancement of multi-view depth maps for free viewpoint video,” IEEE J. Sel. Topics Signal Process., 2011.
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (1) View (2) View (n)
View (m)
Page 21
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement
New depth enhancement framework
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 22
• Concatenation of view imagery
• Multiview color classification
• Multiview depth classification
• Depth image enhancement
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 6
Overview of new depth enhancement framework
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 23
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 7
Concatenation of view imagery
• The captured MVV imagery of the scene has inherent inter-view similarity
• To have a unique model for the captured natural scene,
The MVV inter-view similarity is exploited by concatenating views from multiple viewpoints
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-1) View (n) View (n+1) View (n+2)
Page 24
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 7
Concatenation of view imagery
• The captured MVV imagery of the scene has inherent inter-view similarity
• To have a unique model for the captured natural scene,
The MVV inter-view similarity is exploited by concatenating views from multiple viewpoints
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-1) View (n) View (n+1) View (n+2)
View (n-1) View (n) View (n+1) View (n+2)
Page 25
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 8
Multiview color classification
Gaussian mixture model with variational Bayes inference
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.
• The goal of classification is to partition an image into regions each of which has a reasonably
homogeneous visual appearance
• Usually, classification algorithm, such as expectation-maximization for Gaussian mixtures,
suffers from two main drawbacks:
– model over-fitting and
– the number of clusters has to be known, (similar to the K-means algorithm)
• The Gaussian mixture model is used with variational Bayes inference [4] because
– no model over-fitting and
– the number of clusters is treated as a random variable
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 26
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 9
Multiview color classification
Gaussian mixture model with variational Bayes inference
RGB
3 X (NXHXW)
N views
Initial number
of clusters
50 35
Final number
of clusters
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View (n-1) View (n) View (n+1) View (n+2)
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.
Page 27
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 10
Multiview color classification
Example: Newspaper
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Color classification input
Color clusters
Page 28
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 10
Multiview color classification
Example: Newspaper
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Color classification input
Color clusters
Page 29
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 10
Multiview color classification
Example: Newspaper
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Color classification input
Color clusters
Page 30
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 11
Multiview depth classification
Exploiting the per-pixel association between color and depth
Background Motivation Enhancement framework Experimental results Conclusions Future directions
View image Depth image
Concatenated view imagery Concatenated depth imagery
Page 31
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 12
Multiview depth classification
Example: Newspaper
Depth clusters Color clusters
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 32
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 13
Multiview depth enhancement
Difference between color and depth clusters
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Members have similar colors pixels Members may have different depth values
Page 33
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 13
Multiview depth enhancement
• Why?
– Due to foreground and background depth difference
– Due to inter-view inconsistency
Difference between color and depth clusters
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Members have similar colors pixels Members may have different depth values
Page 34
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 14
Multiview depth enhancement
Members have similar colors pixels Members may have different depth values
• Why?
– Due to foreground and background depth difference
– Due to inter-view inconsistency
• Our approach: K-means sub-clustering
– Computationally fast
– Assigns the mean to depth pixels irrespective of the originating viewpoints
– Usually, Bayesian approaches imply higher computational complexity
Difference between color and depth clusters
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 35
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 15
Multiview depth enhancement
Example: Balloons
MPEG depth maps
Enhanced depth maps
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 36
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 15
Multiview depth enhancement
Example: Balloons
MPEG depth maps
Enhanced depth maps
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 37
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 15
Multiview depth enhancement
Example: Balloons
MPEG depth maps
Enhanced depth maps
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 38
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 16
Experimental setup
MPEG 3DTV multiview data set
Newspaper (1024 X 768)
Lovebird1 (1024 X 768)
Kendo (1024 X 768)
Balloons (1024 X 768)
Poznan street (1920 X 1088)
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 39
Left
Right
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 17
Depth image based rendering
MPEG View Synthesis Reference Software (VSRS) 3.5
Enhanced depth map
Enhanced depth map
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 40
Reference view
Reference view
Left
Right
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 17
Depth image based rendering
3D warping
3D warping
MPEG View Synthesis Reference Software (VSRS) 3.5
Enhanced depth map
Enhanced depth map
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 41
Reference view
Reference view
Left
Right
Warped view
Warped view
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 17
Depth image based rendering
3D warping
3D warping
MPEG View Synthesis Reference Software (VSRS) 3.5
Enhanced depth map
Enhanced depth map
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 42
Reference view
Reference view
Left
Right
Warped view
Warped view
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 17
Depth image based rendering
Hole filling &
inpainting
3D warping
3D warping
MPEG View Synthesis Reference Software (VSRS) 3.5
Enhanced depth map
Enhanced depth map
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 43
Reference view
Reference view
Left
Right
Warped view
Warped view
Virtual intermediate view
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 17
Depth image based rendering
Hole filling &
inpainting
3D warping
3D warping
MPEG View Synthesis Reference Software (VSRS) 3.5
Enhanced depth map
Enhanced depth map
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 44
Reference view
Reference view
Left
Right
Warped view
Warped view
Virtual intermediate view
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 17
Depth image based rendering
Hole filling &
inpainting
3D warping
3D warping
MPEG View Synthesis Reference Software (VSRS) 3.5
Enhanced depth map
Enhanced depth map
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Original camera view
Page 45
Reference view
Reference view
Left
Right
Warped view
Warped view
Virtual intermediate view
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 17
Depth image based rendering
Hole filling &
inpainting
3D warping
3D warping
MPEG View Synthesis Reference Software (VSRS) 3.5
Enhanced depth map
Enhanced depth map
Y-PSNR (dB)
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Original camera view
Page 46
Test sequence
Sequence resolution
Input views
Virtual view
MPEG VSRS views Y-PSNR 3.5 [dB]
MPEG depth maps Enhanced depth maps
Newspaper 1024 X 768 4,6 5 31.98 32.10
Kendo 1024 X 768 3,5 5 36.54 36.72
Poznan Street 1920 X 1088 3,5 4 35.56 35.58
Lovebird1 1024 X 768 6,8 7 28.50 28.68
Balloons 1024 X 768 3,5 4 35.68 35.93
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 18
Objective results
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Color classification
– Initial number of color clusters: 50
• K-means sub-clustering
– Number of cluster : 12
Page 47
Test sequence
Sequence resolution
Input views
Virtual view
MPEG VSRS views Y-PSNR 3.5 [dB]
MPEG depth maps Enhanced depth maps
Newspaper 1024 X 768 4,6 5 31.98 32.10
Kendo 1024 X 768 3,5 5 36.54 36.72
Poznan Street 1920 X 1088 3,5 4 35.56 35.58
Lovebird1 1024 X 768 6,8 7 28.50 28.68
Balloons 1024 X 768 3,5 4 35.68 35.93
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 18
Objective results
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Color classification
– Initial number of color clusters: 50
• K-means sub-clustering
– Number of cluster : 12
Page 48
Test sequence
Sequence resolution
Input views
Virtual view
MPEG VSRS views Y-PSNR 3.5 [dB]
MPEG depth maps Enhanced depth maps
Newspaper 1024 X 768 4,6 5 31.98 32.10
Kendo 1024 X 768 3,5 5 36.54 36.72
Poznan Street 1920 X 1088 3,5 4 35.56 35.58
Lovebird1 1024 X 768 6,8 7 28.50 28.68
Balloons 1024 X 768 3,5 4 35.68 35.93
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 18
Objective results
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Color classification
– Initial number of color clusters: 50
• K-means sub-clustering
– Number of cluster : 12
Page 49
Test sequence
Sequence resolution
Input views
Virtual view
MPEG VSRS 3.5 [dB]
MPEG depth maps Enhanced depth maps
Newspaper 1024 X 768 4,6 5 31.98 32.10
Kendo 1024 X 768 3,5 5 36.54 36.72
Poznan Street 1920 X 1088 3,5 4 35.56 35.58
Lovebird1 1024 X 768 6,8 7 28.50 28.68
Balloons 1024 X 768 3,5 4 35.68 35.93
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 18
Objective results
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Color classification
– Initial number of color clusters: 50
• K-means sub-clustering
– Number of cluster : 12
Page 50
Sequence: Newspaper
With MPEG depth With enhanced depth
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 19
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 51
Sequence: Newspaper
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 19
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 52
Sequence: Newspaper
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 19
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth Original
Page 53
Sequence: Kendo
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 20
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 54
Sequence: Kendo
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 20
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 55
Sequence: Kendo
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 20
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth Original
Page 56
Sequence: Lovebird 1
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 21
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 57
Sequence: Lovebird 1
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 21
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 58
Sequence: Lovebird 1
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 21
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth Original
Page 59
Sequence: Lovebird 1
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 21
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 60
Sequence: Lovebird 1
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 21
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth Original
Page 61
Sequence: Balloons
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 22
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 62
Sequence: Balloons
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 22
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 63
Sequence: Balloons
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 22
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth Original
Page 64
Sequence: Poznan Street
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 23
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 65
Sequence: Poznan Street
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 23
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth
Page 66
Sequence: Poznan Street
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 23
Subjective comparison
Background Motivation Enhancement framework Experimental results Conclusions Future directions
With MPEG depth With enhanced depth Original
Page 67
Conclusions
• We improved the inter-view depth consistency and hence, enhanced the visual experience of free-viewpoint television
• For that, we exploited the per-pixel association between depth and color by classification
• Color classification is accomplished by variational Bayesian inference
• Then, color classes are used for depth classification
• Effectiveness of our approach is demonstrated by both objective and subjective results
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 24
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 68
Future directions
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement 25
• Improve temporal depth consistency
• Improve color classification by using other mixture models
• Improve computational efficiency of color classification
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Page 70
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement X
Gaussian mixture model with variational Bayes inference
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Initialization
Optimization of the
variational posterior
distribution
Variational
lower bound
evaluation
Repeat Until convergence
Color
clusters
RGB
data
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.
Page 71
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement X
Gaussian mixture model with variational Bayes inference
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Initialize number of clusters
• Initialize hyper-parameters
• Initialize responsibilities
Initialization
Optimization of the
variational posterior
distribution
Variational
lower bound
evaluation
Repeat Until convergence
Color
clusters
RGB
data
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.
Page 72
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement X
Gaussian mixture model with variational Bayes inference
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Maximize the variational
lower bound on free energy
• Evaluate the responsibilities
• Update the hyper-parameters
Initialization
Optimization of the
variational posterior
distribution
Variational
lower bound
evaluation
Repeat Until convergence
Color
clusters
RGB
data
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.
Page 73
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement X
Gaussian mixture model with variational Bayes inference
Background Motivation Enhancement framework Experimental results Conclusions Future directions
• Check for convergence
Initialization
Optimization of the
variational posterior
distribution
Variational
lower bound
evaluation
Repeat Until convergence
Color
clusters
RGB
data
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.
Page 74
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement X
Gaussian mixture model with variational Bayes inference
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Initialization
Optimization of the
variational posterior
distribution
Variational
lower bound
evaluation
Repeat Until convergence
Color
clusters
RGB
data
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.
Page 75
ISM 2012 - Pravin Kumar Rana, A Variational Bayesian Inference Framework for Multiview Depth Image Enhancement X
Gaussian mixture model with variational Bayes inference
Background Motivation Enhancement framework Experimental results Conclusions Future directions
Initialization
Optimization of the
variational posterior
distribution
Variational
lower bound
evaluation
Repeat Until convergence
Color
clusters
RGB
data
[4] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed. New York: Springer, 2006.