PowerPoint
Stereo VideoTemporally Consistent Disparity Maps from
Uncalibrated Stereo VideosReal-time Spatiotemporal Stereo Matching
Using the Dual-Cross-Bilateral GridTemporally Consistent Disparity
and Optical Flow via Efficient Spatio-temporal FilteringEfficient
Spatio-temporal Local Stereo Matching Using Information
Permeability Filtering1A. Temporally Consistent Disparity Maps from
Uncalibrated Stereo VideosMichael Bleyer and Margrit Gelautz
International Symposium on Image and Signal Processing and
Analysis (ISPA) 200922B. Real-time Spatiotemporal Stereo Matching
Using The Dual-cross-bilateral GridChristian Richardt, Douglas Orr,
Ian Davies, Antonio Criminisi, and Neil A. Dodgson1
The European Conference on Computer Vision (ECCV) 201033C.
Temporally Consistent Disparity And Optical Flow Via Efficient
Spatio-temporal FilteringAsmaa Hosni, Christoph Rhemann, Michael
Bleyer, and Margrit Gelautz
The Pacific-Rim Symposium on Image and Video Technology (PSIVT)
20114D. Efficient Spatio-temporal Local Stereo Matching Using
Information Permeability FilteringCuong Cao Pham, Vinh Dinh Nguyen,
and Jae Wook Jeon
International Conference on Image
Processing(ICIP)20125OutlineIntroductionRelated WorksMethods and
ResultsA. Median FilterB. Temporal DCB GridC. Spatial-temporal
Weighted Smoothing D. Three-pass
AggregationComparisonConclusion6Introduction7IntroductionStereo
matching issues only focus on static image pairs.The conventional
methods estimate the disparities by using spatial and color
information.
The important problem of extending to video is
flickering.Solution :Base on local methods (for real-time)Enforce
temporally consistent (for flickering)
8
Related Works 9Related Works About Local MethodsThe key of local
method lies in the cost aggregation step.Aggregate the cost data
from the neighboring pixels within a finite size window.The most
well-known method is edge-preserving algorithm.Adaptive support
wight Geodesic DiffusionBilateral filterGuided filter10Related
Works Single-frame stereo matching11
Related Works Spatio-temporal stereo matchingThe inter disparity
difference between two successive frames is minimized to enforce
the temporal consistency.12
Methods and Results13A. Median filter14
A. Median filter15
A. Median filterComputing 1 disparity map takes 1 second.But a
video content about 30~60 frames per second.=> Can NOT achieve
real-time. No data and comparison.
16
B. Temporal DCB GridBilateral GridIt runs faster and uses less
memory as increases.
Dual-Cross-Bilateral Grid
17
B. Temporal DCB GridDichromatic DCB Grid
Comparison (fps)18
200xB. Temporal DCB GridTemporal DCB Grid
Last n = 5 frames, each weighted by wii=0 : current framei=1 :
previous frame19
Weighted Sum
B. Temporal DCB Grid20
16 fps14 fps21
B. Temporal DCB GridSource dataB. Temporal DCB GridOnly use
intensity informationJust near-real-time22
C. Spatial-temporal Weighted Smoothing Cost
initializationConstruct a spatio-temporal cost volume for each
disparity d.Cost aggregationSmooth cost volume with a
spatio-temporal filter.(Guided filter [1])Disparity
computationSelect the lowest costs as
disparity(WTA)RefinementWighted median filter
23[1]Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz,
M.Fast Cost-Volume Filtering for Visual Correspondence and Beyond.
CVPR(2011) and PAMI (2013)C. Spatial-temporal Weighted Smoothing
24
C. Spatial-temporal Weighted Smoothing Cost initialization
Cost aggregation
25
wk: wx * wy* wt : smoothness parameter
C. Spatial-temporal Weighted Smoothing The guided filter weights
can be implemented by a sequence of linear operations.
All summations are 3D box filters and can be computed in O(N)
time.26
C. Spatial-temporal Weighted Smoothing Disparity computation :
Winner take all
Refinement : Wighted Meadian filter
=> Just adjust to reduce single frame error.
27
C. Spatial-temporal Weighted Smoothing Temporal vs.
frame-by-frame processing. 2nd row: Disparity maps computed by a
frame-by-frame implementation show flickering artifacts. 3rd row:
Our proposed method exploits temporal information, thus can remove
most artifacts28
C. Spatial-temporal Weighted Smoothing 29
C. Spatial-temporal Weighted Smoothing 30
C. Spatial-temporal Weighted Smoothing 31
D. Three-pass cost aggregationThree-pass cost aggregation
technique based on information permeability(Adaptive
Support-Weight).[2]
32[2] Yoon, K.J., Kweon, I.S.: Locally Adaptive Support-Weight
Approach for VisualCorrespondence Search. In: CVPR (2005)
D. Three-pass cost aggregation33
Frame i+1Frame iFrame i-1D. Three-pass cost aggregationMatching
cost initialization
v = (x, y, t) represents the spatial and temporal positions of a
voxel.
Similarity(weighted) function
34
Show the effectiveness of using temporal information in addition
to spatial information .D. Three-pass cost aggregationSpatial
Aggregation : Horizontal and then Vertical
35
D. Three-pass cost aggregationTemporal Aggregation : Forward and
backward
Disparity computation : WTARefinementconsistency check 3 3
median filter.
36
D. Three-pass cost aggregationComputational ComplexityOnly six
multiplications and nine additions per voxelIt is still more
efficient than the adaptive support-weight approach.
Without motion estimation
37D. Three-pass cost aggregation38
D. Three-pass cost aggregation39
Comparison40ComparisonA.B.C.D.MethodOptical flow +Median
filterWeighted last 5 framesGuided filter temporallyThree pass
DrawbackToo slowOver smoothnessReference frame number3 frames-1~15
frames-4~05 frames-2~23frames-1~141Comparison42
No post-processingInclude post-processing : consistency check
and 3 3 median filter Conclusion43ConclusionBased on
edge-preserving methods.Extend these concepts to time
dimension.
These methods only solved slow motion scenes.They do not perform
well with dynamic scenes that contain large object motions.
44