1 Non-Linear Least Squares and Sparse Matrix Techniques: Applications Richard Szeliski Microsoft Research UW-MSR Course on Vision Algorithms CSE/EE 577, 590CV, Spring 2004 5/4/2004 NLS and Sparse Matrix Techniques 2 Readings • R.Szeliski. Fast surface interpolation using hierarchical basis functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):513-528, June 1990. • R. Szeliski and S. B. Kang. Recovering 3D shape and motion from image streams using nonlinear least squares. J. Vis. Comm. Image Representation, 5(1):10-28, March 1994. (Also available as CRL-93-3.) • A. Levin and R. Szeliski, Visual Odometry and Map Correlation, CVPR’2004. • P. Pérez, M. Gangnet, and A. Blake. Poisson Image Editing. SIGGRAPH’2003. • L. Zhang, G. Dugas-Phocion, J.-S. Samson, and S. M. Seitz, Single view modeling of free-form scenes, J. Vis. Comp. Anim., 2002, vol. 13, no. 4, pp. 225-235. • A. Agarwala, A. Hertzmann, D. H. Salesin, S. M. Seitz, Keyframe- Based Tracking for Rotoscoping and Animation, SIGGRAPH’2004. 5/4/2004 NLS and Sparse Matrix Techniques 3 Outline Preconditioning • diagonal scaling, partial Cholesky factorization • hierarchical basis functions (wavelets) • quadtree splines • 2D application: height and normal interpolation (Single View Modeling) 5/4/2004 NLS and Sparse Matrix Techniques 4 Outline Structure from motion • problems with size, linearity, conditioning • alternative parameterizations • uncertainty modeling 5/4/2004 NLS and Sparse Matrix Techniques 5 Outline Visual Odometry and Map Correlation • fast visual localization • visual odometry↔map correlation • open problems Preconditioned Conjugate Gradient (See Shewchuk’s TR)
16
Embed
Readings Non-Linear Least Squares and Sparse Matrix ...Quadtree splines Only populate (estimate) a subset of mesh [Szeliski & Shum, PAMI’96] 5/4/2004 NLS and Sparse Matrix Techniques
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Non-Linear Least Squares and Sparse Matrix Techniques:
Applications
Richard SzeliskiMicrosoft Research
UW-MSR Course onVision Algorithms
CSE/EE 577, 590CV, Spring 2004
5/4/2004 NLS and Sparse Matrix Techniques 2
Readings• R.Szeliski. Fast surface interpolation using hierarchical basis functions.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):513-528, June 1990.
• R. Szeliski and S. B. Kang. Recovering 3D shape and motion from image streams using nonlinear least squares. J. Vis. Comm. Image Representation, 5(1):10-28, March 1994. (Also available as CRL-93-3.)
• A. Levin and R. Szeliski, Visual Odometry and Map Correlation,CVPR’2004.
• P. Pérez, M. Gangnet, and A. Blake. Poisson Image Editing. SIGGRAPH’2003.
• L. Zhang, G. Dugas-Phocion, J.-S. Samson, and S. M. Seitz, Single view modeling of free-form scenes, J. Vis. Comp. Anim., 2002, vol. 13, no. 4, pp. 225-235.
• A. Agarwala, A. Hertzmann, D. H. Salesin, S. M. Seitz, Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH’2004.
• Use a multi-level(pyramid-like) basis for system
x = Sy, S = S1… SL-1
5/4/2004 NLS and Sparse Matrix Techniques 12
2-D Hierarchical basis
Like a pyramid, but complete (no extra variables)
3
5/4/2004 NLS and Sparse Matrix Techniques 13
Surface interpolation example
5/4/2004 NLS and Sparse Matrix Techniques 14
Surface interpolation - performance
5/4/2004 NLS and Sparse Matrix Techniques 15
Surface interpolation - performance
5/4/2004 NLS and Sparse Matrix Techniques 16
Surface interpolation - performance
5/4/2004 NLS and Sparse Matrix Techniques 17
Eigenvalue analysis
5/4/2004 NLS and Sparse Matrix Techniques 18
Comparison with multigrid
Solve a series of nested problems at different resolutions, using injection and prolongationto move between levels [Briggs, 1987]
(Personal experience): does not seem to work well on inhomogeneous problems (as opposed to very fine grid refinement)
4
5/4/2004 NLS and Sparse Matrix Techniques 19
Open issues
How many levels to use?Which order interpolant?Adaptation to data density and continuity• intuition: when data terms dominate, want
nodal basis; when smoothness, want hierarchical
5/4/2004 NLS and Sparse Matrix Techniques 20
Quadtree splines
Only populate (estimate) a subset of mesh
[Szeliski & Shum, PAMI’96]
5/4/2004 NLS and Sparse Matrix Techniques 21
Quadtree spline
Equivalent to sparse hierarchical basis
5/4/2004 NLS and Sparse Matrix Techniques 22
Quadtree splines - application
Discontinuous optic flow
Discontinuous single view modeling (next)
Single view modelingof free-form scenes
L. Zhang, G. Dugas-Phocion, J.-S. Samson, and S. M. Seitz
5/4/2004 NLS and Sparse Matrix Techniques 24
Problem
Extract a 3D model from a single image
5
5/4/2004 NLS and Sparse Matrix Techniques 25
Normal constraints
Fix the local orientation of the surface
• Dy f(P) =
• Dx f(P) =
P : point where we set the
normal constraint
α
β
γ εaf(ε) – f(γ)
a
f(β) – f(α)a
5/4/2004 NLS and Sparse Matrix Techniques 26
Sample deformations
5/4/2004 NLS and Sparse Matrix Techniques 27
Speed up the solver
Wavelet transformation (from coefficient to height)
1D Convolution 2D Convolution
hI = (hA + hB) / 2 + cI
5/4/2004 NLS and Sparse Matrix Techniques 28
Speed up the solver
What happens with discontinuities ?
Instead of : hI = (hA + hB) / 2 + cI
We get :hI = hB + cI
1D Convolution
5/4/2004 NLS and Sparse Matrix Techniques 29
Advantage of discontinuities
5/4/2004 NLS and Sparse Matrix Techniques 30
User specifiedUser specified High curvatureHigh curvatureAlong a curveAlong a curve
Hierarchical structure
Ways of changing the resolution.
Along a curve High curvature
6
5/4/2004 NLS and Sparse Matrix Techniques 31
Final results
Structure from motion
5/4/2004 NLS and Sparse Matrix Techniques 33
Problem areas
Problems with size, linearity, conditioning• reduce the number of variables
[Shum, Ke, Zhang]• partition the cameras / data [Steedly et al.]Uncertainty modelingAlternative parameterizations
5/4/2004 NLS and Sparse Matrix Techniques 34
Uncertainty modeling
Bas-relief ambiguity:• correlation between depth and
motion/rotation
5/4/2004 NLS and Sparse Matrix Techniques 35
Uncertainty modeling
5/4/2004 NLS and Sparse Matrix Techniques 36
Alternative projection
Use object-centered representation
7
Video-Based Rendering
5/4/2004 NLS and Sparse Matrix Techniques 38
Video-Based Virtual Tours
Move camera along a rail (“dolly track”) and play back a 360° video
Applications:• Homes and architecture• Outdoor locations
(tourist destinations)
5/4/2004 NLS and Sparse Matrix Techniques 39
360° video camera
OmniCam (six-camera head)5/4/2004 NLS and Sparse Matrix Techniques 40
OmniCam
Built by Point Grey Research (Ladybug)
Six camera head
Portable hard drives, fiber-optic link
Resolution per image: 1024 x 768
FOV: ~100o x ~80o
Acquisition speed: 15 fps uncompressed
5/4/2004 NLS and Sparse Matrix Techniques 41
Acquisition platforms
Robotic cart
Wearable
5/4/2004 NLS and Sparse Matrix Techniques 42
Stitching
(Only 4 of 6 images shown here)
8
5/4/2004 NLS and Sparse Matrix Techniques 43
Feature tracking for stabilization
Points tracked in all frames in all 6 cameras
Edges tracked in all frames in all 6 cameras5/4/2004 NLS and Sparse Matrix Techniques 44
Stabilization
Before motion stabilization After motion stabilization
Align frame-
to-frameand
distribute∆heading
5/4/2004 NLS and Sparse Matrix Techniques 45
Head-mounted outdoor scene acquisitionMap controlLocalized audio for a richer experienceComplex path navigation
(greater freedom ofmotion)
Outdoor Garden Tour
5/4/2004 NLS and Sparse Matrix Techniques 46
Map control• The garden map
• Drawing the acquisition path
• Mapping video framespositions on map
• Placing sound sources(background and dynamic)
• Output is a descriptiveXML file
5/4/2004 NLS and Sparse Matrix Techniques 47
Bifurcation handling
Hypothesize, align, choose best pair as bifurcation
Choice of path depends on current heading and bifurcation point orientation
Current heading
5/4/2004 NLS and Sparse Matrix Techniques 48
Bifurcation handling
Hypothesize, align, choose best pair as bifurcation
Choice of path depends on current heading and bifurcation point orientation
Current heading
9
Demo 1A walk in Bellevue Botanical Garden
Demo 2A High-End Home
5/4/2004 NLS and Sparse Matrix Techniques 51
Future directions
Towards true 3D:• better compression• object interaction (e.g., vase on table)• object compositing (video game)• more general motion (“true” IBR)• automate generation of bifurcation points and
map control (UI)
Visual Odometry andMap Correlation
Anat Levin and Richard SzeliskiCVPR’2004
5/4/2004 NLS and Sparse Matrix Techniques 53
Map Correlation
Four stages:1. Overlap detection (path bifurcations, better