3D Registration 3D Fusion
3D Registration 3D Fusion
3D photography course schedule
Topic
Feb 21 Introduction
Feb 28 Lecture: Geometry, Camera Model, Calibration
Mar 7 Lecture: Features & Correspondences
Mar 14 Project Proposals
Mar 21 Lecture: Epipolar Geometry
Mar 28 Depth Estimation + 2 papers
Apr 4 Single View Geometry + 2 papers
Apr 11 Active Ranging and Structured Light + 2 papers
Apr 18 Project Updates
Apr. 25 --- Easter ---
May 2 SLAM + 2 papers
May 9 3D & Registration + 2 papers
May 16 SfM/Self Calibration + 2 papers
May 23 Shape from Silhouettes + 2 papers
May 30 Final Projects (if not demo day)
3D modeling
• Aligning range images
• Pairwise
• Globally
• Surface reconstruction
• Single range image
• Merged
(some slides from S. Rusinkiewicz, J. Ponce,…)
Aligning 3D Data
• If correct correspondences are known, it is possible to find correct relative rotation/translation
q = a +
q is a quaternion,
a R is its real part, and
R3 is its imaginary part.
• Sum of quaternions:
• Multiplication by a scalar:
• Quaternion product:
• Conjugate: )
Operations on quaternions:
• Norm:
Note:
Intermezzo: quaternions
Let R denote the rotation of angle about the unit vector u.
Define
Then for any vector ,
Reciprocally, if q = a + ( b, c, d )T is a unit quaternion,
the corresponding rotation matrix is:
Intermezzo: quaternions and rotations
Problem: Find the rotation matrix R and the vector t that
minimize
i=1
E = | xi’ – R xi – t |2 . n
At a minimum: 0 = E/ t = –2 ( xi’ – R xi – t ) . n
i=1
Or.. t = x’ – R x.
If yi = xi –x and yi’ = xi’ –x’, the error is (at a minimum):
i=1
E = | yi’ – R yi |2
n
i=1
= | yi’ q – q yi |2
n
i=1 = | yi’ – q yi q|2|q|2
n
Or E =
Linear least squares !!
Estimate rigid transformation
Aligning 3D Data
• How to find corresponding points?
• Previous systems based on user input, feature matching, surface signatures, etc.
Spin Images
• Johnson and Hebert
• “Signature” that captures local shape
• Similar shapes similar spin images
Computing Spin Images
• Start with a point on a 3D model
• Find (averaged) surface normal at that point
• Define coordinate system centered at this point, oriented according to surface normal and two (arbitrary) tangents
• Express other points (within some distance) in terms of the new coordinates
Computing Spin Images
• Compute histogram of locations of other points, in new coordinate system, ignoring rotation around normal:
n̂ p n̂ p
n̂ p n̂ p
“radial dist.”
“elevation”
Computing Spin Images
“radial dist.”
“elevation”
Spin Image Parameters
• Size of neighborhood
• Determines whether local or global shape is captured
• Big neighborhood: more discriminatory power
• Small neighborhood: resistance to clutter
• Size of bins in histogram:
• Big bins: less sensitive to noise
• Small bins: captures more detail
Alignment with Spin Image
• Compute Spin Image for each point / subset of points in both sets
• Find similar spin images => potential correspondences
• Compute alignment from correspondences
Same problems as with image matching:
- Robustness of descriptor vs. discriminative power
- Mismatches => robust estimation required
Solving 3D puzzles with VIPs
SIFT features
• Extracted from 2D images
• Variation due to viewpoint
VIP features
• Extracted from 3D model
• Viewpoint invariant
15
(Wu et al., CVPR08)
Presented afterwards !
Aligning 3D Data
Alternative: assume closest points correspond
to each other, compute the best transform…
Aligning 3D Data
… and iterate to find alignment
Iterated Closest Points (ICP) [Besl & McKay 92]
Converges if starting position “close enough“
ICP Variant – Point-to-Plane Error Metric
• Using point-to-plane distance instead of point-to-point lets flat regions slide along each other more easily [Chen & Medioni 91]
Finding Corresponding Points
• Finding closest point is most expensive stage of ICP
• Brute force search – O(n)
• Spatial data structure (e.g., k-d tree) – O(log n)
• Voxel grid – O(1), but large constant, slow preprocessing
Finding Corresponding Points
• For range images, simply project point [Blais 95]
• Constant-time, fast
• Does not require precomputing a spatial data structure
Efficient ICP
• “Efficient Variants of the ICP algorithm” [Rusinkiewicz & Levoy, 3DIM 2001]
=> Presented afterwards
3D Global Registration
3D Global registration The problem:
• Given: n scans around an object
• Goal: align them all
• First attempt: ICP each scan to one other
• Want method for distributing accumulated error among all scans
3D Global registration
Approach #1: Avoid the Problem
• In some cases have 1 scan that covers large part of surface (e.g., cylindrical scan)
• Align all other scans to this “anchor”
• Disadvantage: not always practical to obtain anchor scan
Approach #2: The Greedy Solution
• Align each new scan to all previous scans
• Disadvantages:
• Order dependent
• Doesn’t spread out error
Approach #3: Brute-Force Solution
• While not converged:
• For each scan:
• For each point:
– For every other scan
» Find closest point
• Minimize error w.r.t. transforms of all scans
• Disadvantage:
• Solve (np)(np) matrix equation, where n is number of scans and p is number of points per scan
Approach #3a: Slightly Less Brute-Force
• While not converged:
• For each scan:
• For each point:
– For every other scan
» Find closest point
• Minimize error w.r.t. transform of this scan
• Faster than previous method (matrices are pp)
Graph Methods
• Many globalreg algorithms create a graph of pairwise alignments between scans
Scan 1 Scan 1 Scan 5 Scan 5
Scan 4 Scan 4
Scan 3 Scan 3
Scan 2 Scan 2 Scan 6 Scan 6
Pulli’s Algorithm
• Perform pairwise ICPs, record sample (e.g. 200) of corresponding points
• For each scan, starting w. most connected
• Align scan to existing set
• While (change in error) > threshold
• Align each scan to others
Sharp et al. Algorithm
• Perform pairwise ICPs, record only optimal rotation/translation for each
• Decompose alignment graph into cycles
• While (change in error) > tolerance
• For each cycle:
• Spread out error equally among all scans in the cycle
• For each scan belonging to more than 1 cycle:
• Assign average transform to scan
Lu and Milios Algorithm
• Perform pairwise ICPs, record optimal rotation/translation and covariance for each
• Least squares simultaneous minimization of all errors (covariance-weighted)
• Requires linearization of rotations
Open Questions in Global Registration
• Best way to do correctly-weighted globalreg without linearizing rotations?
• How to prevent bias (if many scans in one area, few scans in another)?
• Robust outlier detection
• One bad ICP can throw off the entire model
Surface Reconstruction
Problems With Reconstruction from General Point Clouds
Surface Reconstruction from Range Images
• Often an easier problem than reconstruction from arbitrary point clouds
• Implicit information about adjacency, connectivity
• Roughly uniform spacing
Surface Reconstruction From Range Images
• First, construct surface from each range image
• Then, merge resulting surfaces
• Obtain average surface in overlapping regions
• Control point density
Range Image Tesselation
• Given a range image, connect up the neighbors
Range Image Tesselation
• Caveat #1: can’t be too aggressive
• Introduce distance threshold for tesselation
• Caveat #2: Which way to triangulate?
• Possible heuristics:
• Shorter diagonal
• Dihedral angle closer to 180
• Maximize smallest angle in both triangles
• Always the same way (best triangle strips)
Range Image Tesselation
Scan Merging Using Zippering
• Turk & Levoy, 1994
• Erode geometry in overlapping areas
• Stitch scans together along seam
• Re-introduce all data
• Weighted average
Zippering
Point Weighting
• Higher weights to points facing the camera
• Favor higher sampling rates
Point Weighting
• Lower weights (tapering to 0) near boundaries
• Smooth blends between views
Volumetric Reconstruction
• Implicit function defined volumetrically
• Usually stored sampled on a 3D grid
• Can be compressed (e.g., using RLE)
• Another possibility: hierarchical data structures
• Can extract isosurface (i.e., subset of space where implicit function = some constant)
Volumetric Reconstruction Overview
• Generate signed distance function (or something close to it) for each scan
• Compute average (possibly weighted)
• Extract isosurface
Volumetric integration (Curless and Levoy, Siggraph´96)
sensor
range surfaces
volume
distance
depth
weight (~accuracy)
signed distance
to surface
surface1
surface2
combined estimate
• use voxel space
• new surface as zero-crossing
(find using marching cubes)
• least-squares estimate
(zero derivative=minimum)
Volumetric Reconstruction Benefits
• Always generates a manifold surface
• Can control sampling density
• Averaging of signed distance functions corresponds to averaging the surfaces
Volumetric Reconstruction Drawbacks
• Represent a 3D entity rather than 2D
• Running time
• Storage
• Resampling step – bandlimits the function
• Generates consistent topology, but not always the topology you wanted
• Problems with very thin surfaces
From volume to mesh: Marching Cubes
“Marching Squares” in 2D
“Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
From volume to mesh: Marching Cubes
“Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
Multiple depth images Volumetric integration
Volumetric 3D integration
Presentations
• Efficient Variants of the ICP algorithm
• Model Matching with Viewpoint Invariant Patches