Image Stitching Shangliang Jiang Kate Harrison
Mar 21, 2016
Image Stitching
Shangliang JiangKate Harrison
What is image stitching?
What is image stitching?
Introduction• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
Introduction• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°
Introduction• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
Recognizing Panoramas
• 2D Rotations (, )– Ordering matching images
• 1D Rotations ()– Ordering matching images
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
• 2D Rotations (, )– Ordering matching images
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
• 2D Rotations (, )– Ordering matching images
Recognizing Panoramas
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching
– SIFT Features– Nearest Neighbor Matching
• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
SIFT Features• SIFT features are…
– Geometrically invariant to similarity transforms,• some robustness to affine change
– Photometrically invariant to affine changes in intensity
Overview• Feature Matching
– SIFT Features– Nearest Neighbor Matching
• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Nearest Neighbor Matching• Find k nearest neighbors for each feature
– k number of overlapping images (we use k = 4)• Use k-d tree
– k-d tree recursively bi-partitions data at mean in the dimension of maximum variance
– Approximate nearest neighbors found in O(nlogn)
Overview• Feature Matching
– SIFT Features– Nearest Neighbor Matching
• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching
– Random Sample Consensus (RANSAC) for Homography
– Probabilistic model for verification• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching
– Random Sample Consensus (RANSAC) for Homography
– Probabilistic model for verification• Bundle Adjustment• Image Compositing• Conclusions
RANSAC for Homography
RANSAC for Homography
RANSAC for Homography
RANSAC for Homography
RANSAC for Homography
Overview• Feature Matching• Image Matching
– Random Sample Consensus (RANSAC) for Homography
– Probabilistic model for verification• Bundle Adjustment• Image Compositing• Conclusions
Probabilistic model for verification
Finding the panoramas
Finding the panoramas
Finding the panoramas
Overview• Feature Matching• Image Matching
– RANSAC for Homography– Probabilistic model for verification
• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment
– Error function• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment
– Error function• Image Compositing• Conclusions
Bundle Adjustment• New images initialised with rotation, focal
length of best matching image
Bundle Adjustment• New images initialised with rotation, focal
length of best matching image
Error function• Sum of squared projection errors
– n = #images– I(i) = set of image matches to image i– F(i, j) = set of feature matches between images i,j– rij
k = residual of kth feature match between images i,j
• Robust error function
Overview• Feature Matching• Image Matching• Bundle Adjustment
– Error function• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Blending
Gain compensation
How do we blend?
Linear blending Multi-band blending
Multi-band Blending• Burt & Adelson 1983
– Blend frequency bands over range
Low frequency ( > 2 pixels)
High frequency ( < 2 pixels)
2-band Blending
3-band blendingBand 1: high frequencies
3-band blendingBand 2: mid-range frequencies
3-band blendingBand 3: low frequencies
Panorama straightening
Heuristic: people tend to shoot pictures in a certain way
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
Conclusion
Algorithm
AutoStitch program
AutoStitch.net
Open questions• Advanced camera modeling
– radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash …
• Full 3D case – recognizing 3D objects/scenes in unordered datasets
Credits• Automatic Panoramic Image Stitching Using
Invariant Features, 2007– Matthew Brown and David G. Lowe (Uni. of British
Columbia)• Recognising Panoramas, 2003
– Matthew Brown and David G. Lowe (Uni. of British Columbia)
– 2003– Thanks for the slides!
• Image Alignment and Stitching: A Tutorial, 2006– Richard Szeliski (Microsoft)
Questions?