Image Stitching Shangliang Jiang Kate Harrison. What is image stitching?

Image Stitching

Shangliang JiangKate Harrison

What is image stitching?

What is image stitching?

Introduction

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°

Introduction

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°

Introduction

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°

Recognizing Panoramas

• 1D Rotations ()– Ordering matching images

Recognizing Panoramas

• 1D Rotations ()– Ordering matching images

Recognizing Panoramas

• 1D Rotations ()– Ordering matching images

Recognizing Panoramas

• 2D Rotations (, )– Ordering matching images

• 1D Rotations ()– Ordering matching images

Recognizing Panoramas

• 1D Rotations ()– Ordering matching images

• 2D Rotations (, )– Ordering matching images

Recognizing Panoramas

• 1D Rotations ()– Ordering matching images

• 2D Rotations (, )– Ordering matching images

Recognizing Panoramas

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching– SIFT Features– Nearest Neighbor Matching

• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

SIFT Features

• SIFT features are…– Geometrically invariant to similarity transforms,

• some robustness to affine change

– Photometrically invariant to affine changes in intensity

Overview

• Feature Matching– SIFT Features– Nearest Neighbor Matching

• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Nearest Neighbor Matching

• Find k nearest neighbors for each feature– k number of overlapping images (we use k = 4)

• Use k-d tree– k-d tree recursively bi-partitions data at mean in the

dimension of maximum variance– Approximate nearest neighbors found in O(nlogn)

Overview

• Feature Matching– SIFT Features– Nearest Neighbor Matching

• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching

– Random Sample Consensus (RANSAC) for Homography

– Probabilistic model for verification

• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching

– Random Sample Consensus (RANSAC) for Homography

– Probabilistic model for verification

• Bundle Adjustment• Image Compositing• Conclusions

RANSAC for Homography

RANSAC for Homography

RANSAC for Homography

RANSAC for Homography

RANSAC for Homography

Overview

• Feature Matching• Image Matching

– Random Sample Consensus (RANSAC) for Homography

– Probabilistic model for verification

• Bundle Adjustment• Image Compositing• Conclusions

Probabilistic model for verification

Finding the panoramas

Finding the panoramas

Finding the panoramas

Overview

• Feature Matching• Image Matching

– RANSAC for Homography– Probabilistic model for verification

• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment

– Error function

• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment

– Error function

• Image Compositing• Conclusions

Bundle Adjustment

• New images initialised with rotation, focal length of best matching image

Bundle Adjustment

• New images initialised with rotation, focal length of best matching image

Error function

• Sum of squared projection errors

– n = #images– I(i) = set of image matches to image i– F(i, j) = set of feature matches between images i,j

– rijk = residual of kth feature match between images i,j

• Robust error function

Overview

• Feature Matching• Image Matching• Bundle Adjustment

– Error function

• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Blending

Gain compensation

How do we blend?

Linear blending Multi-band blending

Multi-band Blending

• Burt & Adelson 1983– Blend frequency bands over range

Low frequency ( > 2 pixels)

High frequency ( < 2 pixels)

2-band Blending

3-band blendingBand 1: high frequencies

3-band blendingBand 2: mid-range frequencies

3-band blendingBand 3: low frequencies

Panorama straightening

Heuristic: people tend to shoot pictures in a certain way

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions

Conclusion

Algorithm

AutoStitch program

AutoStitch.net

Open questions

• Advanced camera modeling– radial distortion, camera motion, scene motion,

vignetting, exposure, high dynamic range, flash …

• Full 3D case – recognizing 3D objects/scenes in unordered datasets

Credits

• Automatic Panoramic Image Stitching Using Invariant Features, 2007– Matthew Brown and David G. Lowe (Uni. of British

Columbia)

• Recognising Panoramas, 2003– Matthew Brown and David G. Lowe (Uni. of British

Columbia)– 2003– Thanks for the slides!

• Image Alignment and Stitching: A Tutorial, 2006– Richard Szeliski (Microsoft)

Questions?