Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de Department of Informatics | Technische Universität München Fachgebiet Augmented Reality Panoramic Image Mosaics Dimitar Dimitrov Heung-Yeung Shum and Richard Szeliski Microsoft Research 15.01.2007
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
Department of Informatics | Technische Universität München
FachgebietAugmented Reality
Panoramic Image Mosaics
Dimitar Dimitrov
Heung-Yeung Shum and Richard SzeliskiMicrosoft Research
15.01.2007
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Motivation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Motivation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Motivation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Motion Parallax
Image blending
Image warping
Image registration
Jacobian, Hessian
Planar Homography
Image intensity gradient
Terms that should be known:
Panoramic Image Mosaics
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Motion parallax: The motion parallax is the effect of
closer objects moving by faster than more distant ones
“apparent motion” of an object against a distant background because of a perspective shift.
change of angular position of two stationary points relative to each other as seen by an observer, due to the motion of an observer
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Jacobian matrix – the derivative of multivariate function. The Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued function. Its importance lies in the
fact that it represents the best linear approximation to a differentiable function near a given point.
Hessian matrix – the matrix of f is the matrix of second partial derivatives evaluated at x.
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Matematically homography is defined as a relation between two figures, such that to any point in one figure corresponds one and only one point in the other, and vise versa.
In the field of Computer Vision, a homography is defined in 2 dimensional space as a mapping between a point on a ground plane from as seen from one camera, to the same point on the ground plane as seen from a second camera.
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Image registration is the process of finding a homography between two images (reference and target) of a scene and spatially align them.
> Terms that should be known > Image registration process
What is image registration:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Building point correspondences through extracting feature points from which a homography is estimated
Iteratively comparing intensity pattern of the images, search of minimum of the intensity error, and so finding the homography.
Once a homography is estimated warping and resampling of the target image is made. Bilinear pixel resampling is used in this paper.
Image with extracted feature points used for correspondence
Edges and lines detected with Moravec corner detector
> Terms that should be known > Image registration process
How is the registration performed:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Measuring similarity between images by means of intensity difference between all overlapping pixels of the two images
Defining a function which gives the amount of error between the compared images under the current transformation
Minimizing this error function – finding at which transformation the images have minimum differences in intensity. Minimizing the error function by taking it’s derivative to null with any gradient descent algorithm.
Problem - Algorithms for searching minimum of a function often find local minimums, not global.
Principle of intensity-based approach to image registration:> Terms that should be known > Image registration process
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
1 1 3
2 2 4
Registering two identical images by iterative intensity error minimization
Searching the best alignment of the patch to the image by intensity error minimization
> Terms that should be known > Image registration process
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Local minima problem by intensity-based image registration
> Terms that should be known > Image registration process
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Constructing Cylindrical or spherical panorama: Images from camera on a fixed tripod with
fixed focal length and field of view.
World transformation to cylindrical or spherical coordinates.
Translation of the images with each’s panning angle, recovered through estimation of intensity difference between the images.
Limitations: Tilt angle problems. Tilting towards north
or south pole, requires knowing the focal length
Estimating the focal length can be quite. Inaccurate.
> Cylindrical and spherical panoramas – example
Cylindrical and spherical panorama
Warping an image into cylindrical
coordinates
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Warping the images into cylindrical (spherical) coordinates: Converting plane coordinates into cylindrical
Converting plane coordinates into spherical
panning angle
scanline
> Cylindrical and spherical panoramas – example
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
The error function estimates the squared error over each pixel i between the images for a given incremental δt
Linearizing the error function about I1
> Cylindrical and spherical panoramas – example
Defining the intensity minimization function:
Coordinate vector
Coordinate vector with shift
The incremental value
Intensity error
Image gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Solving the linear least-squares for the linearized equation about the incremental δt
When solved about δt we can now update our initial estimate t with t = t + δt
Blending images with simple feathering algorithm
> Cylindrical and spherical panoramas – example
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Replace cylindrical and spherical coordinates with rotational matrices (and focal length) for each image. Performing the registration in the input image’s coordinate system.
develop a global optimization technique to find the optimal overall registration. compute local motion estimates (block-based optical flow) between pairs of
overlapping images, and use these estimates to warp each input image so as to reduce the ghosting.
Create a panorama without using of special hardware, and remove the need for pure panning motion with no motion parallax.
Paint a full view panoramic mosaic with a simple hand-held camera or camcorder. accumulated misregistration errors, which are always present in any large image
mosaic. any deviations from the pure parallax-free motion model may result in local
misregistrations, which are visible as a loss of detail or multiple images (ghosting).
Problems to solve:
Idea:
> Schum – Szelski Algorithm
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Creating small mosaic using planar projective motion model and then calculating the rough focal length from it.
Sequentially assemble complete panoramic image using rotational motion model and patch-based alignment
Block adjustment (global alignment) modifies each image’s transformand focal length to minimize global error. Removes large inconsistensies, gaps
Deghosting (local alignment) reduces any local missregistration errors
Environment mapping. The mosaic is stored like a collection of images with their associated parameters or mapped over a polyhedron
Processing flow:
> Schum – Szelski Algorithm
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Representing the mosaic as collection of images with associated geometrical transformations
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Warping image into another using planar perspective motion model
> 8–parameter motion model
Rewriting the matrix equation into function from x and y
Homography equationwith homogenous coordinates
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Iterative equation
> 8–parameter motion model
Rewriting the coordinates function according the iteration equasion
Where D is the deformation matrix
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Intensity error function with the iterative equation integrated.
> 8–parameter motion model
First order taylor expansion over bivariate function
Intensity error between the images under current transformation
The Jacobian matrix
Image gradient at xi
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Basic least-squares problem through normal equations
Basic linear least-squares equation
The hessian from image gradients
Gradient-weightend intensity error
Solving the least-squares problem for the intensity error function with normal equations
> 8–parameter motion model
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Limitations: Has more free parameters than
necessary thus slow convergence
Gets easily stuck in local minima when the initial misregistration is big
> 8–parameter motion model
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> 3-d rotations and zooms
From the fact the camera and the scene are in common coordinate system follows that the movement of the camera between consequtive images could be described by translation scaling and rotation.
Given a focal length and assuming no translational motion, one must estimate only 3 more parameters
3-d rotations and zooms:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> 3-d rotations and zooms
Point in 3-d space
Translation, Scale and Rotation matrices
Recovering 3-d direction from 2-d projection coordinates
Point in 2-d space Homogeneous coordinates
Perspective transformation exists described by
Rotational matrix equation
Inverse equation.Assuming no translation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Perspective projection between two images
Rewriting the equation as parameter based. The Parameter is the angular velocities vector
> 3-d rotations and zooms
The parameter matrix presented in rotational model
Rodriguez formula for constructing rotation matrix from tree angles
Angular velocities vector. The unknown in the equation.
Cross product operator for the rotation matrix with the tree angles
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Building the iterative parameter estimation equation
> 3-d rotations and zooms
Equivalent to the 8-parameter method iterative equation
Defining the Jacobian for the rotational model parameter estimation algorithm
The deformation matrix for the rotational model
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Significantly faster convergence caused by only 3 degrees of freedom of the parameter matrix
The same problems as by 8-parameter motion estimation exist, stucking in local minima
> 3-d rotations and zooms
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Estimating focal length
The correspondence between the two matrices
From the orthonormality of the rotation matrix
Orthogonal: all row (column) vectors have the same norm
Normal: all row (column) vector have dot product equal to null
Estimating f0 (analog f1)
Estimating the focal length from the initial mosaic:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Always present due to errors
Closing by registering the first image at the end – the quotient of the parameter matrices gives the error coefiticent
Converting the error into angle and update the focal length
Registering the sequence with the new focal length
> Estimating focal length
Closing the gap in the panorama:
Example for gap in a panorama by registering the first image at the end
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Computational effort for taking a single gradient step consisist of wapring (resampling) the target image, computing the local intensity errors and gradients and accumulation of A and b. Very computationaly expensive Such precisenes is not requried
Approximation Divide the image into little patches (8 x 8)
Approximate the Jacobian Assume that the Jacobian is constant for each patch Evaluate it only once at patch center Calculate A and b with this approximation
Image divided into patches
Patch-based Alignment:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Block matching technique Search the minimum intensity
difference between target patch and reference image in a given neighbourhood
Framework Divide the image into patches Estimate the best integral shift
for each patch which will best align the two image
Calculate a displacement and use it to calculate new patch hessian and gradient matrices
Limitations Very computationaly expensive Apply only on the coarsest level
of the refinement image pyramid
Searching for the best integral shift by matching patch with a neighborhood
> Patch-based alignment
Local Search:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Motivation Big mosaics show significant
amount of error due to unaviodable rotational parameters focal length estimation errors.
The accumulated error causes visible image gaps and artifacts.
Bundle block adjustment technique Optimizes globally the
transformation parameters.
Panoramic Image Mosaics
Block adjustment is a feature-based technique. Establishing point
correspondences Usind local seach algorithm for
correcting the accumulated error
Bundle block adjustment of a feature point over tree
overlapping images
Block Adjustment (Global Alignment):
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Block Adjustment > Point correspondences
Dividing the image into patches (16 x 16)
Filter the patches Only patches which have interesting gradient
map Corners, Edges and Lines All patches with uniform texturing are
excluded
Use the patches as feature points with Local Search algorithm to build the corespondences
Point correspondences by rotation
Establishing the point correspondences:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
Estimating the homography matrix as to minimize the difference in screen coordinates for all the images
Problem – the gradients with respect to motion parameters are complicated
Solution – Minimizing distance between ray directions of corresponding points usint rotational representation Minimizing the distance to the center
Correct the Mk as to minimize the distance
Converting from k into l coordinate system
The perfect match is in the centre so the bundle of rays must converge to that point
Minimizing the distance between every two images
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Block Adjustment
Calculating the bundle ray direction Used in the deghosting algorithm
Direction of 3d bundle ray from 2d point in image l
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Motivation Addresses problems, which cannot be solved
with the global alignment, due to deviations from the perfect rotational (projective) model used.
Radial distortion Moving objects Motion parallax
Framework Computing the optical flow over the whole
mosaic to find local missregistrations.
Using inverse mapping technique to localy warp images as to neutralize the optical feld
Panoramic Image Mosaics> Deghosting
Calculates the projection of the bundle ray vector into image kThat represents the perfect position of the feature point
Defines a motion vector between the perfect and the current position
Deghosting (Local Alignment):
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Deghosting > Optical Field examples
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
The algorithms presented in this paper provide a efficient and reliable way of constructing full-view panoramic image mosaics without the need of special hardware equipment.
Conclution:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de