-
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
Mohammad Amin Sadeghi, Seyyed Mohammad Mohsen Hejrati and
Niloofar GheissariComputer Vision Group, School of Mathematics
Institute for studies in theoretical Physics and Mathematics,
Tehran, [email protected], [email protected],
[email protected]
Keywords: Image stitching, Poisson equation, Panorama, Color
correction, Registration, Image mosaicing.
Abstract: A new method for seamless image stitching is
presented. The proposed algorithm is a hybrid method whichuses
optimal seam methods and smoothes the intensity transition between
two images by color correction. Adynamic programming algorithm that
finds an optimal seam along which gradient disparities are
minimizedis used. A modification of Poisson image editing is
utilized to correct color differences between two images.Different
boundary conditions for the Poisson equation were investigated and
tested, and mixed boundaryconditions generated the most accurate
results. To evaluate and compare the proposed method with
competingones, a large image database consisting of more than two
hundred image pairs was created. The test imagepairs are taken at
different lighting conditions, scene geometries and camera
positions. On this database theproposed approach tested favorably
as compared to standard methods and has shown to be very effective
inproducing visually acceptable images.
1 INTRODUCTION
Since two registered images usually have local andglobal
intensity differences, in order to create visu-ally acceptable
images it is essential to complementthe registration process with
seamless image stitch-ing. As noted in (Hasler and Susstrunk,
2000), sev-eral factors reflect the intensities recorded by
cameras.They include: exposure variances, white balancing,gamma
correction, vignetting and digitizer parame-ters. Seamless
stitching has important applications inmany high level tasks such
as building panoramic im-ages, virtual reality, super resolution
and texture syn-thesis. Recently there have been many
breakthroughattempts in seamless image stitching such as (Levinet
al., 2003), (Uyttendaele et al., 2001), (Jia and Tang,2005a), (Jia
and Tang, 2005b).
There are two main approaches for the eliminationof seams and
artifacts in the overlapping images. Thefirst approach is to find
an optimal seam along whichsome energy function is minimized (Jia
and Tang,2005a), (Agarwala et al., 2004). Jia and Tang (Jia
andTang, 2005a) have described different possible energyfunctions
(weighting functions) that might be use tofind the optimal seam.
These energy functions maybe based on intensity or gradient
differences of thetwo images. Dynamic programming or graph cut
is
used to minimize the energy function. One advantageof optimal
seam methods is that they are applicableto dynamic scenes. This is
because the optimal seamdoes not pass through areas of high
gradient disparity.However, optimal seam algorithms are less
suitablewhen the illuminations of the input images are not thesame
or their overlapping area consists of thin strips.
The second approach is based on enhancing theintensity values of
one or both images so that the re-sulting stitched image looks
natural. This task can beperformed either globally or locally.
Szeliski and Shum (Shum and Szeliski, 2001) pro-pose a method
for building panoramic images capa-ble of correcting local and
global intensity misalign-ments. Their feathering algorithm results
in more nat-ural blending effects. Still in many cases it
producesunnatural seams under different illumination condi-tions.
Belt and Adelson (Burt and Adelson, 1983)use multiresolution
splines for color correction. Uyt-tendaele et al. (Uyttendaele et
al., 2001) proposed amethod capable of stitching multiple images
taken atvarious exposures and illuminance conditions. Theyallow for
large scene changes or misregistrations be-tween two images. Levin
et al. (Levin et al., 2003)introduced Gradient Domain Image
Stitching (GIST)method. GIST utilizes on an optimization
algorithmto minimize a cost function over image derivatives.
275
-
This cost function is a dissimilarity measure betweenthe
derivatives of the resulting stitched image and thederivatives of
the input images. Jia and Tang (Jia andTang, 2005b) proposed a
tensor voting method forglobal and local intensity alignment. They
use a sym-metric tensor representation to model the joint
imagespace (source image and target image intensities) as atoken.
They extract curves in the image and let tokenson a curve
communicate mutually to reinforce eachother. A dense tensor field
postulates smooth connec-tions among tokens and ensures natural
color transi-tions. This method can successfully handle
occlusionand motion blur. Their local intensity alignment
ap-proaches is devised to correct vignetting effects.
We have observed that a hybrid method that cantake advantage of
the strengths of both main ap-proaches, generates more reliable
results. Thereforethe proposed hybrid method first finds an
optimalseam to stitch two images. Then it uses the Pois-son
equation with mixed boundary conditions in thecolor domain. As a
result, the local color differencesbetween two images are
corrected. Poisson equationhas been already used for image editing
(Perez et al.,2003) or correcting structure misalignments
producedby stitching (Jia and Tang, 2005a). An advantage ofthe
proposed method is its low computational timethat makes it
particularly desirable for applications tohigh resolution images.
The proposed method relieson a model based registration algorithm
to align twoimages (Gheissari et al., 2008). In fact, the
registra-tion algorithm computes the transformation parame-ters
after recovering the transformation model.
This paper is organized as follows: Section 2,gives a brief
summery of the registration methodused. Methods for finding optimal
cutting curve aredescribed in section 3. In section 4, the
proposedmethod, Poisson Local Color Correction (PLCC) isexplained.
Different boundary conditions for the so-lution of the Poisson
equation are described in thefollowing section (4). Some
challenging images forlocal color correction and the results of
applying theproposed algorithm on them are shown in the
section5.
2 MODEL-BASED IMAGEREGISTRATION
In this work, a registration method introduced in(Gheissari et
al., 2008) is utilized. In summary thisregistration method first
finds a set of SIFT inter-est points (Lowe, 1999) and the
correspondences be-tween them. The outliers are eliminated to
gener-ate an outlier-free set of correspondences. This is
achieved by robustly fitting a translation model to
thecorrespondences. Next all models in the model libraryare fitted
to the above set of inliers.
(a) (b)
(c) (d)
(e)
Figure 1: Different transformation models for registration:(a)
Translation, (b) Euclidian (c) Similarity, (d) Affine and(e)
Projective transformation.
CAIC is applied to select the correct transforma-tion model from
a library of candidate models (Boz-dogan, 1987). Figure 1 shows the
set of candidatemodels. The final set of inliers is determined by
ap-plying the chosen model to the data set. The algorithmestimates
the parameters of the chosen model (basedon the set of inliers) to
generate the transformationmatrix.
Before finding the optimal seam and using thePoisson equation to
correct color difference, we ap-ply the transformation matrix to
the source image.This results in two registered images with an
over-lapping area which can be a triangle, a rectangle, ora higher
order polygon depending on the transforma-tion model.
3 OPTIMAL SEAM METHODS
Optimal seam algorithms are used widely in appli-cations such as
pattern synthesis and image stitching(Kwatra et al., 2003), (Jia
and Tang, 2005a), (Uyt-tendaele et al., 2001). These methods find a
cuttingcurve in the overlapping area and stitch the images
VISAPP 2008 - International Conference on Computer Vision Theory
and Applications
276
-
along the cutting curve as their common boundary.The main
approach in optimal seam algorithms is tospecify a weight for each
pixel belonging to the over-lapping area. Then a curve that
minimizes this weightfunction is selected as the optimal cutting
seam.
(a)
(b)
(e)
Figure 2: (a) Two registered images and their overlappingarea,
(b) optimal cutting curve, (c) final stitched images.
Assume we have registered images IT and IS. Theoverlapping area
is IovT in image IT and IovS in image IS(as shown in figure 2). We
would like to cut and stitchIT and IS along an optimal curve C.
Let C be a cutting curve, then the cost of having Cas cutting
curve is defined as:
W (C) = pC
W (p), (1)
where W (p) is the value of the weight function asso-ciated with
pixel p. The optimal curve(seam) is thecurve which minimizes the
cost W (C).
Different candidates for weight function are
Wg(p) = ||IS(p)IT (p)||nn, (2)and
Wi(p) = ||IS(p) IT (p)||nn, (3)where ||.||nn is the nnorm. A
linear combination ofWg(p) and Wi(p) could also be considered as
weightfunction (Jia and Tang, 2005a). We have tested differ-ent
energy functions and observed that the function
W (p) = ||IS(p)IT (p)||11 (4)
(a) (b)
(c) (d) (e)
( f ) (g) (h)Figure 3: (a) And (b) are the input images. The
images areborrowed from figure 6 of (Jia and Tang, 2005a). The
im-ages, used in our experiments, are of our size 337 334.(c)
cutting curve using intensity based weight functionWi(p) = ||IS(p)
IT (p)||11. (d) Stitching using intensitybased weight function
without color correction (withoutPLCC) (e) Stitching using
intensity based weight functionwith color correction (PLCC) (f)
cutting curve using gra-dient based weight function Wg(p) =
||IS(p)IT (p)||11.(g) Stitching using gradient based weight
function withoutcolor correction (without PLCC) (h) Final result
after usinggradient based weight function and applying PLCC
method.
is more reliable than the other candidates speciallywhen the
illuminations of IT and IS are different. Af-ter finding the
optimal seam PLCC is applied to cor-rect color and illumination
differences between twoinput images. Setting n = 1 (using 1norm)
leadsto better results because small intensity differencesare
ignored in higher norms. We use a 3-dimensionalRGB color vector for
each pixel as I(p), and observedthat using HSV color space has the
same performanceas the RGB color space.
The result of using intensity based weight func-tion and
gradient based weight function is shown inFigures 3 and 5.
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
277
-
There are various algorithms with different com-plexities for
finding the optimal cutting curve. Min-Cut(Max-flow) algorithm
(Kwatra et al., 2003) worksaccurately. However due to its high
complexity, it im-poses significant computational burden on images
ofhigh resolution. We used a combination of dynamicprogramming and
BFS (Breadth First Search) algo-rithm in our experiments to find
optimal cutting curve.Its complexity is O(n) where n is the number
of pixelsin the overlapping area.
Figure 4: In some cases the optimal seam is not a
simplecurve.
In some cases the optimal seam gets more com-plex as shown in
Figure 4. Therefore an optimal seamis not necessarily one simple
curve.
4 COLOR CORRECTION BASEDON POISSON EQUATION
Poisson equation has been used for image editingand image
alignment applications, for example (Perezet al., 2003), (Jia and
Tang, 2005a) and (Agarwalaet al., 2004). Here we use Poisson
equation to cor-rect color differences between two already
registeredimages.
Lets be the region (in IS) that lies on one side ofthe optimal
seam (the side belonging to the source im-age). The boundary of is
denoted by . We sep-arate into two parts: the common boundary
withIT (referred to as C), and the other boundaries (de-fined as
B). Our notations are illustrated in Figure6.
To eliminate artifacts and color differences overthe seam C, we
define a correction function on as
IS = IS + (5)where IS is the corrected image. Then the
Poissonequation is solved on and the resulting correctionfunction ,
is added to IS to compensate for color dif-ferences between two
images. For colored images,we apply the same method on each color
channel ofthe input image. According to our experiments
theperformance of the method is the same in RGB and
(a) (b) (c)
(d) (e) ( f )
(g) (h) (i)Figure 5: Illustration of applying PLCC on two
images.(a) and (b) input images of size 800 600. (c) the resultof
putting two registered images together without applyingany
corrections. (d) a cutting curve found by using inten-sity based
weight function. (e) the solution of the poissonequation solved on
the corresponding boundary. (f) the re-sult of stitching the images
using an intensity based seamline. (g) shows a cutting curve found
using Gradient basedweight function. (h) the solution of the
poisson equationassociated with the cutting curve in (g) and (i)
the final re-sult of stitching using intensity based seam line. As
youcan see, the color is not corrected perfectly in (f) becauseits
corresponding cutting curve has passed dark areas andthe boundary
conditions of the poisson equation in (e) is notvalid.
Figure 6: Notation of the region : is the region of thevisible
part of IS. The common boundary with IT is definedas C, and the
other boundaries are defined as B.
HSV color spaces. For solving Poisson equation on several
boundary conditions could be assumed. The
VISAPP 2008 - International Conference on Computer Vision Theory
and Applications
278
-
(a) (b)
(c) (d)
(e) ( f )Figure 7: Comparison between the results of using
Drich-let boundary conditions or Neumann boundary conditionson B.
(a) and (b) source and target images, both of size800600. (c) the
solution of Poisson equation with Drich-let boundary conditions on
B as color correction factor.(d) the respective result of color
correction. It is equivalentto applying Poisson image editing
(Perez et al., 2003) (e)the solution of Poisson equation with
Neumann boundaryconditions on B and (f) the the result of color
correctionusing the correction factor (e). The correction factor
(c) and(e) are solved on a rectangular domain before applying
thetransformation and added to source images after
transfor-mation.
Dirichlet boundary condition equal to the differencebetween
intensity of two input images along C wasimposed on C. For B the
Neumann boundarycondition was used.Imposing the proper boundary
condition is critical inour application. One might use Dirichlet
boundarycondition on B (equation 6). However, the prob-lem here is
the choice of the boundary function onB. For the Dirichlet boundary
condition, an ap-proximation to the value on B is necessary.
Thetrivial approximation zero for the boundary value re-sults in
imperfect color correction as shown in Fig.7.
= 0 over with
= IT IS, On C
= Constant, On B,(6)
where denotes the Laplace operator.
The Neumann boundary condition on B inPLCC (equation 7) produced
significantly better re-sults. The superior performance of the
Neumannboundary condition on B is explicable in view ofthe fact
that it does not fix intensity values along Bbut constraints the
gradient values along B to bezero. Fixing the intensity on B
results in tobe constant over B, while the ideal values of over the
are not necessarily constant. Further-more, there is no
straightforward theoretical way toset the values of over B and an
approximation isinevitable. This problem is aggravated where the
il-lumination condition between the input images
differsignificantly.
= 0 over with
= IT IS, On Cn = 0, On B,
(7)
where n is normal vector at . This Poisson equa-tion is the
result of the following minimization prob-lem (equation 8).
(a)IT (b)IS
(a) (b)
(c) (d)Figure 8: Applying PLCC and optimal seam to dynamicscene.
(a) and (b) input images of size 800600. (c) Opti-mal seam based on
gradient disparities. (d) Stitching with-out using optimal seam.
The shadow of pedestrians walk-ing behind is visible, while the
pedestrians body are elimi-nated from the stitching result. The
color difference is veryprominent. (e) Stitching using optimal
seam. The shadowof pedestrians are no longer visible. The color
differences isstill prominent (f) Final result of applying color
correction(PLCC) and optimal seam.
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
279
-
(a) (b)
(c) (d)
(e)
( f )Figure 9: Creating super resolution images using imagesof
poor quality. (a), (b), (c) and (d) are input images (of apuzzle
hanged to a wall) taken at four different viewpointsand
illumination conditions. input image (a) is taken un-der yellow
light source, (b) taken under yellow and whitelight sources, (c)
taken at low exposure time under whitelight source and (d) taken
under white light source. (e) thestitching result without PLCC. (f)
the stitching result afterPLCC.
min
||2 with
= IT IS, On Cn = 0, On B,
(8)
where the gradient operator [ x ,y ] is represented by
. This minimization problem results in visually nat-ural output
images because it minimizes the 2normof the gradient of intensities
near the boundary.
We used the method of finite differences and thecentral
difference formula to solve the Poisson equa-tion and the resulting
system of linear equations.
5 EXPERIMENTAL RESULTS
A database for testing and evaluating image stitchingalgorithms
was created. This database contains overtwo hundred pairs of images
from over forty scenes.All pairs are available in three different
standard sizes.The scenes vary in color, camera properties,
cameralocation and lighting. The database will be publiclyavailable
at math.ipm.ac.ir/vision. The transforma-tion matrix for each pair
of images was determinedusing (Gheissari et al., 2008). This
database containschallenging images for registration, color
correction,and building panoramic images.
As mentioned before the optimal seam method isapplicable to
dynamic scenes. This is because the op-timal seam does not pass
through areas of high gra-dient disparity. This capability is shown
in figure 8.As can be seen from this figure, two different groupsof
pedestrians are present in the input images. Theoptimal seam passes
through the narrow gap betweenthese two groups and therefore both
groups are pre-sented in the resulting stitched image. If the
opti-mal seam has not been used, then the shadow of thepedestrian
group behind would be visible without thepedestrians
themselves.
Performance of the proposed method is comparedwith some standard
competing methods. As shown infigure 10, a combination of PLCC and
optimal seamoutperforms feathering (Uyttendaele et al.,
2001),pyramid blending (Adelson et al., 1984), GIST (Levinet al.,
2003) and Poisson image editing (Perez et al.,2003).
The proposed algorithm is desirable for makingsuper resolution
images due to its high accuracy andlow time complexity. To show
this capability wechose four images of a puzzle hanged to a wall.
Theimages were taken at four different view points andsignificantly
various illumination conditions as shownin figure 9. The results of
applying the proposed al-gorithm shows its success .
VISAPP 2008 - International Conference on Computer Vision Theory
and Applications
280
-
(a)IT (b)IS
(c)
(d)
(e) ( f ) (g) (h) (i) ( j)Figure 10: Comparing PLCC with some
competing meth-ods: (a) and (b) are input images borrowed from
figure 4of (Levin et al., 2003). The images are of size 348 255and
290257 respectively. (c) Stitched image after apply-ing optimal
seam , (d) The color correction result generatedby PLCC after
finding optimal seam, (e) Feathering (Uyt-tendaele et al., 2001),
(f) Pyramid blending (Adelson et al.,1984), (g) Feathering on the
gradients (Uyttendaele et al.,2001), (h) Poisson editing (Perez et
al., 2003), (i) GIST(Levin et al., 2003) (j) The result of PLCC
method.
6 CONCLUSIONS
A new method for seamless image stitching was pre-sented. The
proposed algorithm is a hybrid method,that combines optimal seam
approaches and methods
which smooth the intensity transition between twoimages (by
color correction). Using a dynamic pro-gramming algorithm, an
optimal seam along whichgradient disparities are minimized is
determined. Alocal color correction method based on Poisson
equa-tion is used to enhance color variations between im-ages.
Different boundary conditions to solve the Pois-son equation were
investigated and tested. We con-cluded that using mixed boundary
conditions gen-erates the most accurate results. To evaluate
andcompare the proposed method (with some competingmethods), a
large image database consisting of overtwo hundred image pairs was
utilized. The test im-age pairs are taken at different lightening
conditions,scene geometry and camera positions. Compared tostandard
methods, proposed approach performed fa-vorably on this database
and has shown to be veryeffective in producing visually acceptable
images.
ACKNOWLEDGEMENTS
We would like to acknowledge Dr. Mohammad RezaMokhtarzade for
his valuable advices on solvingPDEs. In addition, we would like to
acknowledgeDr. Mehrdad Shahshahani for his kind helps in
thisproject and in writing this paper. Mr. Mostafa Ka-mali Tabrizi
helped us in modifying the source codefor registration. We
appreciate the valuable discus-sions he had with us. Finally we
thank Zohre Sharafi,Parisa Mirshams, Paria Mehrani, Sheyda Beigpoor
forproviding us with beautiful test images. This projectwas
supported by Institute for studies in theoreticalPhysics and
Mathematics (IPM).
REFERENCESAdelson, E. H., Anderson, C. H., Bergen, J. R., Burt,
P. J.,
and Ogden, J. M. (1984). Pyramid methods in imageprocessing. RCA
Engineer, 29(6).
Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S.,Colburn,
A., Curless, B., Salesin, D., and Cohen, M.(2004). Interactive
digital photomontage. ACM Trans.Graph., 23(3):294302.
Bozdogan, H. (1987). Model selection and akaikes infor-mation
criterion (aic): The general theory and its ana-lytical
extensions.
Burt, P. J. and Adelson, E. H. (1983). A multiresolutionspline
with application to image mosaics. ACM Trans.Graph.,
2(4):217236.
Gheissari, N., Tabrizi, M. K., Sharafi, Z., and Mirshams,
P.(2008). Model-based global image registration. InProc. of the 3rd
International Conference on Com-puter Vision Theory and
Applications, page to appear.
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
281
-
Hasler, D. and Susstrunk, S. (2000). Colour handling inpanoramic
photography.
Jia, J. and Tang, C.-K. (2005a). Eliminating structure
andintensity misalignment in image stitching. In ICCV05:
Proceedings of the Tenth IEEE InternationalConference on Computer
Vision, pages 16511658,Washington, DC, USA. IEEE Computer
Society.
Jia, J. and Tang, M.-C.-K. (2005b). Tensor voting for im-age
correction by global and local intensity alignment.IEEE Trans.
Pattern Anal. Mach. Intell., 27(1):3650.
Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick,
A.(2003). Graphcut textures: image and video synthe-sis using graph
cuts. In SIGGRAPH 03: ACM SIG-GRAPH 2003 Papers, pages 277286, New
York, NY,USA. ACM Press.
Levin, A., Zomet, A., Peleg, S., and Weiss, Y. (2003).Seamless
image stitching in the gradient domain.
Lowe, D. G. (1999). Object recognition from local
scale-invariant features. In Proc. of the InternationalConference
on Computer Vision ICCV, Corfu, pages11501157.
Perez, P., Gangnet, M., and Blake, A. (2003). Poisson im-age
editing. In SIGGRAPH 03: ACM SIGGRAPH2003 Papers, pages 313318, New
York, NY, USA.ACM Press.
Shum, H.-Y. and Szeliski, R. (2001). Construction ofpanoramic
image mosaics with global and local align-ment. pages 227268.
Uyttendaele, M., Eden, A., and Skeliski, R. (2001). Elim-inating
ghosting and exposure artifacts in image mo-saics. pages II509
II516 vol.2.
VISAPP 2008 - International Conference on Computer Vision Theory
and Applications
282