Image Appearance Exploration by Model-Based Navigation › ~dcor › articles › 2009 › Image-Appearance.pdf · L. Shapira & A. Shamir & D. Cohen-Or / Image Appearance Exploration

EUROGRAPHICS 2009 / P. Dutré and M. Stamminger(Guest Editors)

Volume 28 (2009), Number 2

Image Appearance Exploration by Model-Based Navigation

L. Shapira1, and A. Shamir2 and D. Cohen-Or1

1Tel-Aviv University, Israel2Interdisciplinary Center Herzliya, Israel

Abstract

Changing the appearance of an image can be a complex and non-intuitive task. Many times the target image colorsand look are only known vaguely and many trials are needed to reach the desired results. Moreover, the effect ofa specific change on an image is difficult to envision, since one must take into account spatial image considera-tions along with the color constraints. Tools provided today by image processing applications can become highlytechnical and non-intuitive including various gauges and knobs.In this paper we introduce a method for changing image appearance by navigation, focusing on recoloring im-ages. The user visually navigates a high dimensional space of possible color manipulations of an image. He caneither explore in it for inspiration or refine his choices by navigating into sub regions of this space to a specificgoal. This navigation is enabled by modeling the chroma channels of an image’s colors using a Gaussian MixtureModel (GMM). The Gaussians model both color and spatial image coordinates, and provide a high dimensionalparameterization space of a rich variety of color manipulations. The user’s actions are translated into transfor-mations of the parameters of the model, which recolor the image. This approach provides both inspiration andintuitive navigation in the complex space of image color manipulations.

Categories and Subject Descriptors (according to ACM CCS): Computer Graphics [I.3.6]: Interaction Techniques—Image Processing [I.4.3]: Enhancement—Image Processing [I.4.9]: Applications—Image Processing [I.4.10]: Im-age Representation - Multidimensional—

1. Introduction

The shift into digital form opened up vast possibilities forimage editing and manipulations. One of the more funda-mental manipulation types is changing the appearance of theimage by applying color modifications. Such modificationsare needed to change the tone or mood of the image, to fita given style or design, or to re-select the color palette foraesthetic or artistic reasons.

Nevertheless, such color manipulations are highly chal-lenging. Color spaces are highly complex and non-intuitive.Color modifications are difficult to describe and, typically,numerous experiments are needed to reach the desired re-sult. For instance, even when the target palette of colors isgiven or taken from an example image, it is difficult to findthe correct modifications to apply these colors to a differ-ent image. Moreover, many times the artist does not have aclear concept of the desired modification and inspiration orexploration are needed to assist in reaching this goal. Lastly,

altering the chroma of an image is highly unstable. Modifi-cations in color space are perceived in image space, and aslight modifications in color may cause large artifacts in theimage. Hence, any modification of colors must take into ac-count constraints in both color space, and the spatial neigh-borhoods on the image. In summary, both envisioning andapplying desired color modifications to an image are diffi-cult tasks.

Tools available today for color manipulations, requiremanual knobs-tuning in color space or histogram manipu-lations. There are several drawbacks in this approach. First,there are many different colors in an image and it is tediousto manually change them individually. Second, the resultsof manipulations in color space are often unexpected, sincethere is no visual link between the user’s actions and theireffect on the image. Third, working only in color space withno spatial constraints may cause noticeable artifacts to ap-pear.

c© 2008 The Author(s)Journal compilation c© 2008 The Eurographics Association and Blackwell Publishing Ltd.Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and350 Main Street, Malden, MA 02148, USA.

L. Shapira & A. Shamir & D. Cohen-Or / Image Appearance Exploration by Model-Based Navigation

Figure 1: Exploring image appearance: Starting from the original image on the left, the user makes a series of selections in agallery interface. Each selection presents more refined variations, until finally, in the left, the user selects his final result.

In this paper we use a navigation metaphor to create andinspire image color changes. We first model the distribu-tion of the color channels of the image with a spatially con-strained Gaussian Mixture Model (GMM). The parametersof the GMM are combined to form the appearance search-space defining as all possible color manipulations by simpletransformations of the model. Instead of tediously experi-menting with different parameter values and guessing theiroutcome, we use a WYSIWYG approach. We present theuser with a set of image variants and allow navigation inspace by direct user selections. The actual model is transpar-ent to the user, while it is implicitly affected by his actions.Navigation assists both in inspiring and in applying specificcolor modifications to an image (Figure 1).

At all times, the user is provided with a visual represen-tation of the appearance search-space and the navigationpath. The system allows the user to guide the navigationin an iterative manner by choosing specific variants of im-ages and synthesizing new image variations from them. Weuse a Monte Carlo approach to create samples in regions ofthis high dimensional space, and create image color variantsguided by the user choices. The user can traverse backwardsor forward in his path, jump to other regions of space, or useadvanced tools that guide the search by controlling the varia-tions of images. These include modifying the model param-eters on the image with a brush, modifying the GMM palettein directly model space, or matching a target image palette.Using these tools, the user’s actions are translated into trans-formations on the Gaussians of the model, focusing the nav-igation in a new region in the search-space. New images aresynthesized around this position and a new gallery is created,providing direct depictions of his actions. This approach (asdemonstrated in Figure 1) is both inspirational, allowing toexplore new variants that might not have been considered,and effective, providing means to narrow down towards adesired image goal.

2. Related Work

There are many effective tools and algorithms for image ap-pearance editing which concentrate on the luminance chan-nel of an image, or manipulate gray-level images. For in-stance, histogram equalization of the luminance channelis one of the oldest color correction algorithms. More re-cently, works such as [BPD06, LFUS06] apply manipula-

tions to the luminance channel for tone mapping or com-pressing high dynamic range images [RWPD05]. Never-theless, such approaches are not easily extendible to multidimensional chroma channels. Dealing with each channelseparately is undesirable since the channels are highly cor-related and manipulating them separately leads to artifactsand limits the type of manipulations that can be performed.In [AP08] changes to image appearance are seeded by userbrush strokes and propogated to similar areas in the image.

Colorization algorithms attempt to re-color a gray scaledimage [LLW04,LWCO∗07] using scribbles marked by hand.Recently, several color manipulation works have used an ex-ample image as inspiration for recoloring an image. Giventhe example target image, the input image’s colors canmatch those of the example target image by either histogrammatching [GW01, PKD05], parametric matching [RAGS01,TJT05] or by non-parametric sampling [ICOL05]. However,finding the correct image to be used as an example is notalways easy. Moreover, there are times when the need forcolor manipulation is not guided by an image, but more byabstract aesthetic ideas or even by pure exploration.

There are other model based approaches for color manip-ulation in literature. In [OW04] the concept of color linesis introduced for the RGB color space. Manipulating theselines allows for simple yet believable color changes in the

Figure 2: (a) The original image, (b) The distribution ofpixels in the Hue and Saturation space, (c) The histogramfunction of pixels in HS, (d) A GMM model of the HS his-togram with 3 clusters, (e) A GMM model with 5 clusters.

c© 2008 The Author(s)Journal compilation c© 2008 The Eurographics Association and Blackwell Publishing Ltd.


image. In [COSG∗06] a psychological color model is used,in which the hue and saturation of the pixels compose theradial 1-dimensional color space. Their algorithm attemptsto ‘shift’ an image to match one of a set of predefined mod-els, hence improving its harmony. In contrast, our work useshigher dimensional models and implements more flexiblemanipulations, which are influenced by user choices and nota pre-established template.

In [MAB∗97] parameter-based representations of a high-dimensional data are displayed in a Design Gallery, whichis a set of variation created for the user to choose from. Adistance metric is defined on the high dimensional space,ensuring that options displayed in the gallery differ fromeach other. Design galleries have been shown to be usefulfor volume rendering transfer functions, 3D scene lightingplacement, and more [JKM01]. In [Ado07], a feature calledBrainstorm utilizes a design gallery based interface for ani-mation variations. Brainstorm seems to operate on indepen-dent 1-D parameters, by displaying random combinations ofvalues for the user. Unfortunately, there is no technical pub-lication to explain it. Our mechanism is visually similar increating a set of image variations. However, our focus is toenable meaningful navigation of the parametric space rep-resenting a D-dimensional color space and not only displayvariations.

3. Our Approach

The space of all color variations of a given image contains analmost infinite number of possibilities and is extremely highdimensional. Furthermore, there is no metric that can ade-quately express the perceptual distances among these varia-tions. Our goal is to model the space by a structured appear-ance search-space with a relatively small number of dimen-sions and a given metric.

First, we calculate the color distribution of an image us-ing one of two color spaces, namely the HSV or L*A*B*color space. Next, we model this distribution using a Gaus-sian mixture (GMM) with a small number of Gaussians (SeeFigure 2 for an example where an image is modeled by 3and 5 Gaussians). We limit all possible color variations bypermitting only variations defined by the set of transforma-tions applied to the parameters of the GMM color model.The dimension of the appearance search space is thereforethe number of parameters times the number of Gaussians inthe model. In practice, we allow only translations, rotationsand scaling of the individual Gaussians arriving at d < 30.Each point in the search space represents one color varia-tion of the original image (see Figure 3). The actual imagevariation can be synthesized by first accounting for all thetransformations applied to the model and then changing eachpixel’s color by transforming it in a similar manner as its as-sociated Gaussian in the model.

Although the search space is of relatively low dimensioncompared to the space of all color variations of a given

image, it is still difficult to grasp and explore. To assistusers in navigating towards their favorite color variation weuse a simple 2D interface. First, measuring the distance be-tween two mixture variants [SCLG05] induces a metric onthe appearance search-space. Using this metric and multi-dimensional scaling we project samples from the search-space onto 2D and display them to the user as a gallery ofimages in the foreground. In Section 4 we describe the modelconstruction and the metric induced.

Initially, we fill the gallery with random samples from thesearch space employing Monte Carlo sampling. Next, theuser navigates by defining regions in space which interesthim. This is done using one of several methods. The mostbasic one involves simply selecting one or more favored im-ages from the foreground. A new set of samples from therelevant subspace are created and displayed for further ex-ploration, while the old set of images are moved to the back-ground. The user can go backward or forward in his searchpath, or jump to other positions in space. This promotes atop-down perceptual dominant order. At first strong varia-tions are created by shifting the hues of the model, whilelater more subtle transformations are applied. At all timesthe user can control the degree of variation by setting theamount of randomness in the sampling process.

The user can also initialize the navigation to a more spe-cific region of the search space by using more direct manip-ulation tools. He can modify the Guassians directly in modelspace using a palette editor. He can choose an example im-age and match the model to its colors. He can directly painton the image with a global brush tool that is used to modifyall pixels that are members in one Gaussian simultaneously.This approach provides direct visual connections betweensimple user actions and the resulting modified images, it isinspiring and does not require an understanding of complexnotions and parameters. The navigation technique is illus-trated in Figure 1 and explained in Section 5.

4. Modeling Color Manipulations

4.1. The Image Color Model

The key to effective navigation is to model it with a relativelysmall and structural space that we call the appearance searchspace. The search space definition is based on a Gaussianmixture model of the selected color channels of the image.We have chosen to use two possible color spaces: the HSVcolor space, that describes perceptual color relationships ina natural way to users, and the L*A*B* color space, that re-tains perceptual distance between colors. We use an acceler-ated Expectation Maximization (EM) algorithm [VNV06] tocreate a model with k components. This algorithm iterativelycomputes clusters C1...Ck of a given distribution (Figure 2).In every iteration it splits the least probable cluster into twoclusters. Each cluster is represented by its mean and covari-ance matrix. Each pixel in the image is associated with aprobability vector p = (p1, . . . , pk) such that pi = P(x|Ci)



Figure 3: An example of five variant images created fromfive vectors sampled from the search space. The top left im-age is the original one. We first show three simple transfor-mations, applied to a single Gaussian (translation, rotationand scaling). The two bottom images represent more com-plex color manipulations.

and ∑i=1..k pi = 1. We have found that a low value of k (3 to6) is sufficient for the GMM to provide a good approxima-tion of the color distribution of an image, while still allowinga rich space of variation.

4.2. Spatial Considerations

The basic GMM model is defined by considering only thecolor distribution. To take into account image spatial con-siderations, we interleave EM iterations with a graph-cutstep which increases the probability of neighboring pixelsto belong to the same Gaussian. Every t iterations(typicallyt = 3), during the expectation part of the EM algorithm, weapply an alpha-expansion graph-cut algorithm [ZK04] onthe current Gaussians. The graph-cut uses a natural imageboundary map [MFM04] as a smoothness factor and resultsin a labeling of each pixel, in which pixels in neighboring ar-eas tend to be labeled in the same manner due to the smooth-ness term of the graph-cut. This labeling is used as input forthe next Maximization step. We have found that employingthis method reduces artifacts when dealing with noisy andhighly textured images. The graph-cut optimization mini-mizes the following energy functional, which is built frome1 the data term, and e2, the smoothness term:

E( f ) = ∑p∈IMG

e1(p, fp)+λ ∑{p,q}∈N

e2( fp, fq)

e1(p, fp) = − log(P(p| fp)+ ε)

e2( fp, fq) ={

1/(1+max(edge(p),edge(q))) fp �= fq0 fp = fq

where fp is the label assigned to pixel p, linked to thehighest probable Gaussian in the GMM model. edge(p) isthe value of pixel p in an image boundary map [MFM04]. Nis the set of pairs of 4-neighborhood pixels in the image, λ isa parameter defining the degree of smoothness (we’ve usedλ = 2 for all images we’ve encountered, regardless of theirscale or composition).

Given a GMM model we can calculate for each pixel x ,a probability vector px such that px(i) is the probability ofpixel x to belong to the i− th Gaussian. This in effect definesa vector field over the image, creating a soft segmentation.To further reduces artifacts in the synthesized images, we ap-ply a median filter over this vector field. This constrains co-herent areas in the image to undergo similar transformations.In figure 4 you can see the effect of the spatial considerationimprovements on the generated color image variant.

4.3. Appearance Search Space

The actual appearance search space, containing all the pos-sible color manipulations in our model, is the space of allpossible affine transformations on all k Gaussians of the mix-ture. For each Gaussian Ci we allow the following transfor-mations:

• Translation - shift the mean µi of Ci. When dealing with aradial dimension such as Hue (in HSV), the shift is cyclic.

• Rotation - apply a rotation on the covariance matrix Σi ofCi.

• Scaling - Scale the principal axes of the covariance matrixΣi, elongating or shrinking the Gaussian along its Eigen-vectors.

Figure 4: (a) Original image (b) An image variant with nospatial considerations (c) spatial considerations are applied,a refinement of the GMM using a natural edge map, and amedian filter on the soft clustering probability vectors of thepixels.



When changing only the chroma of an image (using only2-channels), we use two translation values, rotation angleand two scaling values (five parameters). To allow luminancechanges, we restrict the covariance such that the luminanceand color channels are orthogonal, and use three translationand scaling values, and a rotation angle for the color chan-nels only, in total seven parameters. Therefore the dimensionof the search space, d, is either 5k or 7k.

As described earlier, each d-dimensional vector v in thesearch space represents a specific set of transformation forthe GMM, defining a specific color manipulation. EachGaussian Ci has an associated affine transformation Tv de-fined by the coordinates of v related to Ci. We synthesize theactual image variant by accounting for all the transforma-tions applied to the mixture of the original image. Let µi,Σibe the original Gaussian Ci mean and covariance, and µ̃i, Σ̃ibe the transformed mean and covariance, A pixel’s color xin the original image, is translated to new position xi in theappropriate space by the Gaussian Ci using the following for-mula:

xi = Σ̃i ·Σ−1i · (x−µi)+ µ̃i, (1)

The final position xnew of x is a weighted blend of all xiby the probabilities pi taken from the GMM:

xnew =k

∑i=1

pixi (2)

5. Navigating the Search Space

The appearance search space is still a relatively high dimen-sional space. Therefore, navigation it is not trivial. We pro-vide a direct visual mechanism along with a Monte Carlosampling method to assist navigation. Each position in thesearch space is represented by a gallery of possible imagevariations sampled around that position to represent direc-tions in which the user could follow. We begin with a sparse,random sampling of the search space, and then graduallyconverge to smaller and smaller sub spaces guided by theuser’s choices (Figure 1). In each iteration we create a smallnumber of samples (usually between 6 and 16) and displaythem to the user. This creates a direct visual aid for naviga-tion, which is effective for explorations, and helps find thedesired color variation.

This approach needs three components. First, a way tosample the search space. Second, a way to project samplesfrom a d dimensional space position onto a 2D canvas fordisplay. Third, a layout mechanism of the actual images ona 2D canvas that reduces cluttering and allows easy access.

Each scalar coordinate in the search space affects oneparameter of one Gaussian in the mixture. To sample the

Figure 5: The process of generating the gallery of images:(a) The different variant vectors sampled from the searchspace represent variations of the GMM, (b) Using the dis-tances between the models we build an affinity matrix be-tween the samples, (c) We use MDS to project the vectors to2D, (d) We build a kd-tree using a median-cut algorithm in2D, (e-f) We arrange the images by mapping the kd-tree to aregular grid to reduce cluttering and overlaps.

space, we can uniformly sample each coordinate within itsrange and generate an image variation. However, uniformsampling is impractical, and furthermore, not all coordinateshave the same perceptual importance. There is a need todifferentiate between the coordinates corresponding to themeans of each Gaussian, which represent that Gaussian’scolor, and the other coordinates which describe the covari-ance. Furthermore, in the HSV model, a greater importanceis given to the Hue channel, while in the L ∗A ∗B∗ modeleach channel is of equal importance. Therefore, a biasedMonte Carlo search by random sampling is performed, giv-ing the first coordinates (mean positions) the prominent role.Later, as the user navigates into smaller sub-spaces, subtlervariations are created by fixing the mean and sampling theother coordinates.

Let vt ∈ Rd be the base vector. Initially, when t = 0,

this vector is defined to be the GMM of the original image.Later, at iteration t > 0, vt is defined by the user’s imagechoices from the gallery. vt is separated into its “major co-ordinates” and “minor coordinates”: vt = (vt

ma j,vtmin). Let

r = rand([−1,1]d), then a random sample vt+1 in our pro-cess is generated by:

v(t+1) = (vt+1ma j, vt+1

min ) = (vtma j +α · rma j, vt

min +β · rmin),

where α and β control the level of variation. Initially, β =0 and α > 0. Later, as the user converges towards a variation,β is gradually enlarged in proportion to α , but the magnitudeof both become smaller.

During the search, a user can select more than one im-age variation. By choosing several images, we get a set ofbase vectors {vt

i}, corresponding to all the selected image



color variations. In this case, new samples are derived forthe next iteration as a random weighted affine combinationof the vectors vt

i . δi noise (user configurable parameter) isadded to the interpolated samples, linearly dependent on thenumber of images selected:

v(t+1) = ∑i

wi(vti +δi)

For each one of the generated vectors, a new image varia-tion based on Equations 1 and 2 is synthesized.

To continue the search we need to display a gallery ofimages to the user. Again, modeling the color variationsenables us to define a distance measure between the im-age variations, since each image is associated with a mod-ified GMM. To properly display the images, we build anaffinity matrix from the mutual distances of these GMM’s,and project the variations onto 2D using Multi-DimensionalScaling (MDS) [CCC00]. The matrix expresses the percep-tual distance among all the images (Figure 5). The distancemetric between two GMM’s follows [SCLG05] and is de-fined in closed form as:

dist(p, p′) =− log

⎡⎢⎢⎢⎢⎢⎢⎣

2 ∑i, j

πiπ′j

√|Vi j|

eki j |Σi|∣∣∣Σ′

j

∣∣∣

∑i, j

{πiπ j

√|Vi j|

eki j |Σi||Σ j|}

+ ∑i, j

⎧⎨⎩π′

i π′j

√|Vi j|

eki j |Σ′i |

∣∣∣Σ′j

∣∣∣⎫⎬⎭

⎤⎥⎥⎥⎥⎥⎥⎦

(3)

Where, µ,Σ and µ′,Σ′ are the mean and covariance ma-trices for the kernels of the Gaussian mixtures p(x) andp′(x) respectively, i and j are the indexes on the Gaussiankernels, π{i, j},π′

{i, j} are the mixing weights of the respec-

tive GMMs, ki j = µTi Σ−1

i (µi − µ′j)+ µ′Tj Σ

′−1j (µ′j − µi), and

Vi j = (Σ−1i +Σ

′−1j )−1. Note that this measure does not have

to be precise since its main objective is to arrange the imagesin the gallery.

To reduce cluttering and image overlapping in the displaywe relax the inter-sample distance constraints and place theoutput images on a simpler, non-overlapping grid-like ar-rangement using a simple median cut algorithm. First, webuild a binary axis-aligned Kd-tree, where each leaf containsonly one sample. Each split is performed at the median of itsparent, where the axis is chosen to divide the longest dimen-sion. Finally, a trivial mapping between the tree leaves anda regular grid defines the layout of the samples (see Figure5(e)-(f)).

As the user navigates through the galleries, a path of im-ages is created. This path represents the color manipulationsthe user has already visited. These images are kept in thebackground as thumbnails, on a ring circling the currentgallery, arranged by hue and saturation (Figure 6). These

Figure 6: A screen capture of a navigation session show-ing images the user has already visited in the background.The images are placed on a ring surrounding the gallery,arranged by hue and saturation

thumbnails act as signposts, and the user can jump back andvisit them by selecting them or by backtracking his path.Still, there are times when random exploration and selectionof favored images is not enough, and users require a finer de-gree of control over the navigation. We provide the user withthree methods to initialize the navigation in a more explicitmanner.

Initialization by Palette The palette selection tool (Fig-ure 7) allows a user to move markers around on a 2-dimensional color map. Each marker is connected directlyto one Gaussian in the GMM, and moving it translates thatGaussian in the color space. Each movement the user per-forms is reflected immediately in a preview window on theimage. Once the user approves of the result, navigation con-tinues by creating a new gallery of image variations centeredaround the user’s final selection.

Initialization by Color Brush The color brush allows di-rect interaction with the image (Figure 8). Upon activatingthis tool, the user selects a preferred color, and brushes it

Figure 7: Initializing by directly editing the original palette:(a) The user changes the original palette by dragging aroundmarkers, which represent the prevalent colors in the originalimage, once a satisfied result is achieved (b) a new set ofvariations based on the result is created.



Figure 8: Initializing by brushing on colors: (a) The userselects a color and paints it directly on the image (b) The un-derlying model is shifted towards the user’s selected palette(c) A variety of results easily achieved using palette naviga-tion.

over the image. Instead of coloring all the pixels under thebrush, we first choose the most probable Gaussian under thebrush, and color just pixels belonging to this Gaussian, butglobally on the image. Given an image I, a scribble S repre-sented as a 1-bit mask over the image (defined by the pixelsunder the tool), and a brush color cbrush, we calculate p̂, thevector of probabilities for the scribble to belong to a specificGaussian in the mixture as follows:

p̂ = (p̂1, ..., p̂k) = ∑i∈mask

(p1(i), ..., pk(i))/ |mask| ,

where i is a pixel in the image, (p1(i), . . . , pk(i)) is a vec-tor containing the probability for pixel i to belong to Gaus-sian j in the GMM. We select the Gaussian Cj with thehighest probability j = argmax(p̂1, . . . , p̂k), and translate itsmean towards cbrush.

Note that our color brush is somewhat similar to the ap-pearance editing strokes presented in [AP08]. In their ap-proach, the user’s changes are applied to areas under thebrush stroke and propagated. The propagation is done bysolving an optimization problem, using the fact that the sim-ilarity matrix of the image is low rank. In our work, we usethe coherency of the image while modeling its color distri-bution. Therefore, we are able to directly apply the changeson the corresponding Gaussian.

Initialization by Example Image Selecting an exampleimage as an inspiration for navigation is a convenient wayto capture the color palette of that image in one operation,rather than trying to manually recreate it [PKD05, RAGS01,TJT05]).

Given a source image S and an example image T , we

Figure 9: Initializing by example: Given a target image,our algorithm finds a correlation between the current im-age model and the target image’s model. Transforming themodel creates a painting inspired by the target image. Thisintegrates smoothly in the creative process.

first build a GMM for both images with the same number ofGaussians, k. Next, we calculate a k by k affinity matrix, con-taining the distance between each pair of Gaussians in thetwo models, calculated similar to equation 3, but restrictedto each pair of Gaussians. The distance is then proportionalto the probability of each Gaussian in the source image to beassigned to each Gaussian in the target image. Starting withthe highest probability, each Gaussian in the source model ismatched with a Gaussian in the target model greedily. Oncethere is a complete match, it is easy to calculate the transfor-mation from each source Gaussian to its target, and createa new base GMM for the image. Using this mapping, a setof variations are presented to the user for further navigation.Some results of using this tool can be seen in figure 9.

There are cases when navigation creates variations thatare unfit or wrong. For instance, when wrongly matching asource image, or when artifacts are evident on the images.Using our navigation tool the user can easily reject unwantedvariations navigating away from them and refining his selec-tions toward better ones.



6. Results

When first loading an image we construct the Gaussian Mix-ture Model of its color distribution. During the constructionwe utilize a natural boundary map [MFM04] to add spatialconsiderations, and preserve edges. Generating a new imagefor the gallery consists of creating a vector within the paint-ing search space, transforming the GMM, and synthesizingthe new image. Performance statistics on all steps of the al-gorithm can be found in Table 1. A wide variety of resultscan be seen in Figure 12 and the supplementary material.

User Study To validate our results and test the effective-ness of our approach, we have conducted an informal userstudy. Seventeen people participated in the study, most ofthem experienced computer users, seven of them profes-sional graphic designers. Although the target audience forthe tool is varied, the selection of subjects enabled a substan-tive comparison against a well established image processingapplication, such as Adobe Photoshop.

The test subjects received a ten minute introduction to ourapplication, following which, they were given four tasks.They were asked to perform each task both in Photoshop,and in our application (see Figure 10 and supplemental ma-terial). Once completing the tasks, each subject filled out ananonymous questionnaire. The analyzed results of the studyare presented below.

To measure the effectiveness of our approach, we com-pared the performance of users in the four tasks ver-sus their performance in Photoshop (Questions 9,13,17,22in supplemental material). In order to analyze the resultswe employed a paired t-test. Each test requires pairs ofobservations,(Ai,Bi), which are independent across i. In ourcase each pair consists of a single user’s answer for a ques-tion regarding our application (Ai) and Photoshop (Bi). Apaired t-test was defined check if µa is significantly largerthan µb using the following hypotheses

H0 : µa −µb ≤ 0H1 : µa −µb > 0

H0 is the hypothesis that completing the tasks in Photo-

Image Size GMM SynthesizeDress (Fig. 8) 300x500 0.766 0.23Tulips (Fig. 6) 500x400 1.7 0.26Blue (Fig. 5) 700x700 2.38 1

Nature (Fig. 12) 1000x760 2.57 1.45

Table 1: Performance statistics for our application. GMM isthe time (in seconds) required to model the image using EM.Synthesize is the time (in seconds) it takes to synthesize onecolor manipulated image and insert it into the gallery. Alltests were done on a 1.8Ghz Pentium 4 machine with 2GBRAM.

Figure 10: In the user study, each user was asked to performfour tasks in the painting images application, and again inAdobe Photoshop: (a) Change the color of the leaves to red,(b) Change the painting of the image to match the colors ofthe website, (c) Starting from a modified version of an image,return it to its original colors, and (d) Select an image ofyour choice and freely create an alternative creative vision.

shop is more effective than in the navigation demo. A pairedt-test on our observations yielded a T value of 3.3788 and ap-value of 0.0037, signifying that H0 is rejected with statis-tical significance. Hence, the tasks were accomplished moreeffectively and quickly using our navigation application.

An important aspect of image processing and manipula-tion of colors is creativity. In order to test if our demo sup-ports inspiration and creativity, we designed the tasks suchthat each could be “solved” in various ways. Moreover, suc-cess in each test was measured by the users themselves. Weasked the users if they found the demo intuitive and inspir-ing (Questions 8,12,16,21 in supplemental material). For allthese tests, a paired t-test resulted in a p-value smaller than0.05, meaning that users found the navigation applicationmore intuitive than Photoshop with statistical significance.To further demonstrate the effectiveness of our approach, weenclose a demonstration of our application in the accompa-nying material.

Limitations Our appearance search space represents awide variety of manipulations. However, not all color manip-ulations can be expressed by our model. For example, apply-ing different manipulations to different regions in the imageas seen in Figure 11 (right). A simple way to overcome thisis to allow to model an image by parts or use masks on theresulting image.

Changing colors in an image may still cause artifacts. Ar-tifacts usually occur when Gaussians, whose mean valuesare similar, are translated in opposite directions (Figure 11,left). Using the L∗A∗B∗ color appearance model, designedto approximate perceptual distance, can significantly reducethe number of such artifacts, and sometimes using moreGaussians in the mixture can also alleviate the problem.



7. Conclusion

In this paper we presented a method to modify the appear-ance of an image by navigation. Modeling an image’s colordistribution via a GMM, allowed us to define a space of richcolor variants on the image. Each point in this space rep-resents a specific variant, and we provide tools and an in-terface for a user to navigate within it. Working directly onthe image, selecting favorites or brushing on colors, remainssimple and abstract, while giving the users a high degree ofcontrol. It also promotes inspiration by providing access to avast space of options with simple exploration.

In the future we would like to improve our underlyingcolor model, for instance use a newer color appearance mod-els such as CIECAM02 in order to better reflect perceptualcolor differences on images. Moreover, we believe that mod-eling the color distribution dynamically, as the user navi-gates, could enhance the range of possible color manipula-tions, and model more accurately the user’s desires. Lastly,providing a means to work on sub-parts of the image cangreatly enhance the local control of color changes.

References

[Ado07] ADOBE SYSTEMS: After-effects cs3. Software, July2007.

[AP08] AN X., PELLACINI F.: Appprop: all-pairs appearance-space edit propagation. ACM Trans. Graph. 27, 3 (2008), 1–9.

[BPD06] BAE S., PARIS S., DURAND F.: Two-scale tone man-agement for photographic look. In SIGGRAPH ’06: ACM SIG-GRAPH 2006 Papers (New York, NY, USA, 2006), ACM Press,pp. 637–645.

[CCC00] COX T. F., COX M. A. A., COX T. F.: Multidimen-sional Scaling, Second Edition. Chapman & Hall/CRC, Septem-ber 2000.

[COSG∗06] COHEN-OR D., SORKINE O., GAL R., LEYVAND

T., XU Y.-Q.: Color harmonization. ACM Trans. Graph. 25, 3(2006), 624–630.

[GW01] GONZALEZ R. C., WOODS R. E.: Digital Image Pro-cessing. Addison-Wesley Longman Publishing Co., Inc., Boston,MA, USA, 2001.

[ICOL05] IRONY R., COHEN-OR D., LISCHINSKI D.: Coloriza-tion by example. In Proceedings of Eurographics Symposiumon Rendering (EGSR’05) (Konstanz, Germany, June 29 - July 12005), pp. 201–210.

[JKM01] JANKUN-KELLY T., MA K.-L.: Visualization explo-ration and encapsulation via a spreadsheet-like interface. IEEETransactions on Visualization and Computer Graphics 07, 3(2001), 275–287.

[LFUS06] LISCHINSKI D., FARBMAN Z., UYTTENDAELE M.,SZELISKI R.: Interactive local adjustment of tonal values. InSIGGRAPH ’06: ACM SIGGRAPH 2006 Papers (New York, NY,USA, 2006), ACM Press, pp. 646–653.

[LLW04] LEVIN A., LISCHINSKI D., WEISS Y.: Colorizationusing optimization. ACM Trans. Graph. 23, 3 (2004), 689–694.

[LWCO∗07] LUAN Q., WEN F., COHEN-OR D., LIANG L., XU

Y.-Q., SHUM H.-Y.: Natural image colorization. In RenderingTechniques 2007 (Proceedings Eurographics Symposium on Ren-dering) (June 2007), Kautz J., Pattanaik S., (Eds.), Eurographics.

Figure 11: Limitations: The left image exhibits color bleed-ing. These artifacts occur when Gaussians modeling pixelssimilar in colors and close spatially, receive transformationsin opposite directions. The right image demonstrates a typeof manipulation which cannot be expressed by our model. Inthis example a single Gaussian was split in three, each partshifted towards a different color.

[MAB∗97] MARKS J., ANDALMAN B., BEARDSLEY P. A.,FREEMAN W., GIBSON S., HODGINS J., KANG T., MIRTICH

B., PFISTER H., RUML W., RYALL K., SEIMS J., SHIEBER

S.: Design galleries: a general approach to setting parametersfor computer graphics and animation. Computer Graphics 31,Annual Conference Series (1997), 389–400.

[MFM04] MARTIN D. R., FOWLKES C. C., MALIK J.: Learningto detect natural image boundaries using local brightness, color,and texture cues. IEEE Transactions on Pattern Analysis andMachine Intelligence 26, 5 (2004), 530–549.

[OW04] OMER I., WERMAN M.: Color lines: Image specificcolor representation. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR 04) (June2004), vol. II, IEEE, pp. 946–953.

[PKD05] PITIÉ F., KOKARAM A., DAHYOT R.: N-DimensionalProbability Density Function Transfer and its Application toColour Transfer. In International Conference on Computer Vi-sion (ICCV’05) (Beijing, October 2005).

[RAGS01] REINHARD E., ASHIKHMIN M., GOOCH B.,SHIRLEY P.: Color transfer between images. IEEE Comput.Graph. Appl. 21, 5 (2001), 34–41.

[RWPD05] REINHARD E., WARD G., PATTANAIK S., DEBEVEC

P.: High Dynamic Range Imaging, 1 ed. The Morgan KaufmannSeries in Computer Graphics. Morgan Kaufmann Publishers, Au-gust 2005.

[SCLG05] SFIKAS G., CONSTANTINOPOULOS C., LIKAS A.,GALATSANOS N. P.: An analytic distance metric for gaussianmixture models with application in image retrieval. In Artifi-cial Neural Networks: Formal Models and Their Applications -ICANN 2005 (2005), vol. 3697/2005, Springer Berlin / Heidel-berg, pp. 835–840.

[TJT05] TAI Y.-W., JIA J., TANG C.-K.: Local color trans-fer via probabilistic segmentation by expectation-maximization.In CVPR ’05: Proceedings of the 2005 IEEE Computer Soci-ety Conference on Computer Vision and Pattern Recognition(CVPR’05) - Volume 1 (Washington, DC, USA, 2005), IEEEComputer Society, pp. 747–754.

[VNV06] VERBEEK J. J., NUNNINK J. R., VLASSIS N.: Accel-erated em-based clustering of large data sets. Data Min. Knowl.Discov. 13, 3 (2006), 291–307.

[ZK04] ZABIH R., KOLMOGOROV V.: Spatially coherent clus-tering using graph cuts. cvpr 02 (2004), 437–444.



Figure 12: A gallery of explored image variations, each source image is displayed with several creative and easy to achievevariations.


Image Appearance Exploration by Model-Based Navigation › ~dcor › articles › 2009 › Image-Appearance.pdf · L. Shapira & A. Shamir & D. Cohen-Or / Image Appearance Exploration

Documents