Fast Texture Synthesis Tree-structure Vector Quantization Li-Yi WeiMarc Levoy Stanford University.

Fast Texture Synthesis Fast Texture Synthesis Tree-structure Vector Tree-structure Vector

QuantizationQuantization

Li-Yi WeiLi-Yi Wei Marc LevoyMarc LevoyStanford UniversityStanford University

OutlineOutline

► IntroductionIntroduction►AlgorithmAlgorithm►TSVQ AccelerationTSVQ Acceleration►ApplicationsApplications

Introdction Introdction How texture differ from images?How texture differ from images?

Gaussian pyramidGaussian pyramid

IntrodcutionIntrodcution► In this paper, we present a very simple algorithIn this paper, we present a very simple algorith

m that can efficiently synthesize a wide variety om that can efficiently synthesize a wide variety of textures.f textures.

► The inputs consist of an example texture patch aThe inputs consist of an example texture patch and a random noise image with size specified by nd a random noise image with size specified by user.user.

► The algorithm modifies this random noise to maThe algorithm modifies this random noise to make it look like the given example.ke it look like the given example.

► New textures can be generated with little compuNew textures can be generated with little computation time, and their tileability is guaranteed.tation time, and their tileability is guaranteed.

AlgorithmAlgorithm

►Single resolution synthesisSingle resolution synthesis►NeighborhoodNeighborhood►Multiresolution synthesisMultiresolution synthesis►Edge handingEdge handing► initializationinitialization

Single resolution synthesisSingle resolution synthesis

► The algorithm starts with an input texture sample The algorithm starts with an input texture sample IaIa and and a white random noise a white random noise IsIs..

► We force the random noise We force the random noise IsIs to look like to look like IaIa by transform by transforming ing IsIs pixel by pixel in a raster scan ordering. pixel by pixel in a raster scan ordering.

► To determine the pixel value p at Is, its spatial neighborTo determine the pixel value p at Is, its spatial neighborhood N(p) is compare against all possible neighborhoodhood N(p) is compare against all possible neighborhoods N(pi) from s N(pi) from IaIa..

► The input pixel pi with the most similar N(pi) is assigned The input pixel pi with the most similar N(pi) is assigned to p.to p.

► We use a simple We use a simple L2L2 norm (sum of squared difference) to norm (sum of squared difference) to measure the similarity between the neighborhoods.measure the similarity between the neighborhoods.

Single resolution synthesisSingle resolution synthesis

NeighborhoodNeighborhood

►Because the set of local neighborhoods NBecause the set of local neighborhoods N(pi) is used as the primary model for textu(pi) is used as the primary model for textures, the quality of the synthesized results res, the quality of the synthesized results will depend on its size and shape.will depend on its size and shape.

►The shape of the neighborhood will directThe shape of the neighborhood will directly determine the quality of ly determine the quality of IsIs. .

NeighborhoodNeighborhood

NeighborhoodNeighborhood

Multiresolution synthesisMultiresolution synthesis►For textures containing large scale structuFor textures containing large scale structu

res we have to use large neighborhoods, ares we have to use large neighborhoods, and large neighborhoods demand more cond large neighborhoods demand more computation.mputation.

►This problem can be solved by using a mulThis problem can be solved by using a multiresloution image pyramid.tiresloution image pyramid.

Multiresolution synthesisMultiresolution synthesis► Two Gaussian pyramids, Ga and Gs, are first built from Two Gaussian pyramids, Ga and Gs, are first built from IaIa

and and IsIs, respectively., respectively.► The algorithm then transfroms Gs from lower to higher rThe algorithm then transfroms Gs from lower to higher r

esolutions, such that each higher resolution level is consesolutions, such that each higher resolution level is constructed from the already synthesized lower resolution letructed from the already synthesized lower resolution levels.vels.

► The only modification is that for the multiresolution casThe only modification is that for the multiresolution case, each neighborhood N(p) contains pixels in the current e, each neighborhood N(p) contains pixels in the current resolution as well as those in the lower resolutions.resolution as well as those in the lower resolutions.

► The similarity between two multiresolution neighborhoThe similarity between two multiresolution neighborhoods is measured by computing the sum of the squared dods is measured by computing the sum of the squared distance of all pixels within them.istance of all pixels within them.

Multiresolution synthesisMultiresolution synthesis

Edge handingEdge handing

►For the synthesis pyramid the edge is treaFor the synthesis pyramid the edge is treated toroidally.ted toroidally.

►For the input pyramid Ga, toroidal neighbFor the input pyramid Ga, toroidal neighborhoods typically contain discontinuities orhoods typically contain discontinuities unless unless IaIa is tileable. is tileable.

►We use only those N(pi) completely insideWe use only those N(pi) completely insided Ga, and discard those crossing the bound Ga, and discard those crossing the boundaries.daries.

InitializationInitialization

► We initialize the output image We initialize the output image IsIs as a white rand as a white random noise, and gradually modify this noise to looom noise, and gradually modify this noise to look like the input texture k like the input texture IaIa..

► This initialization step seeds the algorithm with This initialization step seeds the algorithm with sufficient entropy, and lets the rest of the synthsufficient entropy, and lets the rest of the synthesis process focus on the transformation of esis process focus on the transformation of IsIs to towards wards IaIa..

► To make this random noise a better initial guess, To make this random noise a better initial guess, we also equalize the pyramid histogram of Gs wiwe also equalize the pyramid histogram of Gs with respect to Ga.th respect to Ga.

TSVQ AccelerationTSVQ Acceleration

►TSVQ takes a set of training vectors as inpTSVQ takes a set of training vectors as input, and generates a binary-tree-structure ut, and generates a binary-tree-structure codebook.codebook.

►The tree generated by TSVQ can be used aThe tree generated by TSVQ can be used as a data structure for efficient nearest-pois a data structure for efficient nearest-point queries.nt queries.

►To use TSVQ in our synthesis algorithm, wTo use TSVQ in our synthesis algorithm, we simply collect the set of neighborhood pe simply collect the set of neighborhood pixels N(pi) for each input pixel in N(pi)ixels N(pi) for each input pixel in N(pi)

TSVQTSVQ

TSVQTSVQ

TSVQTSVQ

TSVQTSVQ

ApplicationApplication

► Constrained Texture SynthesisConstrained Texture Synthesis

► Temporal Texture synthesisTemporal Texture synthesis

Constrained Texture Constrained Texture synthesissynthesis

►Texture replacement by constrained Texture replacement by constrained synthesis must satisfy tow synthesis must satisfy tow requirements:requirements: The synthesized region must look like the The synthesized region must look like the

surrounding texturesurrounding texture The boundary between the new and old The boundary between the new and old

regions must be invisible.regions must be invisible.

Temporal texture synthesisTemporal texture synthesis

►The low cost of our accelerated The low cost of our accelerated algorithm enables us to consider algorithm enables us to consider synthesizing textures of dimension synthesizing textures of dimension greater than two.greater than two.

►An example of 3D texture is a temporal An example of 3D texture is a temporal texture.texture.

►Temporal textures are motions with Temporal textures are motions with indeterminate extent both in space and indeterminate extent both in space and time.time.

► Figure 14: Temporal texture synthesis results. (a) fire (b) smoke (c) ocean waves. In each pair of images, the spatial-temporal volume of the original motion sequence is shown on the left, and the corresponding synthesis result is shown on the right. A 3-level Gaussian pyramid, with neighborhood sizes 5x5x5,2 , 3x3x3,2 , 1x1x1,1 , are used for synthesis. The original motion sequences contain 32 frames, and the synthesis results contain 64 frames. The individual frame sizes are (a) 128x128 (b) 150x112 (c) 150x112. Accelerated by TSVQ, the training times are (a) 1875 (b) 2155 (c) 2131 seconds and the synthesis times per frame are (a) 19.78 (b) 18.78 (c) 20.08 seconds. To save memory, we use only a random 10 percent of the input neighborhood vectors to build the (full) codebooks.

► http://graphics.stanford.edu/~liyiwei/project/texture/demo/temporal_synthesis.html