Page 1
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Hybrid Texture SynthesisHybrid Texture Synthesis
Andrew NealenAndrew NealenMarc AlexaMarc Alexa
Discrete Geometric Modeling Group (DGM)Discrete Geometric Modeling Group (DGM)Technische Universität DarmstadtTechnische Universität Darmstadt
Page 2
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
2D Texture Synthesis
nxm Input Texture
NxM Output Texture
► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.
Page 3
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Existing (and Impressive) Technology• Pixel-Based:
─ Non-parametric Sampling [Efros and Leung 1999]─ Tree-structured Vector Quantization [Wei and Levoy 2000]─ Synthesizing Natural Textures [Ashikhmin 2001]─ Image Analogies [Hertzmann et al. 2001]
• Patch-Based:─ Patch-Based Sampling [Guo et al. 2001]─ Image Quilting [Efros and Freeman 2001]─ Hierarchical Pattern Mapping [Soler et al. 2002]
Page 4
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
Page 5
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
Wei/Levoy Algorithm:Can occur when using structured Textures with rich histograms.This is due to the L2 norm.
Page 6
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
Patch-Based Sampling Algorithm:Can occur when texture features are not well-aligned across patch boundaries
Page 7
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
► Is it possible to reduce all deficiencies for an even larger class of textures ?
Page 8
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Step n in Algorithm
Page 9
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Build L-shaped Neighborhood
Page 10
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Find Matching Neighborhood(s) in Input
Page 11
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Select Best Neighborhood from all Candidates
Page 12
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Copy Selected Pixel to Output
Page 13
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Repeat for all Pixels in this Row...
Page 14
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
...and all other Output Rows
Page 15
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
► Main difference between Efros/Leung (EL), Wei/Levoy (WL) and Ashikhmin (AS): Per-Pixel Search Strategy.
► Ashikhmin’s insight: “WL tends to produce textures with smoothed out edges in some cases.”
► Ashikhmin’s solution: consider only coherent pixels in the search, which encourages region growing.
► Problem: inconsistencies can still occur for highly-structured textures.
Page 16
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Step n in Algorithm
Page 17
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Patch to be Synthesized in this Step
Page 18
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Build Neighborhood (Image Mask)
Page 19
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Find Matching Neighborhood(s) in Input
Page 20
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Select Best Neighborhood from all Candidates
Page 21
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Copy Selected Patch to Output
Page 22
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Apply some Overlap Repair Strategy ...
Page 23
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
... And we are done with this patch
Page 24
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Repeat for all Patches in Output
Page 25
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis
► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.
► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).
► PBS feathers the overlap region (alpha-blending).
► Pro: patch-based algorithms are good at maintaining global structure.
► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).
Page 26
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis
► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.
► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).
► PBS feathers the overlap region (alpha-blending).
► Pro: patch-based algorithms are good at maintaining global structure.
► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).
Page 27
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis
► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.
► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).
► PBS feathers the overlap region (alpha-blending).
► Pro: patch-based algorithms are good at maintaining global structure.
► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).
Page 28
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Hybrid Texture Synthesis (HTS)
Algorithm
► Hybrid, two-fold approach to texture synthesis• Adaptive patch sampling [Soler et al. 2002]
─ Start with a uniform quadrilateral grid of patches (the output)─ Recursively split a patch if the best fit taken from the input
texture is not good enough (a user-defined tradeoff max)
• Overlap re-synthesis (novel overlap repair strategy)─ Mark invalid pixels in the overlap region
─ Validity map is defined by the error surface E(x)=(xsrc – xdst)2
and a user-defined tolerance parameter max
─ Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy
Page 29
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingPrinciple
► Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002]
► Analogy: Assembling a quadtree puzzle
p > max Split PatchSynthesize White Patch p < max
Page 30
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Page 31
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Page 32
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Page 33
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Page 34
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesSynthesizing a single patch
Synthesize Black Patch i
Page 35
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesExtracting the image mask
Synthesize Black Patch i
► Grow patch by overlap (toroidally)
Page 36
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesExtracting the image mask
Image Mask (Ii)Synthesize Black Patch i Binary Support (Ji)
► Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner
Page 37
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesComputing the error image
Image Mask (Ii)Input Texture (T) Binary Support (Ji)
► Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T
Page 38
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesComputing the error image
Error Image (Ei)Input Texture (T)
x
xxxxxc
cicicii
i TIwJκ
E 20,,0 ))()(()(
1)(
Page 39
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesSelecting a Patch
Error Image (E)Input Texture (T) Selected Patch (P)
x
xxxxxc
cicicii
i TIwJκ
E 20,,0 ))()(()(
1)(
Page 40
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesSelecting a Patch
Error Image (E)Input Texture (T) Selected Patch (P)
► Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch.
Page 41
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Patch and Image Mask Selected Patch (P)
Page 42
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Image Mask (I) Selected Patch (P)
c
cicicii PIwJS 2,, ))()(()()( xxxx
Page 43
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Image Mask (I) Selected Patch (P)Error Surface (S)
c
cicicii PIwJS 2,, ))()(()()( xxxx
Page 44
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Patch and Image Mask P with Invalid Pixels (blue)Error Surface (S)
► Trade-off: user-defined max is the pixel error tolerance. Setting to 1 results in pure feathering.
Page 45
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisCompositing
Intermediate Result Patch with Invalid Pixels
P OVER Ri-1
Page 46
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisCompositing
Composited Result Patch with Invalid Pixels
► Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region
► Solution: alternative ordering (Pixel Traversal Map)
Page 47
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).
Page 48
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: step 1
Page 49
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: step 2 ...
Page 50
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: ... step n
Page 51
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.
Page 52
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Pixel Traversal Map (M)
► Synthesize red pixel from valid (or already re-synthesized) pixels
Intermediate Result
Page 53
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Image Mask (Ij)
Binary Support (Jj)
► Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.
Intermediate Result
Page 54
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result Pixel Traversal Map (M)
Page 55
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result Shifted
Pixel Traversal Map (M)
Page 56
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result with Traversal Map
Pixel Traversal Map (M)
Page 57
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 1
Pixel Traversal Map (M)
Page 58
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 2
Pixel Traversal Map (M)
Page 59
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 3
Pixel Traversal Map (M)
Page 60
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisResult
Pixel Traversal Map (M)
Page 61
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 62
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 63
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 64
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 65
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 66
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 67
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 68
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 69
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 70
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 71
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 72
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 73
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 74
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Page 75
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Page 76
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Page 77
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Page 78
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Page 79
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Page 80
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Page 81
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Page 82
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
Page 83
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
Page 84
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
Page 85
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Page 86
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Conclusions and Future Work
► Improve Computational Complexity• Pixel neighborhoods of patch and pixel stage are not
known a priori, so precomputation is not straightforward• We can solve this in the pixel stage by employing k-
coherence search [Tong et al. 2002] [Ashikhmin 2001]
► Improve Error Metric• Still using the L2 norm due to its simplicity
• Develop a metric which takes feature mismatch into account
• Texton approach looks promising [Zhang et al. 2003]
Page 87
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Questions ?
► Contact Information
Andrew [email protected]
Marc [email protected]
http://www.dgm.informatik.tu-darmstadt.de
Matlab code and extended Thesis:http://www.nealen.com