Pierre Bénard Ph.D. defense, 2011/07/07

Pierre Bénard Supervised by François Sillion & Joëlle Thollot

07/07/2011

• Expressive rendering

An alternative to photorealism

Tools for visual communication

• Interactive techniques

Direct user feedback

Dynamic visualization, video games…

[Herz98]

[GTDS04]

[CAS+97]

2

Okami Okami

Prince of Persia Prince of Persia

Team Forteress 2 Team Forteress 2

Jet Set Radio

3

Texture painting

McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course

4

McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course

Improved Sobel filter

5

McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course

Final compositing (textures + edges + fog)

6

McGuire et al., Stylized Rendering in Games, SIGGRAPH 2010 Course

Original Stylized

7

Temporal incoherencies

8

© Patrick Martin

Spatial Spatial

Attributes Attributes

Primitives Primitives Marks Marks 3D scene 2D image

perspective projection

2D lines and regions

color, shading, curvature…

pastel strokes

[Willats 97, Durand 02]

9

• Marks medium and pattern

Marks Marks

10

vs. Temporal continuity

« Il p

leut b

ergère », Jérém

y Dep

uyd

t (200

5)

11

Random changes Popping / Flickering

Static marks Shower-door effect

12

vs. Coherent Motion

Texture mapping 3D appearance

13

vs. Flatness

Motion coherence

Temporal continuity

Flatness

14

Conflicting goals compromise

Motion coherence

Temporal continuity

Flatness

Texture mapping

Static marks Random changes

15

Ideal solution

• REGIONS

Dynamic Solid Textures P. Bénard, A. Bousseau, J. Thollot, I3D 2009

NPR Gabor Noise

P. Bénard, A. Lagae, P. Vangorp, S. Lefebvre, G. Drettakis, J. Thollot, EGSR 2010

• LINES

Self-Similar Line Artmap P. Bénard, F. Cole, A. Golovinskiy, A. Finkelstein, NPAR 2010

Active Strokes work in progress

16

Temporally coherent bricks for stylized animations

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Perceptual Evaluation

Mapping Policies

Self Similar Line Artmap

Snakes-based Tracking and Parameterization

17

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Perceptual Evaluation

Mapping Policies

Self Similar Line Artmap

Active Strokes

19

• Distribution of marks

• Textures

[VBTS07] [VBTS07]

[CTP+03] [CTP+03]

20

• Distribution of marks

Few-marks

Many-marks

• Textures

[VBTS07] [VBTS07]

[CTP+03] [CTP+03]

21

Few-marks [Mei96,Dan99,HE04,

VBTS07,LSF10]

Motion coherence

Temporal continuity

Flatness

[Meie

r96]

3D distribution of anchor points

2D stroke texture

Painterly rendering

22

[VBTS07]

Good compromise: 2D strokes vs. 3D motion

Clutter / holes

Popping / flickering

[Meier96] [Meier96]

or

[LSF10] [LSF10]

[HE04] [HE04]

[Daniels99]

23

Many-marks [KC05,BKTS06]

24

Motion coherence

Temporal continuity

Flatness

Bridge the gap between texture and marks

Strong 2D appearance

Local motion and/or popping

Specific patterns (canvas fiber, watercolor pigments)

[KC05] [KC05]

[BKTS06]

25

• Distribution of marks

• Textures

In image space: local or global

In objet space

[VBTS07] [VBTS07]

[CTP+03] [CTP+03]

26

Local image space [BNTS07]

[BNTS07]

27

Motion coherence

Temporal continuity

Flatness

Strong 2D appearance

Regeneration artifacts

Post-production

[BNTS07]

28

Global image space [CTP+03,CDH06,

BSM+07]

+ +

[CTP+03]

29

Motion coherence

Temporal continuity

Flatness

Strong 2D appearance

Infinite zoom mechanism

3D motion approximated sliding

[CDH+06] [CDH+06]

[BSM+06] [BSM+06]

30

Object space [KLK+00,PHWF01,FMS01]

[PHWF01]

“Tonal Artmaps” 31

Motion coherence

Temporal continuity

Flatness

Exact motion

MIP-mapping

Perspective distortion

Limited styles [FMS01]

[KLK+00]

32

33

Local image space

Motion coherence

Temporal continuity

Flatness

Global image space

Motion coherence

Temporal continuity

Flatness

Object space

Motion coherence

Temporal continuity

Flatness

Few-marks

Motion coherence

Temporal continuity

Flatness

Many-marks

Motion coherence

Temporal continuity

Flatness

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Perceptual Evaluation

Mapping Policies

Self Similar Line Artmap

Active Strokes

Motion coherence

Temporal continuity

Flatness

Dynamic Solid Textures

34

• Real-time texture-based approach

• Accurate 3D motion object space approach

• Infinite zoom Dynamic Canvas [CTP+03]

• Easy parameterization solid textures

35

P. Bénard, A. Bousseau, J. Thollot, Dynamic Solid Textures for Real-Time Coherent Stylization. I3D 2009

36

• Texture “Fractalization”

1 solid texture : 4 octaves

octave 2 octave 1 octave 3 octave 4 weighted sum

freq. x2 freq. x2 freq. x2

37

• Zoom cycle

octave 2 octave 1 octave 3 octave 4 weighted sum

38

39

• One solid texture + 3D coordinates

• Procedural textures [Per85, Ola05]

or synthesized from 2D exemplars [KFCO+07]

• Ogre3D rendering engine

Additional cost: 10% compared to Gouraud shading

40

Homogeneity

Dynamic Solid Textures Standard 3D textures

41

42

Shot in Ogre3D

• Trade-off: Medium fidelity vs. Temporal continuity

• Linear blending new frequencies contrast loss

43

Statistical Analysis

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid Dots Hatching Paper Noise Paint Pigments Regular patterns

Irregular patterns

Cross- hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

Grid Dots Hatching Paper Noise Paint Pigments Near-regular

patterns Irregular patterns

Cross- hatching

S1

S2

P. Bénard, J. Thollot, F. Sillion, Quality Assessment of Fractalized NPR Textures. APGV 2009

44

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Perceptual Evaluation

Mapping Policies

Self Similar Line Artmap

Active Strokes

Motion coherence

Temporal continuity

Flatness

NPR Gabor Noise

45

P. Bénard, A. Lagae, P. Vangorp, S. Lefebvre, G. Drettakis, J. Thollot, A Dynamic Noise Primitive for Coherent Stylization. EGSR 2010

46

• Sparse convolution [Lewis 84,89]

• Spot Noise [van Wijk 91]

• Gabor Noise [LLDD09]

Offers significant spectral control

Support anisotropy

Our solution: NPR Gabor Noise

47

• Definition

Sum of randomly positioned and weighted kernels

Gabor kernel

noise

random positions and weights

48

anisotropic isotropic 𝐺 𝐾, 𝑎 × cos(𝐹0, 𝜔0)

• Flatness

Noise parameters in image space

Evaluation in image space

49

• Flatness

• Coherent motion

Point distribution on the surface of the 3D model

50

• Flatness

• Coherent motion

51

• Flatness

• Coherent motion

• Temporal continuity

Smooth LOD mechanism

52

• Sample 3D triangles (GPU)

2D Poisson distribution with constant screen space density

Far • small screen area • less points

Close • large screen area • more points

53

• Generate 2D point sprites

Point distribution Texture sprites

54

• Blending scheme using statistical properties

Reduce popping

Preserve noise appearance

Far Close

visibility = 0 visibility = 1

55

56

57

58

59

60

• Interactive scheme: remaining popping

“Procedural” approach

Slower, but should avoid popping

Useful for high quality offline rendering

• Gabor kernel

Temporally coherent spot noise

61

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Perceptual Evaluation

Mapping Policies

Self Similar Line Artmap

Active Strokes

62

• Evaluate success of various solutions according to:

Motion

coherence Temporal continuity

Flatness

P. Bénard, A. Lagae, P. Vangorp, S. Lefebvre, G. Drettakis, J. Thollot, A Dynamic Noise Primitive for Coherent Stylization. EGSR 2010

63

• Methodology

15 naïve subjects, ~ 20-30 minutes

• Ranking tasks “Rank the images/videos according to … ”

64

Extreme cases

Local image space Global image space Object space

Many-marks

Adv D2D DST

nprGN SD TM

65

Adv D2D DST

nprGN SD TM

66

• Image space methods more flat for simple scene

• Many 3D cues flatness hard to perceive in complex scene

Flatness

Coherent motion

Temporal continuity

Pleasantness

Shower Door

Texture Mapping

Object space

Local image space

Global image space

Many-marks /

67

• Object space methods more coherent

Flatness Coherent motion

Temporal continuity

Pleasantness

Shower Door

Texture Mapping

Object space

Local image space /

Global image space /

Many-marks / /

68

• High variance

• Advection and NPR Gabor Noise produce more changes: organic motion

Flatness Coherent motion

Temporal continuity

Pleasantness

Shower Door

Texture Mapping

Object space

Local image space /

Global image space / /

Many-marks / / /

69

• Object space approaches more pleasant

• Strong correlation with “motion coherence”

most important criteria to preserve

Flatness Coherent motion

Temporal continuity

Pleasantness

Shower Door

Texture Mapping

Object space

Local image space / /

Global image space / /

Many-marks / / / /

70

• First formal evaluation

• Intrinsic limitations

Hatching

3D scenes

Naïve users

71

• Two new solutions

Object space: Dynamic Solid Texture

Many-marks: NPR Gabor Noise

72

Motion coherence

Temporal continuity

Flatness

NPR Gabor Noise

Motion coherence

Temporal continuity

Flatness

Dynamic Solid Textures

• Simulation of brush strokes

Parameterization

0

l

Brush path

Line texture

74

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Evaluation

Mapping Policies

Self Similar Line Artmap

Active Strokes

75

• Stretching

• Tiling

76

77

• Stretching Temporally coherent

Stretch or compress the texture

• Tiling

Motion coherence

Temporal continuity

Flatness

Stretching

78

79

• Stretching Temporally coherent

Stretch or compress the texture

• Tiling Preserve the texture characteristics

Incoherent sliding

• Fading / Artmap

Motion coherence

Temporal continuity

Flatness

Tiling

80

81

• Stretching Temporally coherent

Stretch or compress the texture

• Tiling Preserve the texture characteristics

Incoherent sliding

• Fading / Artmaps No sliding

Blending artifacts Motion

coherence Temporal continuity

Flatness

Fading

82

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Evaluation

Mapping Policies

Self Similar Line Artmap

Active Strokes Motion coherence

Temporal continuity

Flatness

SLAM

83

P. Bénard, F. Cole, A. Golovinskiy, A. Finkelstein Self-Similar Texture for Coherent Line Stylization. NPAR 2010

84

Lmax / 2

Lmax

85

1) Density 2) Smooth variation

86

3) Self-Similarity

87

3) Self-Similarity

88

89

• Parametric texture synthesis [PS00]

• Property: (empirical observation, no formal proof)

small change in noise seed small change in output texture [PS00] Portilla and Simoncelli,

A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients, 2000

90

Exemplar

Synthesized Texture

White Noise Seed

[PS00]

=

Wsource Wf

Seed Pyramid

Wf

[WfWf]

0.5

1

α Wf + (1-α) [WfWf]

Lmax

Lmax

Lmax / 2

91

92

Synthesized textures Examples

93

94

95

96

• Input

Silhouettes

Suggestive contours

Apparent ridges

• Arc-length artifacts

Coherent parameterization

97

Previous Work

Dynamic Solid Textures

NPR Gabor Noise

Evaluation

Mapping Policies

Self Similar Line Artmap

Active Strokes

Work in progress in collaboration with C. Lu, F. Cole and A. Finkelstein 98

• Flatness

Remain linear in screen-space

• Motion coherence

Evolve according to the motion of the objet

• Temporal continuity

Adapt to the topological events

99

Coherent Stylized Silhouettes, Kalnins et al., SIGGRAPH 2003

Optimization between 2D and 3D

Multiple brush strokes per line

Dependent on the input connectivity

100

Kalnins et al., Coherent Stylized Silhouettes, SIGGRAPH 2003

101

Snakes

Snakes

3D scene

Lines extraction

Lines extraction

Feature samples

Line drawing

Brush Paths

Brush Paths

Parameterization Parameterization

Stylization Stylization Cleaning Vectorization

Cleaning Vectorization

Tracking Tracking

102

Feature samples

Line drawing

• Position • Local tangent • Velocity

Snakes

Snakes

3D scene

Lines extraction

Lines extraction

Feature samples

Line drawing

Brush Paths

Brush Paths

Parameterization Parameterization

Stylization Stylization Cleaning Vectorization

Cleaning Vectorization

Tracking Tracking

103

3D scene

Lines extraction

Lines extraction

Feature samples

Line drawing

Snakes

Snakes

Brush Paths

Brush Paths

Parameterization Parameterization

Stylization Stylization Cleaning Vectorization

Cleaning Vectorization

Tracking Tracking

104

• Polylines • In image space • Persistent

• Advection

• Relaxation

105 Frame f Frame f+1

• Advection

• Relaxation

106

107

108

3D scene

Lines extraction

Lines extraction

Feature samples

Line drawing

Snakes

Snakes

Brush Paths

Brush Paths

Parameterization Parameterization

Stylization Stylization Cleaning Vectorization

Cleaning Vectorization

Tracking Tracking

109

• Linear screen space parameterization

110

• Segments fitting

111

• Arcs fitting

112

113

114

115

116

Robustness toward noise / outliers

Temporal smoothing

117

Input samples Active strokes

• 4 new bricks for time coherent stylization

Dynamic Solid Textures

NPR Gabor Noise

Self-Similar Line Artmap

Active Strokes

• Formal evaluation

Formulation into 3 goals

2 perceptual studies

1 objective quality metric

119

• 2D Animation

Ongoing work with Antoine Boellinger (Master 1)

120

• 2D Animation

Ongoing work with Antoine Boellinger (Master 1)

• Evaluation & Objective metrics

Optical flow analysis

• Stylization by optimization

Versatile system to explore the space of trade-offs

121

122

123

124

125

• Simple

• Strong motion coherence and temporal continuity

• More suitable for unstructured patterns

• Real-time animation

126

• Input: pairs of 2D texture

• Definition: texture distortion = visual dissimilarity between original and transformed textures

• Goal: define a quantitative metric of this distortion

original transformed

P. Bénard, J. Thollot, F. Sillion, Quality Assessment of Fractalized NPR Textures. APGV 2009

127

• Procedure:

User study ranking of texture pairs according to their distortion

Statistical analysis scale of perceived quality

Correlation investigation objective metric

• Restrictions:

No texture mapping

Static images

128

• Ordinal scale

S1

S2

Statistical Analysis

129

• Thurstone’s law of comparative judgment [Tor58]

Statistical Analysis

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid Dots Hatching Paper Noise Paint Pigments Regular

patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

Grid Dots Hatching Paper Noise Paint Pigments Near-regular

patterns

Irregular

patterns

Cross-

hatching

S1

S2

130

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

Grid Dots Hatching Paper Noise Paint Pigments Near-regular

patterns Irregular patterns

Cross- hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid Dots Hatching Paper Noise Paint Pigments Regular patterns

Irregular patterns

Cross- hatching

S1

S2

131

Statistical Analysis

S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid Dots Hatching Paper Noise Paint Pigments Regular patterns

Irregular patterns

Cross- hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

Grid Dots Hatching Paper Noise Paint Pigments Near-regular

patterns Irregular patterns

Cross- hatching

132

Overall contrast of patterns Feature shapes

Statistical Analysis

S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid Dots Hatching Paper Noise Paint Pigments Regular patterns

Irregular patterns

Cross- hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

Grid Dots Hatching Paper Noise Paint Pigments Near-regular

patterns

Irregular

patterns

Cross-

hatching

Paper Noise Paint Pigments

133

Overall contrast of patterns

Feature shapes

Statistical Analysis

S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid Dots Hatching Paper Noise Paint Pigments Regular

patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

Grid Dots Hatching Paper Noise Paint Pigments Near-regular

patterns

Irregular

patterns

Cross-

hatching

Irregular patterns Cross-hatching Dots Hatching

134

• Gray level co-occurrence matrix (GLCM) [HSD73]

Texture descriptor [TJ93]

Local image property

Match certain levels of human perception [JGSF76]

Linked to density and pattern coherence criteria

Parameters selection

• Average Co-occurrence Error [CRT01]

High correlation with the perceptual interval scale (Person’s correlation: 0.953 for S1 and 0.836 for S2 )

Correlation with Objective Metrics

135

ACE relevant estimator of the distortion

S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid Dots Hatching Paper Noise Paint Pigments Regular patterns

Irregular patterns

Cross- hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

Grid Dots Hatching Paper Noise Paint Pigments Near-regular

patterns Irregular patterns

Cross- hatching

AC

E

1.2

1.0

0.8

0.6

0.4

AC

E

1.4

1.2

1.0

0.8

0.6

r² = 0.6992

r² = 0.9075

136

• More suitable for low-contrast media

• Simple

• Strong motion coherence and temporal continuity

• Real-time animation

137

l=0.5

l=0

l=1

Lmax

138

Input: white noise seed

…

…

…

…

Parametric Texture Synthesis [PS00]

Seed Pyramid

Input: texture example

Output: SLAM

139

Self-Similar Texture for Coherent Line Stylization, Bénard et al. 2010

2D Infinite zoom: Self-Similar Line Artmap (SLAM)

Propagation using a 2D buffer aliasing ambiguities

One brush stroke per line

140

141

• Linear screen-space parameterization using Self-similar Line Artmaps

• Multiple brush strokes per feature line

• New propagation mechanism

142

• Feature samples – lines extracted in 2D or 3D

3D position

Local tangent

2D velocity

143

Snakes

Snakes

Tracking Advection Advection

3D scene

Lines extraction

Lines extraction

Feature samples

Relaxation Relaxation

Vectorization

Coverage Coverage Connectivity Connectivity

Line drawing

Brush Paths

Brush Paths

Geometry Geometry

Parameterization Parameterization

Stylization Stylization

144

• 2D polylines active contours (Kass et al. 1988)

• Goals

Coherence continuous evolution across frames

Accuracy faithfully represent shape

Coverage Level of Detail

Simplicity simple topology

Length stylization freedom

Trac

kin

g V

ect

ori

zati

on

145

• Directional coverage radius

• Local vectorization operators

Trim

Extend

Merge

Split

Coverage Coverage

Connectivity Connectivity

146

Coverage Coverage

Connectivity Connectivity

147

• Directional coverage radius

• Local vectorization operators

Trim

Extend

Merge

Split

• Applied sequentially in a greedy fashion

Coverage Coverage

Connectivity Connectivity

148

• Reprojection similar to Kalnins et al. 2003

• Computed for feature lines only connection between contour and feature sample needed

Advection Advection

149

• Minimize the energy:

• Re-sampling [DM00]

Relaxation Relaxation

External – Attraction by the features – Mass-spring forces prevents shrinking

Internal – Continuity – Smoothness

150

Snakes

Snakes

Tracking Advection Advection

3D scene

Lines extraction

Lines extraction

Feature samples

Relaxation Relaxation

Vectorization

Coverage Coverage Connectivity Connectivity

Line drawing

Brush Paths

Brush Paths

Geometry Geometry

Parameterization Parameterization

Stylization Stylization

151

• Consistently increasing subset of a snake

152

• Uniform in screen-space linear arc-length parameterization :

avec = slope = phase = arc-length

• Evolve according to the motion and topology of the contours parameterization at each vertex

Parameterization Parameterization

153

• Propagation at each vertex

Parameterization stored between two frames

Parameterization Parameterization

154

• Propagation at each vertex

• Linearization (in the least-square sense) p

aram

eter

izat

ion

arc-length

Parameterization Parameterization

155

• Propagation at each vertex

• Vectorization events

Split

Extend

Trim

Merge: mechanism to avoid parameterization discontinuities

• Linearization

propagated directly

Parameterization Parameterization

156

• Only if the slope and phase match

• Leveling mechanism pushes the 2 parameterizations to their mid-value:

with

par

amet

eriz

atio

n

arc-length

Parameterization Parameterization

157

• Feature lines dependency

Robustness toward noise / outliers

Temporal smoothing

158