Simulation Photoréaliste de l'Éclairage en Synthèse d'Images Nicolas Holzschuch Habilitation à Diriger des Recherches Université Joseph Fourier Grenoble.

Simulation Photoréaliste de l'Éclairage en Synthèse d'Images

Nicolas HolzschuchHabilitation à Diriger des

RecherchesUniversité Joseph Fourier —

Grenoble I

Overview

• Motivations• Previous work• Contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

Overview

• Motivations• Previous work• Contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

Photorealistic Image Synthesis

• Starting from the 3D model of a scene…

© C

. So

ler,

20

07.

Mo

del

by

L. B

ois

sie

ux.

Photorealistic Image Synthesis

…we get a picture of this virtual world

© C

. So

ler,

20

07.

Mo

del

by

L. B

ois

sie

ux.

Motivations

• Why are we doing this?– Many applications:

•Virtual prototyping•Cultural heritage•Video games•Special effects

– Good looking pictures– Multi-disciplinar activity (CS, physics, math)

Motivations

• Why are we doing this?– Many applications:

•Virtual prototyping•Cultural heritage•Video games•Special effects

– Good looking pictures– Multi-disciplinar activity (CS, physics, math)

Model from DaimlerChrysler / RealReflect IST project / picture by C. Soler

Motivations

• Why are we doing this?– Many applications:

•Virtual prototyping•Cultural heritage•Video games•Special effects

– Good looking pictures– Multi-disciplinar activity (CS, physics, math)

© P. Müller

Motivations

• Why are we doing this?– Many applications:

•Virtual prototyping•Cultural heritage•Video games•Special effects

– Good looking pictures– Multi-disciplinar activity (CS, physics, math)

© Valve, 2006

Motivations

• Why are we doing this?– Many applications:

•Virtual prototyping•Cultural heritage•Video games•Special effects

– Good looking pictures– Multi-disciplinar activity (CS, physics, math)

© Lucas Digital Ltd., 2002

Motivations

• Why are we doing this?– Many applications:

•Virtual prototyping•Cultural heritage•Video games•Special effects

– Good looking pictures– Multi-disciplinar activity (CS, physics, math)

• Why is it difficult?

Motivations• Why are we doing this?• Why is it difficult?

– Interactions between light and matter•Complex local effects (reflections)

– Self-dependent– Long distance interactions– Differences of scale/frequency– Time/quality compromise– Blurry effects are costlier– Too many computations?

Overview

• Motivations• Timeline• Selected contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

Timeline

Timeline: positions

Timeline: research themes

Timeline: projects and contracts

Overview

• Motivations• Timeline• Selected contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

Overview

• Motivations• Timeline• Selected contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

Wavelet Radiosity: credits

• 1993-1996: iMAGIS– Under the supervision of C. Puech, F. Sillion, G.Drettakis

• 1997-2000: ISA/LORIA– Working with J.-C. Paul, L. Alonso, F. Cuny, X. Cavin, C. Winkler, H. Barthélémy, S. Petitjean…

• 2000-2004: iMAGIS/GRAVIR– Working with L. Alonso (at ISA), C. Damez, F. Sillion…

Wavelet radiosity• Extension of Hierarchical Radiosity

– [Gortler et al., 1993]

• Uses wavelets for basis functions– Haar wavelets (pw. constant) equivalent to HR

• M2, M3 wavelets : linear, quadratic functions

• Higher order wavelets should work better– Better representation for smooth functions

• Experimental study: no, they don’t!– [Wilmott & Heckbert, 1997]

Wavelet radiosity• Everyone stopped working on WR

– …except us at ISA/LORIA• Point is: it should really work better– So why not?

• Experimental studies• Everyone had used a naïve approach:

– Just changing the basis function– But higher order wavelets are costlier

•n2 for each patch•n4 for each interaction

Wavelet radiosity

• Naïve implementation doesn’t work• Must change the approach• Adapt the algorithm:

– Meshing strategy– No link storage + shooting – Refinement oracle– Visibility tests

Why didn’t it work?

Why didn’t it work?

• Wavelets need quad/triangular patches

Why didn’t it work?

• Patches are then refined• The initial triangulation binds the algorithm

•And each patch is n2 times costlier

Extended mesh

• Keep the original mesh• Adapt the radiosity algorithm

[Holzschuch, Cuny, Alonso, EGWR 2000]

Extended mesh

• Separation between function and geometry

Link storage and consequences• Biggest issue is memory cost

– n4 for each interaction– Solution: don’t store them!– [Stamminger, Schirmacher, Slusallek & Seidel, EG 1998]

• But think further:– Gathering/shooting– Gathering works better with HR– …except if you don’t store the interactions

• We should use shooting!– …then it makes more sense not to store links!

Higher-Order Wavelets• Other changes (refinement oracle, visibility)

• This time it works:– Smaller memory costs– Faster computations

[Cuny, Holzschuch, Alonso, EG 2000]

Memory costs

Observed memory costs for the different wavelet bases(rendering time follows similar curve)

Consistent with [Willmott & Heckbert 1997]!

Higher order wavelets II

• Efficient global illumination algorithm– Uses higher-order wavelets– Robust, works on real-world scenes– Memory efficient

• So, is everything perfect?– Actually, no.– Shadow boundaries are a big issue

Shadow boundaries

• Discontinuity: the oracle keeps refining

• Each patch is n2 times more costly– We lose all the ground we gained elsewhere

Discontinuity meshing• Well-known method:

– Shadow boundary position is known– Cut the mesh according to these discontinuities

– Refine the resulting mesh

• Impossible with wavelet radiosity:– Wavelets only defined over quads/triangles

– …but we already have a workaround!

DM and Higher-Order Wavelets• [Holzschuch and Alonso, EGSR 2004]• Discontinuities are introduced only if necessary – Use regular subdivision as much as possible

• Modified refinement oracle:– If refinement could have been avoided using DM

– Then Subdivide patch using discontinuity

– Embed the new patches inside quads– Regular subdivision over the new quads

DM and Higher-Order Wavelets

• M2 wavelets, point light source, no DM

[Holzschuch and Alonso, EGSR 2004]

DM and Higher-Order Wavelets

• M2 wavelets, point light source, with DM

[Holzschuch and Alonso, EGSR 2004]

DM and Higher-Order Wavelets

• M3 wavelets, area light source, with DM

[Holzschuch and Alonso, EGSR 2004]

Wavelet Radiosity: summary• Compact representation of illumination• Applications:

– Cooperation with Nancy School of Architecture– We tested our algorithms on real-world scenes– Great for robustness

• Research:– Real scientific problems came up– We had to solve them, resulting in nice papers

• Transfer: – The software was transferred to a start-up– VSP-Technology, headed by F. Cuny

Wavelet radiosity: looking back• Shadow boundaries is an interesting issue

• Without DM:– 90 % of computation time for shadows

• With DM:– 90 % of implementation time for shadows

• Only important for (direct lighting+directly visible)

• Indirect lighting can use low quality version

• Waste of time?

Overview

• Motivations• Timeline• Contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

Frequency Analysis: credits

• Joint research work with MIT/CSAIL

• At MIT/CSAIL: F. Durand, E. Chan• At ARTIS/GRAVIR: C. Soler, F. Sillion

• Financed by an INRIA “Équipe Associée”

• [Durand, Holzschuch, Soler, Chan, Sillion, SIGGRAPH 2005]

Illumination effects

• Blurry reflections:

From [Ramamoorthi and Hanrahan 2001]

Illumination effects

• Shadow boundaries:

© U. Assarsson 2005.

Point light source Area light source

Illumination effects

• Indirect lighting is usually blurry:

Complete lighting© C. Soler 2005.

Illumination effects

• Indirect lighting is usually blurry:

Indirect lighting onlyDirect lighting only© C. Soler 2005.

Frequency aspects of light transport

• Blurriness = frequency content– Sharp variations: high frequency– Smooth variations: low frequency

• All effects are expressed as frequency content:– Diffuse shading: low frequency– Shadows: introduce high frequencies– Indirect lighting: tends to be low frequency

Problem statement

• How does light interaction in a scene explain the frequency content?

• Theoretical framework:– Understand the frequency spectrum of the radiance function

– From equations of light transport

Unified framework:

• Spatial frequency (e.g. shadows, textures)

• Angular frequency (e.g. blurry highlight)

Disclaimer: not Fourier optics

• We do not consider wave optics, interference, diffraction

• Only geometrical optics

Fro

m [

Hec

ht]

Disclaimer: not Fourier optics

• We do not consider wave optics, interference, diffraction

• Only geometrical optics

Fro

m [

Hec

ht]

Our approach

• Light sources are input signal • Interactions are filters/transforms– Transport – Visibility– BRDF– Etc.

Our approach

• Light sources are input signal • Interactions are filters/transforms– Transport – Visibility– BRDF– Etc.

Light source signal

Our approach

• Light sources are input signal • Interactions are filters/transforms– Transport – Visibility– BRDF– Etc.

Light source signal

Transport

Our approach

• Light sources are input signal • Interactions are filters/transforms– Transport – Visibility– BRDF– Etc.

Light source signal

Signal 1

Transport

Our approach

• Light sources are input signal • Interactions are filters/transforms– Transport – Visibility– BRDF– Etc.

Light source signal

Transport

Occlusion

Signal 2

Our approach

• Light sources are input signal • Interactions are filters/transforms– Transport – Visibility– BRDF– Etc.

Light source signal

Transport

Occlusion

Signal 3

Transport

Our approach

• Light sources are input signal • Interactions are filters/transforms– Transport – Visibility– BRDF– Etc.

Light source signal

Transport

Occlusion

Signal 4

Transport

BRDF

Local light field

• 4D light field, around a central ray

Central ray

Local light field

• 4D light field, around a central ray

• We study its spectrum during transport

Local light field

• 4D light field, around a central ray

• We study its spectrum during transport

Local light field

• 4D light field, around a central ray

• We study its spectrum during transport

Local light field

• We give explanations in 2D – Local light field is therefore 2D

• See SIGGRAPH paper for extension to 3D

Local light field parameterization• Space and angle

space

angle

Central ray

Local light field representation• Density plot:

Area light source

Space

Ang

le

Local light fieldFourier spectrum• We are interested in the Fourier spectrum of the local light field

• Also represented as a density plot

Local light field Fourier spectrum

Spatial frequency

Ang

ular

freq

uenc

ySpectrum of an area light source:

Basics of Fourier analysis

• Spectrum corresponds to blurriness:– Sharpest feature has size 1/Fmax

• Convolution theorem:– Multiplication convolution

• Classical spectra: – Box sinc– Dirac constant

Example scene

Blockers

Light source

Receiver

Transport in free space

Shear

Space Space

Ang

le

Ang

le

Transport in free space

Shear

Space Space

Ang

le

Ang

leShear

Spatial frequency

Ang

ular

freq

.

Spatial frequency

Ang

ular

freq

.

Occlusion: multiplication

• Occlusion is a multiplication in ray space– Convolution in Fourier space

• Creates new spatial frequency content– Related to the spectrum of the blockers

BRDF integration• Outgoing light:

– Integration of incoming light times BRDF

Outgoing lightIncoming light

BRDF

BRDF integration

• Ray-space: convolution– Outgoing light: convolution of incoming light and BRDF

– For rotationally-invariant BRDFs

• Fourier domain: multiplication – Outgoing spectrum: multiplication of incoming spectrum and BRDF spectrum

BRDF in Fourier: multiplication

=

• BRDF is bandwidth-limiting in angle

Example: diffuse BRDF

• Convolve by constant: – multiply by Dirac– Only spatial frequencies remain

=

Transforms: summaryRadiance/Fourier Effect

Transport Shear

Occlusion Multiplication/Convolution

Adds spatial frequencies

BRDF Convolution/Multiplication

Removes angular frequencies

Curvature Shear

Case study: diffuse soft shadows• Decreasing blockers size:

– First high-frequencies increase– Then only low frequency– Non-monotonic behavior

Diffuse soft shadows (2)

• Occlusion : convolution in Fourier – creates high frequencies– Blockers scaled down spectrum scaled up

Fourier space

v

(ang

le)

x (space)

Fourier space

v

(ang

le)

x (space)

blocker spectrum

Diffuse soft shadows (3)• Transport after occlusion:

– Spatial frequencies moved to angular dimension

• Diffuse reflector: – Angular frequencies are cancelled

Diffuse soft shadows (3)• Transport after occlusion:

– Spatial frequencies moved to angular dimension

• Diffuse reflector: – Angular frequencies are cancelled

Frequency Analysis: summary• Framework for frequency analysis of light transport

• Gives high frequencies locations

• Occlusion creates high frequencies• Transport + reflection removes HFs• High frequencies mostly if:

– Small transport (quasi-contact)– Specular reflection (or highly glossy)

– Point light source

Overview

• Motivations• Timeline• Contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

GPU-based algorithms: credits• Specular reflections: D. Roger

• Real-time soft shadows: L. Atty, M. Lapierre, C. Hansen, J.-M. Hasenfratz, F. Sillion

• Contact shadows: M. Malmer, F. Malmer, U. Assarsson

• Global Illumination: J. Kontkannen, E. Turquin, F. Sillion

GPU-based algorithms• GPUs are highly powerful machines

– Widely available– Used in the industry

•Video games, cinema

• But they also have their limitations– Memory accesses, per-pixel computations…

• They work better on local effects– Direct lighting, shadow boundaries…

• 2003: AS CNRS “Real-Time Rendering”

Looking back…• Radiosity:

– Lots of work for direct lighting+directly visible

• High frequency effects in illumination: – Small transport– Shadow boundaries– Specular reflections– …mostly local or “semi-local” effects

•Involve a small subset of the scene for each pixel

Think globally• Our hard points:

– Direct lighting+directly visible– Local or semi-local effects

• … all correspond to the forte of GPUs

• Main idea: – use GPUs for HF effects+directly visible

– Something else (CPU) for low frequency effects

GPU-based algorithms

• Using GPUs for local/semi-local effects:– Specular reflections– Shadows (soft and hard)– Contact shadows

• Global illumination in static scenes:– GPU for direct lighting, CPU for indirect

Specular reflections

• Vertex based reflections• For each vertex in the scene

– Compute its reflected position– Find light paths of extremal length– Using gradient descent

[Roger and Holzschuch, EG 2006]

Specular reflections

Our method Env. mapping Reference

[Roger and Holzschuch, EG 2006]

Real-time soft shadows

• Shadows: important issue for realism

• Real-time hard shadows: easy with GPU

• Soft shadows: still an open problem

• EG 2003: State-of-the-Art Report– Hasenfratz, Lapierre, Holzschuch, Sillion

Soft Shadow Maps• Discretize occluders using shadow map

• Compute soft shadow for each surfel– Easy because surfel axis-aligned with LS

• Add the contributions of each surfel

[Atty, Holzschuch, Lapierre, Hasenfratz, Hansen, Sillion, CGF 2006]

Contact shadows

• Shadows cast on nearby objects:

[Malmer, Malmer, Assarsson, Holzschuch, JGT 2007]

Contact shadows• Equivalent to ambient occlusion

– Important for realism– Low-cost effect, but with high impact

• Low frequency effect• Several techniques for storing AO field– Moving with the object– Casting shadows on close objects– Indexed by direction: k*n2 storage– Post-processing to extract AO values

Contact shadows

• Our solution:

Contact shadows

• Store raw AO values, un-processed• 3D grid: n3 storage• No post-processing required

• Why is it interesting?– Low frequency effect: small values of n

– Raw values stored: no constant– Similar memory costs, faster rendering

Global Illumination in static scenes• Direct lighting using GPU• Indirect lighting:

– Wavelet representation– precomputed Global Transport Operator

[Kontkanen, Turquin, Holzschuch, Sillion, EGSR 2006]

Global Illumination in static scenes• Wavelet basis for radiance• Local Transport Operator

– expressing a single light bounce– precomputed

• Global Transport Operator– Built from LTO, using Neumann series

• For each frame:– Compute direct lighting, projected on WT basis

– Apply GTO to direct lighting– Result is indirect lighting

Global Illumination in static scenes

Direct Indirect

Global

GPU-based algorithms: summary• GPU techniques for local/semi-local effects:– Specular reflections– Soft shadows– Contact shadows

• Interactive global illumination– GPU for direct lighting, CPU for indirect

– Limited to static scenes

Overview

• Motivations• Previous work• Contributions :

– Wavelet Radiosity – Frequency Analysis– GPU-based algorithms

• Conclusion & future work

Conclusion & future work

• Photorealistic Image Synthesis• Contributions:

– Wavelet radiosity– Frequency analysis– GPU-based algorithms

• Several applications and contracts:– VSP-Tech, SIMULGEN, Eden Games…

Future work: Real Time Global Illumination• Frequency analysis in real-time

– Used to guide computations

• GPU for local and semi-local HF effects– Direct lighting, shadows– Specular/glossy reflections– Occlusion and reflection fields

• CPU for other effects– Indirect lighting– Low Frequency effects

• Lots of applications and contacts:– Video games (GENAC II)– Movies

Future work:Image-Based Rendering• We want more realistic pictures• Requires more realistic models

– Hard to create from scratch– Image-Based Modeling for acquisition

• Opens several new directions of research:– Easy model acquisition– Real-time rendering of the model– Re-lighting of the model

• Need for cooperation:– 2 ANR projects submitted this year

• Lots of potential applications

Thank you!

Acknowledgements• Everyone I worked with• In addition to those already mentioned:

– iMAGIS, 1993-1996: A. Lamouret, F. Faure, JC Lombardo, JD Gascuel, N. Tsingos, M. Desbrun, S. Rivière, R. Orti, JC Vedel, A. Verroust, MP Cani, A. Opalach,…

– UCT, 1996-1997:E. Blake, S. Nirenstein, D. Cook, A. Secchia,…

– ISA, 1997-2000: S. Lazard, S. Merzouk, K. Tombre, H. Everett, MO Berger, A. Tabbone…

– iMAGIS, 2000-2003: J. Thollot, X. Decoret, J. Turbet, F. Neyret, L. Reveret, G. Debunne, S. Grabli…

– ARTIS, 2004-2007: A. Martinet, L. Boissieux, V. Ostromoukhov, E. Eisemann, P. Barla, L. Baboud, H. Bezerra, A. Bousseau, A. Orzan, T. Stein, K. Smith, F. Moulin, D. Vanderhaeghe…