Direct-to-Indirect Transfer for Cinematic Relighting Milos Hasan (Cornell University) Fabio Pellacini (Dartmouth College) Kavita Bala (Cornell University)

Direct-to-Indirect Transfer for Cinematic Relighting

Direct-to-Indirect Transfer for Cinematic Relighting

Milos Hasan (Cornell University)

Fabio Pellacini (Dartmouth College)

Kavita Bala (Cornell University)

IntroductionIntroduction

• Cinematic Relighting [Gershbein 2000, Pellacini 2005]

– Interactive local light movement, fixed camera

– Procedural shaders, various materials

– High-complexity scenes

– Only direct illumination

• The goal of our system:

– Add multiple bouncesof indirect illumination

Direct-to-Indirect TransferDirect-to-Indirect Transfer

• Direct illumination from local lights

– Use known techniques

– Deep frame-buffer, shadow maps

• Indirect illumination – key idea:

– Precompute direct-to-indirect transfer matrix

– Compute indirect from direct on the fly

Direct Indirect Final

Transfer matrix

Interactive DemoInteractive Demo

Related WorkRelated Work

• Cinematic relighting engines

– [Gershbein 00; Pellacini 05, Tabellion 04]

• Precomputed radiance transfer

– [Sloan 02, 03, 05; Kautz 02; Ng 03, 04; Liu 04; Wang 04, 05; Annen 04, etc.]

• Linear combinations of full solutions

– [Kristensen 05; Dorsey 91; Debevec 00; etc.]

Related WorkRelated Work

• Sparse sampling– [Walter 99, 02; Bala 99, 03; Ward 99; Tole 02;

Nijasure 03; Gautron 05; Dayal 05; Simmons 01 etc.]

• Hierarchical / instant radiosity methods– [Hanrahan 91; Keller 97; Drettakis 97; Christensen

97; Dachsbacher 05, 06; etc.]

• Wavelet radiance transport

– [Kontkanen 06] (concurrent work)

OutlineOutline

• Key concepts

• Transfer matrix

– Efficient precomputation

– Efficient compression

– Efficient evaluation

• Results

• Conclusions and Future work

Key ConceptsKey Concepts

• View samples

• Gather samples

• Transfer: A large matrix

– Contributions of gather samples to view samples

i = T d

Indirect on view

Transfer matrix

Direct on gather

View SamplesView Samples

A scene

Camera

View samples

Gather SamplesGather Samples

A scene

Gather samples

Transfer Matrix ElementsTransfer Matrix Elements

Tij = ??

Gather sample j

View sample i

Direct IlluminationDirect Illumination

• Standard techniques: GPU, shaders, shadow maps

Direct on gather samples Direct on view samples

Conceptual OverviewConceptual Overview

Direct on gather

Indirect on view

Final

Transfer matrix

Direct on view

OutlineOutline

• Key concepts

• Transfer matrix

– Efficient precomputation

– Efficient compression

– Efficient evaluation

• Results

• Conclusions and future work

Why Is Precomputation HardWhy Is Precomputation Hard

• Transfer matrix is huge!

– View samples:640 x 480 = 300k rows

– Gather samples: 64k columns

– 19 billion elements

• Columns:

– Images with 1 light and full global illumination

64k gather

300k view

Full global illumination

images

Multi-bounce & Final GatherMulti-bounce & Final Gather

• Split full transfer matrix:

– Multiple-bounce matrix (lower accuracy)

– Final gather matrix (higher accuracy)

Full transfer:Many gather-to-view

Multi-bounce:Many gather-to-gather

Final gather:One gather-to-view

Final Gather Matrix ElementsFinal Gather Matrix Elements

Fij = ??

View sample i

Gather sample j

Single-bounce contribution

Multi-bounce Matrix ElementsMulti-bounce Matrix Elements

Gather sample i

Mij = ??

Gather sample j

Multi-bounce Matrix:Sparse ApproximationMulti-bounce Matrix:Sparse Approximation

• Photon mapping [Jensen 96] analogy

– But, don’t know light positions

• Shoot photons from gather samples instead!

– Photons carry gather sample ID

– Like standard photon mapping with 64k lights!

Multi-bounce Matrix:Computing by Photon MappingMulti-bounce Matrix:Computing by Photon Mapping

Gather sample i

i-th row of M = ?? A gather sample

K nearest photons

Conceptual Overview, UpdatedConceptual Overview, Updated

Direct on gather Indirect on gather Dir + Ind on gather

Indirect on view Final

Multi-bounce matrix

Add direct

Final gather matrix

Add direct

OutlineOutline

• Key concepts

• Transfer matrix

– Efficient precomputation

– Efficient compression

– Efficient evaluation

• Results

• Conclusions and future work

Defining a Wavelet BasisDefining a Wavelet Basis

• Want to use wavelets

– How? There is no obvious domain to apply them…

– Unstructured cloud of gather samples

• Still possible to define wavelets on the cloud!

– Flatten gather cloud to a 2D array

– Maintain coherence

– Use 2D Haar wavelets on the 2D array

Defining a Wavelet BasisDefining a Wavelet Basis

64k 256 x 256

Gather samples

Flattened in a 2D array

Wavelet transform

256 x 256

Hierarchical PartitioningHierarchical Partitioning

Tree ArrangementTree Arrangement

Wavelet CompressionWavelet Compression

• Transform rows into Haar wavelets

• Remove small coefficients

64k

300k

Dense

64k

300k

Sparse

64k

64k

Even more sparse

64k

64k

Sparse

Final gather matrixMulti-bounce matrix

Columns of the Wavelet-Transformed MatrixColumns of the Wavelet-Transformed Matrix

Images lit by wavelet lights

Final gather matrix in wavelets

Full System OverviewFull System Overview

Direct on gather Dir + Ind on gather

Indirect on view

Direct on gather in wavelet space

Dir + Ind on gather in wavelet space

Wavelet Xform

Wavelet Xform

Multi-bounce matrix

in waveletspace

Final gather matrix

in waveletspace

Final Image

Add direct

OutlineOutline

• Key concepts

• Transfer matrix

– Efficient precomputation

– Efficient compression

– Efficient evaluation

• Results

• Conclusions and future work

Sparse Matrix-Vector MultiplicationSparse Matrix-Vector Multiplication

Sparse matrix

Sparse vector

Linear combination of columns

Sparse Matrix-Vector Multiplication on the GPUSparse Matrix-Vector Multiplication on the GPU

• Problem:

– Columns are sparse themselves

– How to represent on the GPU?

• Solution:

– Non-zero elements tend to cluster…

– Cut out rectangular blocks

– Pack blocks into texture atlases

– Converted problem to blending!

Scene: Still LifeScene: Still Life

Precomputation: 1.6 hours 11.4 – 18.7 fps Polygon count: 107k

Scene: TempleScene: Temple

Precomputation: 2.5 hours 8.5 – 25.8 fps Polygon count: 2 million

Scene: Hair BallScene: Hair Ball

Precomputation: 2.9 hours 9.7 – 24.7 fps Polygon count: 320k

Scene: Sponza AtriumScene: Sponza Atrium

Precomputation: 1.5 hours Frame-rate: 13.7 – 24.9 Polygon count: 66k

ComparisonComparison

Our system: 8-25 fps (2.5 hr precomputation)

Monte Carlo path tracer: 32 hours

Conclusions and Future WorkConclusions and Future Work

• Conclusion

– Extend cinematic relighting with indirect illumination

– Interactive performance (GPU), complex scenes

– Arbitrary light shaders, efficient precomputation

• Future work

– Environment mapping

– Subsurface scattering

– Moving camera

Questions?Questions?

• Acknowledgements

– NSF grant CCF-0539996

– Bruce Walter (code, support)

– Veronica Sunstedt, Patrick Ledda (Temple)

– Marko Dabrovic (Sponza atrium)

– Cornell animation TA’s (Still life)

– Pixar, NVidia

• E-mail: [email protected]