Spherical Convolution in Computer Graphics and Vision Ravi Ramamoorthi Columbia Vision and Graphics Center Columbia University SIAM Imaging Science Conference:

Spherical Convolution in Spherical Convolution in Computer Graphics and VisionComputer Graphics and Vision

Ravi Ramamoorthi

Columbia Vision and Graphics CenterColumbia University

SIAM Imaging Science Conference: May 17, 2006

OutlineOutline

Motivation and Practical Problems

Spherical Convolution and Applications

A Theory of Spherical Harmonic Identities

Signal Processing for Visual Appearance

Real-Time RenderingReal-Time Rendering

Motivation: Interactive rendering with natural illumination and realistic, measured materials

Inverse RenderingInverse Rendering

Photographs

Geometric model

InverseRenderingAlgorithm

BRDF


Geometric model

ForwardRenderingAlgorithm

BRDF

Novel lighting

Rendering

Photographs

Direct Object RelightingDirect Object Relighting

?

Unknown Lighting

Unknown BRDF

Checking Image ConsistencyChecking Image Consistency

Easy to tamper / splice images Image processing software widely available

In news reporting and other applications Need to detect tampering or photomontage Verify image consistency

Try to check consistency of lighting, shading

Checking Image ConsistencyChecking Image Consistency

OutlineOutline





Environment MapsEnvironment Maps

Miller and Hoffman, 1984

Later, Greene 86, Cabral 87, 99,…

Irradiance Environment MapsIrradiance Environment Maps

Incident Radiance(Illumination Environment Map)

Irradiance Environment Map

R N

Computing IrradianceComputing Irradiance

Classically, hemispherical integral for each pixel

Lambertian surface is like a low pass filter

Frequency-space analysis (spherical harmonics)

IncidentRadiance

Irradiance

Analytic Irradiance FormulaAnalytic Irradiance Formula

Lambertian surface is low-pass filter

lm l lmE A LlA

2 / 3

/ 4

0

2 1

2

2

( 1) !2

( 2)( 1) 2 !

l

l l l

lA l even

l l

l0 1 2

Basri & Jacobs 01Ramamoorthi & Hanrahan 01a

9 Parameter Approximation9 Parameter Approximation

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

Exact imageOrder 29 terms

RMS Error = 1%

For any illumination, average error < 2% [Basri, Jacobs 01]

Ramamoorthi and Hanrahan 01b

Real-Time RenderingReal-Time Rendering

Simple procedural rendering method (no textures) Requires only matrix-vector multiply and dot-product In software or NVIDIA vertex programming hardware

Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)

( ) tE n n Mn

surface float1 irradmat (matrix4 M, float3 v) {

float4 n = {v , 1} ;

return dot(n , M*n) ;

}

Computer Vision Complex IlluminationComputer Vision Complex Illumination

Low Dimensional Subspace Lighting Insensitive Recognition (Basri and

Jacobs 01, Lee et al. 01, Ramamoorthi 02, …)

Photometric stereo, shape acquisition

Convolution for General MaterialsConvolution for General Materials

Ramamoorthi and Hanrahan 01

( , ) ( ) ,B N V L R N l l V dl

B L

Frequency: product

Spatial: integral

ij i ijB L

Spherical Harmonics

Related Theoretical WorkRelated Theoretical Work

Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99

Reflection as frequency-space operator: D’Zmura 91

Lambertian reflection is convolution: Basri Jacobs 01

Our Contributions Explicitly derive frequency-space convolution formula Formal quantitative analysis in general 3D case Apply to real-time, inverse rendering, computer vision

Natural Lighting, Realistic MaterialsNatural Lighting, Realistic Materials

Ramamoorthi and Hanrahan 02


3 photographs of cat sculpture• Complex unknown illumination• Geometry known• Estimate microfacet BRDF and distant lighting

New View, LightingNew View, Lighting

Photograph RenderingRamamoorthi and Hanrahan, 01c

OutlineOutline





Mahajan, Ramamoorthi, Curless ECCV 06

Ligh

ting

1Li

ghtin

g 2

Material 1 Material 2

Two Objects – Two LightingsTwo Objects – Two Lightings

1111lmllm LAB 1212

lmllm LAB

2121lmllm LAB 2222

lmllm LAB

Ligh

ting

1Li

ghtin

g 2



1111lmllm LAB 1212

lmllm LAB

2121lmllm LAB 2222

lmllm LAB

21212211lmlmlllmlm LLAABB

Ligh

ting

1Li

ghtin

g 2



1111lmllm LAB 1212

lmllm LAB

2121lmllm LAB 2222

lmllm LAB



Ligh

ting

1Li

ghtin

g 2



1111lmllm LAB 1212

lmllm LAB

2121lmllm LAB 2222

lmllm LAB



2211lmlmBB 2112

lmlmBB

Independent of Lighting and BRDF

Ligh

ting

1Li

ghtin

g 2


Image Estimation FrameworkImage Estimation Framework

?

11lmB

21lmB

12lmB

22lmB

21122211lmlmlmlm BBBB

11

211222

lm

lmlmlm B

BBB

Image EstimationImage Estimation

Object 1 Object 2

Ligh

ting

1

Our Method Actual

Ligh

ting

2

?

No BRDF and lighting known or estimated

Image Consistency CheckingImage Consistency Checking

Spliced Image


Spliced Image


Single Image Identitydiffuse + specular case

Two Lightings – Same Reflectance Identity

Two Materials – Two Lightings identity

Tampered Cat

Tampered Cat Untampered Cat

Signal Processing for AppearanceSignal Processing for Appearance

Signal Processing widely applicable visual appearance

Convolution relation for cast shadows [Soler and Sillion 98, Ramamoorthi et al. 04]

Convolution with glows for participating media (mist, fog, haze) [Sun et al. 05]

Signal-Processing analysis of light field and reflectance [Chai et al. 00, Zickler et al. 06]

Triple Product Integrals [Ng et al. 04]

First Order Analysis [Ramamoorthi et al. 06]

Video clip 1

Reflectance Sharing Reflectance Sharing

AcknowledgementsAcknowledgements

Collaborators Dhruv Mahajan Brian Curless Pat Hanrahan Sameer Agarwal (helpful discussions)

Funding: NSF and Sloan Foundation

http://www.cs.columbia.edu/~ravir

QuestionsQuestions

Spherical Convolution in Computer Graphics and Vision Ravi Ramamoorthi Columbia Vision and Graphics Center Columbia University SIAM Imaging Science Conference:

Documents

b slide

varied slide

shading slide

visual appearance slide

shape acquisition slide

interactive rendering

consistency of lighting

realtime rendering motivation