Advanced Computer Graphics (Fall 2010) CS 283, Lecture 17: Frequency Analysis and Signal Processing for Rendering Ravi Ramamoorthi cs283/fa10.

Advanced Computer Graphics Advanced Computer Graphics (Fall 2010)(Fall 2010)

CS 283, Lecture 17: Frequency Analysis and Signal Processing for Rendering

Ravi Ramamoorthi

http://inst.eecs.berkeley.edu/~cs283/fa10

MotivationMotivation

Signal-processing provides new understanding

Methods based on (Spherical) Fourier analysis

Allows understanding of sampling rates (in IBR)

Frequency-domain algorithms like convolution

This lecture high-level, mostly conceptual ideas. Follow original papers for details, applications

Plenoptic SamplingPlenoptic Sampling

Plenoptic Sampling. Chai, Tong, Chan, Shum 00

Signal-processing on light field

Minimal sampling rate for antialiased rendering

Relates to depth range, Fourier analysis

Fourer spectra derived for 2D light fields for simplicity. Same ideas extend to 4D

Siggraph’2000, July 27, 2000

A Geometrical Intuition

Zmin

Zopt

Camera i Camera i+1


A Geometrical Intuition

Zmin

Zopt

Camera i Camera i+1

Disparity Error <

1 Pixel

Rendering Camera


A Constant Plane

Z

vt

t

v

Z1

t

v

Z1


Two Constant Planes

Z

vt Z1

Z2

t

v

t

v

Z1Z2


Between Two Planes

Z

vt

t

v

Z1

t

v

Z1

Z2

Z2


Between Two Planes

Z

vt

t

v

Z1

t

v

Z1

Z2

Z2


Light Field Reconstruction


Minimum Sampling Curve

Joint Image and

Geometry Space

Joint Image and

Geometry Space



Number of Depth Layers

1 2 3 6 12 Accurate Depth

Number of Images

2x2

8x8

4x4

16x16

32x32

Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur

Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur

Kevin Egan

Yu-Ting Tseng

Nicolas Holzschuch

Frédo Durand

Ravi Ramamoorthi

Columbia University

Columbia University

INRIA -- LJK

MIT CSAIL

University of California, Berkeley

ObservationObservation

• Motion blur is expensive

• Motion blur removes spatial complexity

Basic ExampleBasic Example

• Object not moving

x

y

SPACE

f(x, y) f(x, t)

Space-time graph

TIM

E


x

y t

x

f(x, t)

• Low velocity, t [ 0.0, 1.0 ]

f(x, y)


x

y t

x

f(x, t)

• High velocity, t [ 0.0, 1.0 ]

f(x, y)

Shear in Space-TimeShear in Space-Time

x

y t

x

f(x, t)

• Object moving with low velocity

f(x, y)

shear


x

y t

x

• Object moving with high velocity

f(x, y) f(x, t)


• Object moving away from camera

x

y t

x

f(x, y) f(x, t)


• Applying shutter blurs across time

x

y t

x

f(x, y) f(x, t)

Basic Example – Fourier DomainBasic Example – Fourier Domain

• Fourier spectrum, zero velocity

t

x

f(x, t) F(Ωx, Ωt)texture

bandwidth

Ωt

Ωx


• Low velocity, small shear in both domains

f(x, t) F(Ωx, Ωt)

t

x

slope = -speed

Ωt

Ωx


• Large shear

f(x, t) F(Ωx, Ωt)

t

x Ωt

Ωx


• Non-linear motion, wedge shaped spectra

f(x, t)

Ωt

Ωx

F(Ωx, Ωt)

t

x

shutter bandlimits in

time

-min speed

-max speed

shutter applies blur across time

indirectly bandlimits in

space

Sampling in Fourier DomainSampling in Fourier Domain

Ωt

Ωxt

x+

• Sampling produces replicas in Fourier domain

• Sparse sampling produces dense replicas

Fourier DomainPrimal Domain

Standard Reconstruction FilteringStandard Reconstruction Filtering

• Standard filter, dense sampling (slow)

Ωt

no aliasing

Ωx

Fourier Domainreplicas

Standard Reconstruction FilterStandard Reconstruction Filter

• Standard filter, sparse sampling (fast)

Ωt

Fourier Domain

aliasing

Ωx

Sheared Reconstruction FilterSheared Reconstruction Filter

• Our sheared filter, sparse sampling (fast)

Ωt

Ωx

No aliasing!

Fourier Domain

Sheared Reconstruction FilterSheared Reconstruction Filter

• Compact shape in Fourier = wide in primal

t

x

Primal Domain

Ωt

Ωx

Fourier Domain

Car SceneCar Scene

Stratified Sampling4 samples per pixel

Our Method,4 samples per pixel

Teapot SceneTeapot SceneOur Method

8 samples / pix

motion blurred reflection

Reflection as ConvolutionReflection as Convolution

My PhD thesis (A signal-processing framework for forward and inverse rendering Stanford 2002)

Rewrite reflection equation on curved surfaces as a convolution with frequency-space product form

Theoretical underpinning for much work on relighting (next lecture), limits of inverse problems

Low-dimensional lighting models for Lambertian

AssumptionsAssumptions

Known geometry

Convex curved surfaces: no shadows, interreflection

Distant illumination

Homogeneous isotropic materials

Later precomputed methods: relax many assumptions

ReflectionReflection

2

2

id( )iL Lighting

L

i

( )oB Reflected Light Field

( , )i o BRDF

B

o

Reflection as Convolution (2D)Reflection as Convolution (2D)

2

2

id( )iL Lighting


( , )i o BRDF

Li o B

i

B

L L

o


2

2

id( )iL Lighting


( , )i o BRDF

( )iL ( , )i o id2

2

( , )oB

Li o B

L

i

Bo


( )iL ( , )i o id2

2

( , )oB

Li o B

L

i

Bo

ConvolutionConvolution

x

Signal f(x) Filter g(x)

Output h(u)

u


x


Output h(u)

u

1 1( ) ( ) ( )h u g x u f x dx

u1


x


Output h(u)

u

u2

2 2( ) ( ) ( )h u g x u f x dx


x


Output h(u)

u

u3

3 3( ) ( ) ( )h u g x u f x dx


x


Output h(u)

u

( ) ( ) ( )h u g x u f x dx

h f g g f Fourier analysis

h f g


B L

Frequency: product

Spatial: integral

( )iL ( , )i o id2

2

( , )oB

, ,2l p l l pB L Fourier analysis

R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp 195-208

L

i

Bo

Li o B

Spherical HarmonicsSpherical Harmonics

-1-2 0 1 2

0

1

2

.

.

.

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

1

Spherical Harmonic AnalysisSpherical Harmonic Analysis

, ,2l p l l pB L

2D:

22

0 0 ,( , , , ) ( [ , ]) ( , , , )o o i i i i o o i iB L R d d

3D:

( )iL ( , )i o id2

2

( , )oB

, ,lm pq l lm lq pqB L

Insights: Signal ProcessingInsights: Signal Processing

Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution



Filter is Delta function : Output = Signal

Mirror BRDF : Image = Lighting [Miller and Hoffman 84]

Image courtesy Paul Debevec



Signal is Delta function : Output = Filter

Point Light Source : Images = BRDF [Marschner et al. 00]

Am

plit

ude

Frequency

Roughness

Phong, Microfacet ModelsPhong, Microfacet Models

Mirror

Illumination estimationill-posed for rough surfaces

Analytic formulae in R. Ramamoorthi and P. Hanrahan“A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128

LambertianLambertian

Incident radiance (mirror sphere)

Irradiance (Lambertian)

N

l

2 / 3

/ 4

0

l0 1 2

R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) Oct 2001 pp 2448-2459

R. Basri and D. Jacobs “Lambertian Reflectance and Linear Subspaces” ICCV 2001 pp 383-390

2 1

2

2

( 1) !2

( 2)( 1) 2 !

l

l l l

lA l even

l l

9 Parameter Approximation9 Parameter Approximation

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

l

m

Exact imageOrder 29 terms

RMS Error = 1%

For any illumination, average error < 3% [Basri Jacobs 01]

Ramamoorthi and Hanrahan 01b

Convolution for general materialsConvolution for general materials

Ramamoorthi and Hanrahan 01

( , ) ( ) ,B N V L R N l l V dl

B L

Frequency: product

Spatial: integral

ij i ijB L

Spherical harmonic analysis

Real-Time RenderingReal-Time Rendering

Motivation: Interactive rendering with natural illumination and realistic, measured materials

Ramamoorthi and Hanrahan 02

Normal Map Filtering and RenderingNormal Map Filtering and Rendering

Han, Sun, Ramamoorthi, Grinspun 07

More Frequency AnalysisMore Frequency Analysis

Many other papers Shadows: Soler and Sillion 98, Ramamoorthi et al. 04 Gradients: Ward, Heckbert, Arvo, Igehy, Holzschuch,

Chen, Ramamoorthi, … Wavelets: Gortler et al. 93, …, Ng et al. 03

Full frequency analysis of light transport Durand et al. 05 Space and Angle, generalizes previous work

Many recent papers in computational imaging [Levin et al. 08, 09]

Advanced Computer Graphics (Fall 2010) CS 283, Lecture 17: Frequency Analysis and Signal Processing for Rendering Ravi Ramamoorthi cs283/fa10.

Documents

t t x tt

t x sampling

t t x slope

velocity t x fx

t t x shutter bandlimits

spacetime x y t x fx

camera x y t x fx

t object