Inverse Volume Rendering with Material Dictionaries Ioannis Gkioulekas 1 Shuang Zhao 2 Kavita Bala 2 Todd Zickler 1 Anat Levin 3 1 Harvard 3 Weizmann 2 Cornell 1
Feb 24, 2016
Inverse Volume Rendering with Material Dictionaries
Ioannis Gkioulekas1 Shuang Zhao2 Kavita Bala2
Todd Zickler1 Anat Levin3
1Harvard 3Weizmann2Cornell
1
Most materials are translucent
2
jewelry
skin
architecture
Photo credit: Bei Xiao, Ted Adelson
food
We know how to render them
3
Monte-Carlo rendering
material parameters
Veach 1997, Dutré et al. 2006
?rendered image
We show how to measure them
4
inverse rendering
material parameters
rendered imagecaptured photograph
Our contributions
5
material1. exact inverse volume rendering
• with arbitrary phase functions!
2. validation with calibration materials known
parameters
3. database of broad range of materialsthin thick
non-dilutable solids
material sample
Why is inverse rendering so hard?
6
radiative transferrandom walk of photons inside
volume • volume light transport has very complex dependence material parameters
thin thick
non-dilutable solids
thin thick
non-dilutable solids
Light transport approximations
7
Previous approach: single-scattering
random walk of photons inside
volume
single-bounce random walk
Narasimhan et al. 2006
Light transport approximations
8
Previous approach: diffusion
Jensen et al. 2001 Papas et al. 2013
…………
isotropic distribution of
photons
parameter ambiguity
≈ ≠material 1
material 2
random walk of photons inside
volume
thin thick
non-dilutable solids
Inverse rendering without approximations
9
random walk of photons inside
volume
exact inversion of random walk
thin thick
non-dilutable solids
Our approach
10
appearance matching
ii. operator-theoretic analysis
i. material representation
iii. stochastic optimization
Background
11
phase function p(θ)scattering coefficient σs
extinction coefficient σt
θ
m = (σt σs p(θ))
random walk of photons inside
medium
Papas et al. 2013
Phase function parameterization
12
g∈ (−1,1 )❑
not general enough
Henyey-Greenstein lobesChen et al. 2006
Donner et al. 2008
Fuchs et al. 2007
Goesele et al. 2004
Gu et al. 2008
Hawkins et al. 2005
Holroyd et al. 2011
McCormick et al. 1981
Pine et al. 1990
Prahl et al. 1993
Wang et al. 2008
Gkioulekas et al. 2013
Narasimhan et al. 2006
Jensen et al. 2001
Previous approach: single-parameter families
m = Σq πq mqp = Σq πq pq
D = {m1, m2, …, mQ}
Dictionary parameterization
13
tent phase functions
D = {p1, p2, …, pQ}
p1p2p3p4p5p6p7p8p9p10p11
dictionary of
• arbitrary
p • similarly for σt and σs
π1
π2
π3π4π5π6π7π8π9π10π11
D
phase functions
phase functions
materials
materials
σt = Σq πq σt,q σs = Σq πq σs,q
Our approach
14
appearance matching
ii. operator-theoretic analysis
i. material representation
iii. stochastic optimization
m = Σq πq mq
Operator-theoretic analysis
15
m = (σt σs p(θ))
τ τ ττ
random walk of photons inside
mediumdiscretized random walk paths• propagation step τ
• total radiance
K(π) = Σq πq Kq
Operator-theoretic analysis
16
m = (σt σs p(θ))
discretized random walk paths• propagation step τ
L(x, θ)
radiance at all medium points and directions
Ln+1(x, θ) = Ln(x, θ)K
• rendering operator R= (I - K)-1 LinputL = Σn Ln
L(x, θ) = R Linput(x, θ)
radiance after n steps
radiance after n+1
steps
R(π)= (I - Σ q πq Kq)-1
dictionary representation:
m = Σq πq mq
Our approach
17
appearance matching
ii. operator-theoretic analysis
i. material representation
iii. stochastic optimization
m = Σq πq mq
R(π)= (I - Σ q πq Kq)-1
Stochastic optimization
18
appearance matching
analytic operator expression for gradient! =
R(π)
render(π)single-stepq ··render(π)R(π)Kq
• gradient descent optimization for inverse rendering
min ǁ photo - render(π) ǁ2
π
Stochastic optimization
19
• exact gradient descent
for k = 1, …, N,
πk = πk - 1 - ak
end
N = a few hundreds
several CPU hours*
=intractable
exact
Stochastic optimization
20
Monte-Carlo rendering to compute
102 samplesnoisy + fast
104 samples 106 samplesaccurate + slow
Stochastic optimization
21
• exact gradient descent
for k = 1, …, N,
πk = πk - 1 - ak
end
N = a few hundreds
several CPU hours*
=intractable
• stochastic gradient descent
for k = 1, …, N,
πk = πk - 1 - ak
end
N = a few hundreds
few CPU seconds*
=solvable
exact
noisy
Theory wrap-up
22
appearance matching
ii. operator-theoretic analysis
i. material representation
iii. stochastic optimization
m = Σq πq mq
R(π)= (I - Σ q πq Kq)-1
𝜕 loss (π )𝜕 π q
noisy
min ǁ photo - render(π) ǁ2
π
Our contributions
23
material1. exact inverse volume rendering
• with arbitrary phase functions!
2. validation with calibration materials known
parameters
3. database of broad range of materialsthin thick
non-dilutable solids
Measurements
24
multiple lighting multiple viewpoints
appearance matching
min ǁ photo - render(π) ǁ2
π
Acquisition setup
25
material sample
frontlighting
backlightingcamera
Acquisition setup
26
bottom rotation stage
top rotation stage
material sample
frontlighting
backlighting
material samplefrontlighting
camera
backlighting
bottom rotation stage
top rotation stage
camera
Validation
27 Frisvad et al. 2007
polystyrenemonodispersions
aluminum oxidepolydispersions
very precise dispersions (NIST Traceable Standards)
calibration materials
known parameters
Mie theory
size
%particle material
medium material
Parameter accuracy
28
polystyrene 1 polystyrene 2 polystyrene 3 aluminum oxide
all parameters estimated within 4% error
comparison of ground-truth and measured parameters
ground-truthmeasuredHenyey-Greenstein fit
θ
-π π0
p(θ)
Matching novel measurements
29
captured rendered rendered with HG profilespolystyrene 3
comparison of captured and rendered images
images under unseen geometries predicted within 5% RMS error
ground-truthmeasuredHenyey-Greenstein fit
Our contributions
30
material1. exact inverse volume rendering
• with arbitrary phase functions!
2. validation with calibration materials known
parameters
3. database of broad range of materialsthin thick
non-dilutable solids
thin thick
non-dilutable solids
Measured materials
31
mustard
whole milk
shampoo
hand cream
coffee
wine
robitussin
olive oil curacao
mixed soap
milk soap
liquid clayreduced milk
extinction absorption first momentmaterial R G B R G B R G B
whole milk 100.92 105.345 102.768 0.013 0.013 0.041 0.954 0.963 0.946reduced milk 57.291 62.46 63.757 0.007 0.007 0.024 0.954 0.957 0.942mustard 16.447 18.536 6.457 0.057 0.061 0.451 0.155 0.173 0.351shampoo 8.111 9.919 10.575 0.178 0.328 0.439 0.907 0.882 0.874hand cream 20.82 32.353 41.798 0.011 0.011 0.012 0.188 0.247 0.265liquid clay 37.544 48.25 67.949 0.004 0.004 0.005 0.312 0.442 0.512milk soap 7,625 8.004 8.557 0.003 0.004 0.015 0.164 0.167 0.155mixed soap 3.923 4.018 4.351 0.003 0.005 0.013 0.33 0.322 0.316glycerine soap 0.201 0.202 0.221 0.001 0.001 0.002 0.955 0.949 0.943robitussin 0.009 0.001 0.001 0.012 0.197 0.234 0.906 0.977 0.98coffee 0.054 0.051 0.049 0.275 0.309 0.406 0.911 0.899 0.906olive oil 0.041 0.039 0.012 0.062 0.047 0.353 0.946 0.954 0.975blue curacao 0.01 0.012 0.021 0.083 0.048 0.011 0.955 0.973 0.979red wine 0.015 0.013 0.011 0.122 0.351 0.402 0.947 0.975 0.977
whole milk reduced milk mustard shampoo hand cream
liquid clay milk soap mixed soap glycerine soap robitussin
coffee olive oil curacao wine
Measured phase functions
33
θ
-π π0
p(θ)
measuredHenyey-Greenstein fit
Synthetic images
34
mixed soap
glycerine soap olive oil curacao whole milk
rendered image
Synthetic images
35
chromaticity
Synthetic images
36
mixed soap
glycerine soap olive oil curacao whole milk
rendered image
Effect of phase function
37
mixed soap
measured phase function
Henyey-Greenstein fit
θ-π π0
p(θ)rendered image chromaticity
measuredHenyey-Greenstein fit
Discussion
38
• faster capture and convergence: trade-offs between accuracy, generality, mobility, and usability
• more interesting materials: more general solids, heterogeneous volumes, fluorescing materials
• other setups: alternative lighting (basis, adaptive, high-frequency), geometries, or imaging (transient imaging)
Take-home messages
39
material1. exact inverse volume rendering
• with arbitrary phase functions!
2. validation with calibration materials known
parameters
3. database of broad range of materialsthin thick
non-dilutable solids
Acknowledgements
40
• Henry Sarkas (Nanophase)• Wenzel Jakob (Mitsuba)
Funding:• National Science Foundation • European Research Council• Binational Science Foundation• Feinberg Foundation• Intel• Amazon
http://tinyurl.com/sa2013-inverseDatabase of measured materials:
Error surface
41
appearance matching min ǁ photo - render(π) ǁ2
π
Light generation
42
MEMS light switch
RGB combiner
blue (480 nm) laser
green (535 nm) laser
red (635 nm) laser