LANGEVIN MONTE CARLO RENDERING WITH GRADIENT-BASED ADAPTATION
Fujun Luan1, Shuang Zhao2, Kavita Bala1, Ioannis Gkioulekas3
1Cornell University
2University of California, Irvine3Carnegie Mellon University
© 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
GRADIENTS ARE AWESOME
2
Deep learning Inverse problems
paramsscene
Image processing Physics simulation
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
GRADIENTS IN RENDERING
Differentiable rendering is a hot topic:• [Gkioulekas et al. 2013, 2016], [Khungurn et al. 2015], [Zhao et al. 2016], [Che et al.
2018], [Li et al. 2018], [Tsai et al. 2019], [Loubet et al. 2019], [Zhang et al. 2019, 2020], [Nimier-David et al. 2019, 2020]…
3
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
GRADIENTS IN RENDERING
Differentiable rendering is a hot topic:• [Gkioulekas et al. 2013, 2016], [Khungurn et al. 2015], [Zhao et al. 2016], [Che et al.
2018], [Li et al. 2018], [Tsai et al. 2019], [Loubet et al. 2019], [Zhang et al. 2019, 2020], [Nimier-David et al. 2019, 2020]…
4
Mainly used for inverse problem:
rendering params
We focus onforward rendering:
renderingparams
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N 5
Optimization:- Stochastic Gradient Descent (SGD)
MCMC sampling:- Langevin Monte Carlo (LMC)
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
OUTLINE
6
+Introduction to
Langevin Monte Carlo (LMC)Optimization-inspired
acceleration Ensuring unbiasedness Gradient caching
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N 7H2MC [Li et al. 2015] Ours5 mins
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O NRJMLT [Bitterli et al. 2017] Ours10 mins
PART 1Introducing Langevin MC
© 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
SAMPLING IN RENDERING
10
Primary sample space
Light pathconstruction
! = ∫ $ % &%
! ∈ 0, 1 !
!
• Estimated with Monte Carlo• Requires ! ∼ # for efficiency
Use ∇$ to improve sampling
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
SGD OVERVIEW
!!
11
Optimization problem: max" % &
Stochastic gradient descent/ascent:&# = &#$! + )%$!∇%(&#$!)∇% !&
!'!(
!)
scalar step size
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
Sampling problem: &#∼ %
Kelemen [2002]:&# = &#$! +.(0, 1&2)
Apply Metropolis Hastings to accept/reject
KELEMEN 2002
12
Gaussian noise
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
LMC OVERVIEW
13
Sampling problem: &#∼ %
Langevin MC:
&# = &#$! + )#$!∇% &#$! + 1)#$!
.(0, 1&2)
Apply Metropolis Hastings to accept/reject
Paul Langevin
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
LMC OVERVIEW
!!
14
∇%!&
Metropolis Hastings ∇%
!'
Sampling problem: &#∼ %
Langevin MC:
&# = &#$! + )#$!∇% &#$! + 1)#$!
.(0, 1&2)
Apply Metropolis Hastings to accept/reject
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
SGD VS. LMC
15
!!
∇# !"
!#!$
!%
Optimization:max" % &
SGD:&# = &#$! + )#$!∇%(&#$!)
Sampling:&#∼ %
LMC:
&# = &#$! + )#$!∇% &#$! + 1)#$!
.(0, 1&2)
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
Kelemen 2002MSE: 0.2954Original LMCMSE: 0.2465
PART 2Optimization-inspired
acceleration
© 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
18vertex 2vertex 1
Path contrib.
L
vertex 1
vertex 2
L
vertex 1
vertex 2Not much light
L
vertex 1
vertex 2
Not much light
zero contrib.
zero contrib.
Inspired by [Li et al. 2015]
RING CAUSTICS
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
RING CAUSTICS
19
Sampling problem: $!∼ #
Langevin MC:
$! = $!"# + '!"#∇# $!"# + 1'!"#
*(0, .$/)
!#$!
Gradient vanishes on flat surface
∇"
reject
reject
reject
reject
accept
acceptaccept
accept
acceptaccept
preconditioning matrix
Inspired by optimization:$! = $!"# +1%"&∇# $!"# +1%"&
"# *(0, .$/)
Slow explorationIsotropic noise
Fast explorationAdapt to local geometry
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
!!"#
accept
accept
accept
accept
accept
SELECTING THE PRECONDITIONING MATRIX
Exact Hessian of % [Li et al. 2015]requires 2nd-order gradientsfull matrix expensive matrix operations
20
!! = !!"# +$$"%∇& !!"# +$$"%"# '(0, +&,)
quasi-Newton methods: Adam, BFGS, …
Approximate Hessian of % (ours)reuses 1st-order gradientsdiagonal matrix efficient scalar operations
preconditioning matrix
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
LMC + ADAM
21
Adam preconditioning matrix4# is a function of all previous gradients
!#
$# depends on ∇f !#
!$!'
$& depends on ∇f !& , ∇f !#$' depends on
∇f !' , ∇f !& , ∇f !#1&
1(
1)
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
More accurate but expensive
FULL VS. DIAGONAL PRECONDITIONING
22
$! = $!"# +1!"#∇# $!"# +1!"#"# *(0, .$/)
$! = $!"# + 2345(1!"#)∇# $!"# + 2345(1!"#)"# *(0, .$/)
Less accurate but fast
• Matrix-vector multiplication with gradient• Matrix inversion for sampling
Better at equal time!
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
MSE: 0.0779Original LMCMSE: 0.2465LMC + Adam, fullMSE: 0.0807LMC + Adam, diagonalMSE: 0.0779
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N 24
Bias problem
Unfortunately, naïvely combining LMC + Adam causes Bias!
PART 3Ensuring unbiasedness
© 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
&#! = &#!$! +4%!$!∇% &#!$! +4#!$!$! .(0, 1&2)&#" = &#"$! +4%"$!∇% &#"$! +4#"$!$! .(0, 1&2)&## = &##$! +4%#$!∇% &##$! +4##$!$! .(0, 1&2)
WHY BIASED?
26
same pointdifferent
preconditioning matrices
Unbiasedness requires asymptotic time homogeneity:Preconditioning matrix !! → constant, when # → ∞
Adam violates asymptotic time homogeneity:
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
SOLUTION 1: DIMINISHING ADAPTATION
27
!!" = !! /|% + '
Unbiasedness requires asymptotic time homogeneity:Preconditioning matrix !! → constant, when # → ∞
Small t Large t
4# /|7 dominates 4# /|7 vanishes
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N 28
LMC + Adam, diagonalLMC + Adam, diagonal (w/ diminishing adaptation)
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
DRAWBACKS OF DIMINISHING ADAPTATION
• Gradual loss of adaptation – Problematic for complex scenes
• No gradient reuse– Need to re-calculate gradients
29
New path
…
…
!!" = !! /|% + '
PART 4Gradient caching
© 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
SOLUTION 2: CACHE-DRIVEN ADAPTATION
31
• Stage 1- use Adam- cache gradients
• Stage 2– freeze cache– use cached gradients
$! = cacheQuery(7!)
!#
Unbiasedness requires asymptotic time homogeneity:Preconditioning matrix !! → constant, when # → ∞
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
Kelemen 2002MSE: 0.2954LMCMSE: 0.2465LMC + Adam, diagonalMSE: 0.0807
Biased!
Diminishing adaptationMSE: 0.0608
Unbiased
Cache-driven adaptationMSE: 0.0459
RESULTS
© 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
COMPARISONS WITH PRIOR WORK
34
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N 35H2MC [Li et al. 2015] Ours7 mins
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N 36MMLT [Hachisuka et al. 2014] Ours20 mins
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N 37RJMLT [Bitterli et al. 2017] Ours10 mins
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
10 min
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
MM
LT [H
achi
suka
201
4]
H2M
C[L
i 201
5]
RJM
LT [B
itter
li20
17]
MEM
LT[J
akob
201
2]
Our
s
Ref
eren
ce
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
MM
LT [H
achi
suka
201
4]
H2M
C[L
i 201
5]
RJM
LT [B
itter
li20
17]
MEM
LT[J
akob
201
2]
Our
s
Ref
eren
ce
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
LIMITATIONS AND FUTURE WORK
• require gradientsbecoming common in modern renderers
• global explorationpotentially use gradient cache for this purpose
41
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
TAKE-HOME MESSAGE
42
+Introduction to
Langevin Monte Carlo (LMC)Optimization-inspired
acceleration Ensuring unbiasedness Gradient caching
L A N G E V I N M O N T E C A R L O R E N D E R I N G W I T H G R A D I E N T - B A S E D A D A P T A T I O N
ACKNOWLEDGMENTS
43
Our sponsors
Code and more results on the project website!
https://tinyurl.com/RenderLMC