1 Online Construction of Surface Light Fields By Greg Coombe, Chad Hantak, Anselmo Lastra, and Radek Grzeszczuk
1
Online Construction of Surface Light Fields
By Greg Coombe, Chad Hantak, Anselmo Lastra, and Radek Grzeszczuk
2
* *
Image-Based Modeling
Benefits: Photorealistic content Modeling by acquisition
Difficulties: Sampling issues Lack of feedback
3
Motivation
Casual Capture
Pick up a video camera, wave it around the model, and capture reflectance model
4
Surface Light Field
A surface light field is a representation of the appearance of a model with known geometry and static lighting:
( , , , )f u v viewing direction
surface position
(u,v)
(θ,Φ)
5
Our system
Incremental system for online capture, construction, and rendering of surface light fields
Each image is processed as it is captured Interactive feedback
Guide user to undersampled regions
Data-driven heuristic Structured method for handling missing data
6
Previous Work Regular parameterizations light field data [Levoy96, Gortler96] Sparse, scattered data [Debevec96, Debevec98, Buehler01] BRDF capture systems [Dana99, Lafortune97, Marschner99,
Debevec01, Gardner03] Function fitting
Lafortune BRDF [McAllister02], Torrance-Sparrow BRDF [Sato97], Clusters of BRDFs [Lensh01], Homomorphic Factorization [McCool?], Bi-quadratic polynomials [Malzbender01]
Online Methods Fixed viewpoint, progressive refinement [Matusik04] Adaptive Meshing of light field [Schirmacher99] Streaming non-linear optimization [Hillesland04]
Data Mining [Brand02, Roweis97]
7
SLFs using PCA
Represent 4D function as a matrix Use Principal Component Analysis [Noshino01,
Chen02] For each surface patch, break the 4D function into two
2D functions Store the principal components in texture maps
),(*),(),,,( vuhgvuf rank
8
SLFs using PCA
Problems: Requires that the entire data set be available
at once. Difficult to locate undersampled regions Requires recomputation when new images are
added
SVD*
9
Online SVD
Online SVD [Brand02] is a incremental PCA Update output matrices one sample at a time Advantages:
Never store the entire data matrix at once Stream images
Online SVD*...
10
Online SVD Algorithm
( )
0
diag s m
p
[ ', ', ']diagonalize U S V
6. The rotations are computed by re-diagonalizing this matrix
-p a Um3. Compute the orthogonal component
pTm U a
2. Project new image samples onto eigenspace
j 4. If ||p|| is less than a threshold, then we just compute the rotations and update the eigenspace.
U’ = URu V’ = VRv 5. Otherwise, the current rank r is insufficient, so we
append a new eigenvector.
U’ = [U; m]Ru V’ = VRv
1. Existing rank-r PCATA USV
12
Imputation
Missing data in red
Imputed
Matrix factorization approach requires fully resampled data matrices
Missing data is a common problem Occlusion, meshing errors
Imputation is the process of filling in these holes with “reasonable” guesses Better than zeros or mean
Use the current Online SVD approximation to generate these missing values In practice, need about 5-10 initial
images, and 50%+ coverage Further details in [Brand03]
13
Automatic Feedback
Aid the user in capturing surface light fields Direct attention towards
undersampled areas Use a data-driven
quality heuristicE = sqrt( e0^2 + e1^2)
e0 – variation over hemisphere
e1 – variation over surface
14
System
Real Time PoseEstimation
Resampling* Online SVDVisibility*
QualityHeuristic
VideoCamera
Rendering
Capture PC Render PC
* From Intel’s OpenLF system
18
Conclusion
We present a method with Incremental construction Data-driven quality heuristic
Online SVD is well-suited for streaming model Combine distribution and solution Requires only one pass over data
19
Future Work
We are still a long ways from “Casual Capture” Geometry must be known a priori Fixed illumination conditions
Implementation issues Real-time pose estimation errors
21
Online SVD
Problem: Multiple small rotations can accumulate error
Brand proposes splitting output matrices to avoid accumulating error
Advantages: Due to small working set, most data in cache (fast)
TSVD
TSVD
VSVUUA
USVA
''
22
Higher-Dimensional Problems
We would like to extend Online SVD to higher-order factorizations Moveable light source
Introduces another dimension of data
Requires tensor product expansion
),(*),(*),(),,,,,( 22112211 kvuhgvuf
View maps Surface maps Light maps