The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL An Incremental Weighted Least Squares Approach To Surface Light Fields Greg Coombe Anselmo Lastra.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

An Incremental Weighted Least Squares Approach To Surface Light Fields

Greg Coombe

Anselmo Lastra

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

2

Image-Based Rendering

Generate new views of a scene from existing views

Sample appearance from physical worldLightfields [Levoy96], Lumigraphs [Gortler96]

…

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

3

Surface LightfieldsA surface light field [Wood00] represents the appearance of a model with known geometry and static lighting

( , , , )f u vθ φviewing direction

surface position

(u,v)

(θ,Φ)

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

4

Surface Lightfields

f1(Θ, Φ) f2(Θ, Φ)

f3(Θ, Φ)

f4(Θ, Φ)

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

5

SLF - Batch Process

Surface Lightfield Construction

Renderer

…

[Chen02, Hillesland03]

disk

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

6

SLF - Online Process

IncrementalSurface Lightfield

ConstructionRenderer

1024x768@ 15fps

[Coombe05]

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

7

Scattered Data Approximation

[Coombe05] required resampling to gridSamples are at arbitrary locations in domain due to geometry and camera

?

?

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

8

Scattered Data Approximation

Scattered Data Approximation in lightfields

Unstructured Lightfields [Buehler01]• Tesselation of pure lightfield

Polynomial Texture Maps [Malzbender01] • Fit polynomials to set of images

Radial Basis Functions [Zickler05]• Interpolate sparse reflectance data

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

9

Outline

A 2D scattered data approximation

Fast incremental construction

Realtime evaluation

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

10

Our Approach

Represent the surface lightfield using Weighted Least Squares approximation

Modify WLS for the incremental framework

Adaptive and Hierarchical

CPU/GPU implementationProcess each image in about 1-2 seconds

Real-time rendering

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

11

Bust model,75 training images

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

12

Least Squares Fitting

Find the “best” polynomial approximation to the input samples

“best” means minimizes sum of squared differences

the coefficients are determined by solving a linear system

input samples

reconstruction

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

13

Least Squares

domain

reconstructedfunction

input samples

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

14

Weighted Least Squares

Problem: LS is a global approximation

Solution: Divide domain into multiple LS approximations, and combine to get global approximation

Use a set of low-degree polynomials

Non-linear blending (Partition of Unity)

Good discussion in Scattered Data Approximation, Holgar Wendland

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

15

Weighted Least Squares

centers domains

polynomial approximations

reconstructedfunction

input samples

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

16

Weighted Least Squares

xx

x x

Θ

Φ

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

17

Outline

A 2D scattered data approximation

Fast incremental construction

Realtime evaluation

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

18

Incremental WLS

Feedback is important in SLF construction

As each image is captured, it must be incorporated into the representation

How do we determine the size of each domain?

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

19

Adaptive Construction

x

x

x

x x

x

x

x

Start out with large domainsAdaptive shrink as more points arrive

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

20

Hierarchical Construction

xx

xx

x

xx

x

x

x x

Start out with a single domainSubdivide as more points arrive (quadtree)

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

21

HierarchicalConstruction, First 10 images

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

22

Hierarchical and Adaptive

Hierarchical1x1 Polynomial Texture Mapping

Fast at first, slows down as refines

# of domains is a power of 2

AdaptiveSlow at first, speeds up as domains shrink

Can handle arbitrary # of domains

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

23

Outline

A 2D scattered data approximation

Fast incremental construction

Realtime evaluation

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

24

Implementation

Pose Estimation

Visibility / Reprojection

Incremental Weighted Least Squares

1024x768@ 15fps

GPU Renderer

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

25

Φ

Θ

GPU Implementation

Φ

Θ

Φ

Θ

Φ

Θ

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

26

Results

29K patches

4K patches

side by side, 14K patches

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

27

Pitcher model,65 images

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

28

Performance

~ 0.5 - 2 seconds per image for hierarchical construction

0.5s for 4K bust model

2s for 30K pitcher model

95% is Least Squares FittingAdaptive is 2-3x more expensive

Rendering is 60fps or more

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

29

Conclusion

Represent the surface lightfield using Weighted Least Squares approximation

Modify WLS for the incremental framework

Adaptive and Hierarchical

CPU/GPU implementationProcess each image in about 1-2 seconds

Real-time rendering

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

30

Future Work

Since construction is dominated by LS fitting, implement on GPU

Extend to surface reflectance (4D)Change basis functions

Order-of-magnitude more data

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL

31

Thanks

NVIDIA Graduate Student Fellowship

UNC Graphics Group

National Science Foundation

Questions?