Marc Levoy Using Plane + Parallax to Calibrate Dense Camera Arrays Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc Levoy Computer Science Department.

Marc Levoy

Using Plane + Parallax to Calibrate Dense Camera Arrays

Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc Levoy

Computer Science DepartmentStanford University

Marc Levoy

The Stanford Multi-Camera Array

Marc Levoy

Synthetic Aperture Photography: Seeing through Foliage

Marc Levoy

Synthetic Aperture Photography: Seeing through Foliage

Marc Levoy

Outline

• Problem Statement– Synthetic aperture photography using an array

of cameras

– Calibration required

• Calibration Pipeline

• Results

• Future Work

Marc Levoy

Synthetic aperture photography

Marc Levoy

SAP: Prior Work

• Synthetic Aperture Radar

• Light Field Rendering [Levoy 96]

• Dynamically Reparametrized Light Fields [Isaksen 00]

• Single lens SAP [Favaro 03]

Marc Levoy

Outdoor SAP: Array layout

• width of aperture 2m• number of cameras 45• spacing between cameras 13cm• camera’s field of view4.5°

Marc Levoy

Outdoor SAP: The scene

• distance to occluder 33m

• distance to targets 45m

• field of view at target 3m

Marc Levoy

Outdoor SAP: Calibration

• Narrow field of view and long-range imaging makes accurate pose estimation difficult• Cannot take calibration measurements at the desired focal depth (behind occluding

bushes)

Calibration Volume

Focal Depth

28 m 5m

a = 2m

12m

Marc Levoy

Calibration Goal

Given a focal plane, compute the projective transform (homography) to project each camera image onto the plane.

Focal Plane

Marc Levoy

Approaches to Calibration

• Metric Calibration– Computes camera intrinsics, position,

orientation(10 parameters/camera)

– Nonlinear optimization, requires initial guess– Not stable for narrow angle lenses and long

range imaging

• Non-metric Calibration– Plane + Parallax methods [Irani 96, Triggs 00]– Homography Spaces [Zelnik-Manor 99]

Marc Levoy

Calibration Pipeline

• Problem Statement

• Calibration Pipeline– Focus cameras on one plane (using homographies)

– Compute relative camera positions from parallax measurements

– Use camera positions to vary focal plane over a range of depths

• Results

• Future Work

Marc Levoy

Focusing on one plane

+

Add camera images so that points on one plane are in good focus

Marc Levoy

Focusing at different depths

+

Marc Levoy

Focusing at different depths

+

To focus at a different depth, we have to shift the images by an amount equal to the parallax

Marc Levoy

Parallax and Camera Geometry

p1= X1 . z/(Z0 + z) = X1 . dP

Parallax = Camera shift * Relative Depth

Reference Plane

Camera Plane

P

P

p1

X1

z

Z0

Marc Levoy

Parallax and Camera Geometry

Measure parallax of P in all cameras (wrt reference camera)

[ p1 p2 . . . ]T

Reference Plane

Camera Plane

p1

X1

P

p2

X2

Marc Levoy

Recovering Camera Positions

Parallax of point P:

For multiple points P1, … Pn :

Relative camera positions Xi can be recovered robustly (up to scale) using SVD.

Marc Levoy

Computing SAP Images at different focal depths

• To change the focal depth, images have to be shifted by the amount of parallax.

• In camera C1 , the parallax for a parallel focal plane is f . X1 ,where f is a constant that depends only on the depth of the plane.

• f is analogous to the focus distance of the synthetic lens: varying f changes the depth of the focal plane.

p1

X1

Marc Levoy

Algorithm for SAP

1. Focus cameras onto a frontoparallel plane

2. Compute parallax for one (or more) scene points

3. Recover relative camera positions Xi (up to an unknown scale)

4. For a range of values of f :• Shift image from camera Ci by f . Xi and average

shifted images.• Varying f corresponds to changing the focus

Marc Levoy

Results

Synthetic Aperture Sequence

Marc Levoy

Parallax v/s Metric Calibration

Parallax-based calibration Metric calibration

Marc Levoy

Summary

• Calibration of camera arrays for synthetic aperture photography

– Decompose warps into reference homography and shifts

– Use parallax measurements to compute camera positions

– Avoids computing camera intrinsics and orientation explicitly

– Robust, linear solution

• Metric information not available

• Algorithm requires planar camera array and frontoparallel reference plane

Marc Levoy

Extension :Tilted Focal Planes

Reference PlaneFocal Plane

e (epipole)

L (line of intersection)

- Parallax is described by a projective warp (not a shift)

- Rank-1 factorization is still possible

Marc Levoy

Future Work

• Real-time applications– Warp images in hardware– Track moving objects by moving focal plane

• 3D Reconstruction from synthetic focus– Is this more robust to occlusions ?

• Quantitative analysis of synthetic aperture photography

– Effect of occluder density, number of cameras, aperture shape

Marc Levoy

Acknowledgements

• Sponsors– NSF IIS-0219856-001

– DARPA NBCH 1030009

• Assistance in acquisition– Gaurav Garg, Augusto Roman, Billy Chen, Pradeep Sen,

Doantam Phan, Guillaume Poncin, Jeff Klingner

High Speed Videography Using a Dense Camera ArrayB Wilburn, N Joshi, V Vaish, M Levoy, M Horowitz

Session 5A, 4:20pm