Multi Aperture Photography

• 1. Multi-Aperture Photography Paul Green MIT CSAIL Wenyang Sun MERL Wojciech Matusik MERL Frdo Durand MIT CSAIL

2. Motivation http://photographertips.net Portrait Landscape Small Aperture Large Aperture Depth of Field Control Shallow Depth of Field LargeDepth of Field 3. Depth and Defocus Blur plane of focus sensor lens defocus blur depends ondistancefrom plane of focus subject rays from point in focus converge to single pixel circle of confusion 4. Defocus Blur & Aperture lens plane of focus defocus blur depends onaperturesize aperture http://photographertips.net sensor subject circle of confusion 5. Goals

• Aperture size is a critical parameter for photographers

• post-exposuredepth of field control

• extrapolate shallow depth of field beyond physical aperture

6. Outline

• Multi-Aperture Camera

• • New camera design

• • Capture multiple aperture settings simultaneously

• Applications

• • Depth of field control

• • Depth of field extrapolation

• • (Limited) refocusing

7. Related Work

• Computational Cameras

• • Plenoptic Cameras

• • • Adelson and Wang 92

• • • Ng et al 05

• • • Georgiev et al 06

• • Split-Aperture Camera

• • • Aggarwal and Ahuja 04

• • Optical Splitting Trees

• • • McGuire et al 07

• • Coded Aperture

• • • Levin et al 07

• • • Veeraraghavan et al 07

• • Wavefront Coding

• • • Dowski and Cathey 95

• Depth from Defocus

• • Pentland 87

McGuire et al 07 Adelson and Wang 92 Levin et al 07 Veeraraghavan et al 07 Georgiev et al06 Aggarwal and Ahuja 04 8. Plenoptic Cameras

• Capture 4D LightField

• • 2D Spatial (x,y)

• • 2D Angular (u,v Aperture)

• Trade resolution for flexibility after capture

• • Refocusing

• • Depth of field control

• • Improved Noise Characteristics

Lens Aperture u v Sensor (x,y) Lenslet Array Subject Lens (u,v) 9. 1D vs 2D Aperture Sampling u v Aperture 2D Grid Sampling http://photographertips.net 10. 1D vs. 2D Aperture Sampling 4 Samples Aperture 1D Ring Sampling 45 Samples u v Aperture 2D Grid Sampling http://photographertips.net 11. Optical Splitting Trees

• General framework for sampling imaging parameters

• • Beamsplitters

• • Multiple cameras

Large Aperture Camera Small Aperture Camera McGuire et al 07 Beamsplitter Incoming light 12. Goals

• post-exposuredepth of field control

• extrapolate shallowdepth of field

• (limited) refocusing

• 1d sampling

• no beamsplitters

• single sensor

• removable

13. Outline

• Multi-Aperture Camera

• • New camera design

• • Capture multiple aperture settings simultaneously

• Applications

• • Depth of field control

• • Depth of field extrapolation

• • Refocusing

14. Optical Design Principles

• 3D sampling

• • 2D spatial

• • 1D aperture size

• • 1 image for each ring

Aperture Sensor http://photographertips.net 15.

• Goal: Split aperture into 4 separate optical paths

• • concentric tilted mirrors

• • at aperture plane

Aperture Splitting Tilted Mirrors 16. Aperture Splitting Incoming light Sensor Mirrors Focusing lenses Tilted Mirrors 17. Aperture Splitting X Ideally at aperture plane ,but not physically possible! Solution:Relay Opticsto createvirtual aperture plane Photographic Lens Aperture Plane Relay system Aperture splitting optics New Aperture Plane 18. Optical Prototype Mirror Close-up main lens relay optics mirrors tilted mirrors lenses SLR Camera 19. Sample Data

• Raw data from our camera

20.

• Ideally would be rings

• Gaps are from occlusion

Point Spread Function Occlusion combined inner ring 1 ring 2 outer 21. Outline

• Multi-Aperture Camera

• • New camera design

• • Capture multiple aperture settings simultaneously

• Applications

• • Depth of field control

• • Depth of field extrapolation

• • Refocusing

22. DOF Navigation 23.

• Approximate defocus blur as convolution

DOF Extrapolation? ? Depends on depth and aperture size What isat each pixel in? - Circular aperture blurring kernel 24. DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur Blur size Aperture Diameter Largest physical aperture I E I 1 I 2 I 0 I 3 25. Defocus Gradient Defocus blur Gisslopeof this line Defocus Gradient Map Defocus Gradient Blur proportional toaperture diameter Blur size Aperture Diameter D I 1 I 2 I E I 0 I 3 Largest physical aperture focal length aperture diameter sensor distance object distance 26. Optimization

• solve for discrete defocus gradient values G at each pixel

• Data term

• Graph Cuts with spatial regularization term

Defocus Gradient Map Smallest Aperture Image 27. Depth of Field Extrapolation 28. Synthetic Refocusing

• Modify gradient labels and re-synthesize image

gradient map refocused map extrapolated f/1.8 refocused synthetic f/1.8 29. Synthetic Refocusing Video 30. Depth Guided Deconvolution

• Deconvolve (deblur) with kernel givenby defocus gradient map

Before After depth-guided deconvolution Defocus gradient map Smallest aperture image 31. Discussion

• Occlusion

• • Could help depth discrimination (coded aperture)

• Difficult alignment process

• • Mostly because prototype

• Refocusing limited by Depth of Field

• • helped by depth-guided deconvolution

• Texture required for accurate defocus gradient map

• • Not critical for depth of field and refocus

32. Summary

• • Multi-aperture camera

• • • 1D sampling of aperture

• • • Removable

• • Post-Exposure depth of fieldcontrol

• • Depth of field extrapolation

• • Limited refocusing

• • Depth-guided deconvolution

33. Thanks

• People

• • John Barnwell

• • Jonathan Westhues

• • SeBaek Oh

• • Daniel Vlasic

• • Eugene Hsu

• • Tom Mertens

• • Britton Bradley

• • Jane Malcolm

• • MIT Graphics Group

• Funding

• • NSF CAREER award 0447561

• • Ford Foundation predoctoral Fellowship

• • Microsoft Research New Faculty Fellowship

• • Sloan Fellowship