Light field microscopy Marc Levoy, Ren Ng, Andrew Adams Matthew Footer, Mark Horowitz Stanford Computer Graphics Laboratory.

Light field microscopy

Marc Levoy, Ren Ng, Andrew Adams

Matthew Footer, Mark Horowitz

Stanford ComputerGraphics Laboratory

Marc Levoy

Executive summary

• captures the 4D light field inside a microscope

• yields perspective flyarounds and focal stacks from a single snapshot, but at lower spatial resolution

• focal stack → deconvolution microscopy → volume data

Marc Levoy

Devices for recording light fields

smallscenes

bigscenes

• handheld camera [Buehler 2001]

• camera gantry [Stanford 2002]

• array of cameras [Wilburn 2005]

• plenoptic camera [Ng 2005]

• light field microscope (this paper)

(using geometrical optics)

Marc Levoy

Light fields at micron scales

• wave optics must be considered– diffraction limits the spatial × angular resolution

• most objects are no longer opaque– each pixel is a line integral through the object

» of attenuation

» or emission

– can reconstruct 3D structure from these integrals» tomography

» 3D deconvolution

Marc Levoy

Conventional versus plenoptic camera

Marc Levoy

Conventional versus plenoptic camera

uv-plane st-plane

square-sided microlenses

125μ

Marc Levoy

Digital refocusing

• refocusing = summing windows extracted from several microlenses

Σ

Σ

Marc Levoy

Example of digital refocusing

Marc Levoy

Refocusing portraits

Marc Levoy

Macrophotography

Marc Levoy

Digitally moving the observer

• moving the observer = moving the window we extract from the microlenses

Σ

Σ

Marc Levoy

Example of moving the observer

Marc Levoy

Moving backward and forward

Marc Levoy

A light field microscope (LFM)

objective

specimen

intermediateimage plane

eyepiece

Marc Levoy

A light field microscope (LFM)

• 40x / 0.95NA objective

↓

0.26μ spot on specimen× 40x = 10.4μ on sensor

↓

2400 spots over 25mm field

• 1252-micron microlenses

↓

200 × 200 microlenses with12 × 12 spots per microlens

objective

specimen

intermediateimage plane

eyepiecesensor

→ reduced lateral resolution on specimen= 0.26μ × 12 spots = 3.1μ

Marc Levoy

A light field microscope (LFM)

sensor

160mm

2.5mm

0.2mm

Marc Levoy

Example light field micrograph

• orange fluorescent crayon

• mercury-arc source + blue dichroic filter

• 16x / 0.5NA (dry) objective

• f/20 microlens array

• 65mm f/2.8 macro lens at 1:1

• Canon 20D digital camera

ordinary microscope light field microscope

200μ

Marc Levoy

The geometry of the light fieldin a microscope

• microscopes make orthographic views

• translating the stage in X or Y provides no parallax on the specimen

• out-of-plane features don’t shift position when they come into focus

f

objective lensesare telecentric

Marc Levoy

Panning and focusing

panning sequence focal stack

Marc Levoy

Mouse embryo lung(16x / 0.5NA water immersion)

light fieldpan focal stack

200μ

Marc Levoy

Axial resolution(a.k.a. depth of field)

• wave term + geometrical optics term

• ordinary microscope (16x/0.4NA (dry), e = 0)

• with microlens array (e = 125μ)

• stopped down to one pixel per microlens

eNAM

n

NA

nλDOFtot

2

3.34.0

1535.02

8.225.193.31254.016

1

4.0

1535.02

237spots125.193.3

→ number of slicesin focal stack

= 12

(geometrical optics dominates)

(wave optics dominates)

Marc Levoy

3D reconstruction

• confocal scanning [Minsky 1957]

• shape-from-focus [Nayar 1990]

• deconvolution microscopy [Agard 1984]

– 4D light field → digital refocusing →3D focal stack → deconvolution microscopy →3D volume data

(UMIC SUNY/Stonybrook) (Noguchi) (DeltaVision)

Marc Levoy

3D deconvolution

• object * PSF → focus stack {object} × {PSF} → {focus stack} {focus stack} {PSF} → {object}

• spectrum contains zeros, due to missing rays

• imaging noise is amplified by division by ~zeros

• reduce by regularization, e.g. smoothing

focus stack of a point in 3-space is the 3D PSF of that imaging system

[McNally 1999]

{PSF}

Marc Levoy

Silkworm mouth(40x / 1.3NA oil immersion)

slice of focal stack slice of volume volume rendering

100μ

Marc Levoy

Insect legs(16x / 0.4NA dry)

volume rendering all-focus image[Agarwala 2004]

200μ

Marc Levoy

3D reconstruction (revisited)

• 4D light field → digital refocusing →3D focal stack → deconvolution microscopy →3D volume data

• 4D light field → tomographic reconstruction →3D volume data

(from Kak & Slaney)

(DeltaVision)

Marc Levoy

Implications of this equivalence

• light fields of minimally scattering volumes contain only 3D worth of information, not 4D

• the extra dimension serves to reduce noise, but could be re-purposed?

OpticalProjectionTomography[Sharpe 2002]

Marc Levoy

Conclusions

• captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time

Calcium fluorescent imagingof zebrafish larvae optic tectumduring changing visual stimula

Marc Levoy

Conclusions

• captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time

but...

• sacrifices spatial resolution to obtain control over viewpoint and focus

• 3D reconstruction fails if specimen is too thick or too opaque

Marc Levoy

Future work

• extending the field of view by correcting digitally for objective aberrations

Nikon 40x 0.95NA (dry) Plan-Apo

Marc Levoy

Future work

• extending the field of view by correcting digitally for objective aberrations

• microlenses in the illumination path→ an imaging microscope scatterometer

200μ

angular dependenceof reflection fromsingle squid iridophore

Marc Levoyhttp://graphics.stanford.edu/projects/lfmicroscope