[email protected] 17/12/2014 (part) Slides ...cellnetmcweb.bioquant.uni-heidelberg.de/image-analysis/Short... · 17/12/2014 (part) Slides courtesy: SébastienTosi

Image restoration by deconvolution

[email protected]

17/12/2014

(part) Slides courtesy: Sébastien Tosi (IRB Barcelona)

A few concepts related to the topic

Convolution

Deconvolution

Point Spread Function

Noise

Fourier Transform

Spatial resolution

Pixel size

Rayleigh Criterion

Airy disk

Numerial Aperture

Refractive Index

Wavelength

Image formation (in Fluorescence microscopy)

Image from [9]

In fluorescence microscopy (in all its modes including widefield, confocal,

and multi-photon): the imaging process can be mathematically described

by a convolution

Imaging, Convonlution, Deconvolution

Convolution consists of replacing each point in the original object with its blurred image in all dimensions and summing together overlapping contributions from adjacent points to generate the resulting three-dimensional image

All microscopy techniques that include directly or indirectly a convolution in their image formation processes can benefit from image deconvolution.

2D Convolution

The convolution can also be computed by “stamping”

the kernel on each pixel of the image: the kernel is

scaled (multiplied) by the intensity of the central pixel

and accumulated (summed) in the output image.

11111

12221

12321

12221

11111

Filter kernel

� �

Original imageFiltered image Original imageFiltered image

3D Convolution

Convolution extends to more than two dimensions: in 3D the kernel is a small volume (stack)

and the sum is triple (inside a volume around each voxel).

=

A 3D kernel is a stack holdingthe filter coefficients

=⊗

⊗

Fourier transform (FT)

If we take the FT of the

equation, the is replaced by

multiplication, thus image

restoration might be achievable

by:

dividing the FT of the image by

the FT of the kernel and then

taking the inverse Fourier

transform.

⊗⊗

X

Spatial Domain

FourierDomain

=

=

Image from Sébastien Tosi (IRB

Barcelona)

Point Spread Function (PSF)

More slides from Math-Clinic BioImage Analysis website: http://goo.gl/u52WmC

An image resulting from a single small spherical fluorescent bead (smaller than

the optical resolution, thus forms effective a point source of light)

A record of how much the microscope has spread or blurred a single point

Simplified diagram to visualize how a light-emitting point would be imaged using a widefield microscope, Image from [9] Widefield PSF, Image from [3]

Spatial resolution: distance by which two

objects must be separated to be

distinguished, i.e. the radius of the smallest

point source in the image (defined as the

first minimum of the Airy disk)

the Rayleigh criterion:

2

2axial

nr

NA

λ=0.61radialr

NA

λ= λ : fluorophore emission wavelength

NA : objective numerical aperture

n : refractive index of the objective

lens immersion medium

NA can never exceed n, which itself

has fixed values (e.g. 1.0 for air, 1.33

for water, or 1.52 for oil)

NA = n sinθ

Notes:

1. Rayleigh criterion has not taken into account the effects

of: brightness, pixel size, noise

2. High NAs are possible when the immersion refractive

index is high

PSF (in the focal plane)

Widefield PSFs obtained by imaging 100-nm fluorescent beads (excitation

520nm; emission 617nm), Image from [2]

Image from [9]

Experimental PSF

(measured PSF) Theoretical PSF

Theoretical PSFQuotes from [3]:

Most methods are based on the work of Born and Wolf (1980).

A good description for a confocal PSF is given by van der Voort and Brakenhoff (1990), in

which the PSF is calculated from: the NA of the objective, the illuminating and emitted

wavelengths, and the refractive index of the immersion medium in either (simpler) paraxial

forms or with WF integrals.

Theoretical PSF gives an indication of the best possible resolution for a given objective but

these limits are not achievable.

In our experience, real PSFs are typically >20% bigger than calculated versions.

image from [9]

What else does (measured) PSF tells us?

Asymmetry:

radial (x-y): commonly misalignment of optical

components about the z-axis, either as tilt or decentration

along the optical axis (z-axis): commonly due to spherical

aberration, which may result from refractive index mis-

matches between the objective, immersion medium, and

sample or tube length/coverslip thickness errors.

Image from [3]

Notes:

1. The immersion refractive index should match the

refractive index of the medium surrounding the

sample, to avoid spherical aberration

2. Item 1 is often strongly preferable to using the

highest NA objective available, as it is usually

better to have a larger PSF than a highly irregular

one.

Deconvolution Principle

The deconvolution filter F should “undo” the effect of the microscope PSF H by

processing the sampled image R, ideally D = S.

Assuming H known, F linear (convolution) and no noise (N = 0) leads to:

or

or

In this context “1” is a black image holding a single point in its center

In practice the noise N and the error on the estimation of H (measurement or model) are

impeding a perfect deconvolution and we can at best hope for an approximate solution…

Object (sample)

Microscope PSF

AWGN source

Digital filterDeconvolved image

D H S H⊗ = ⊗( )R F H R⊗ ⊗ =

1F H⊗ =�

( )R F H R⊗ ⊗ =

Noise Sources in Digital Microscopy

More/Details

http://blogs.qub.ac.uk/ccbg/fluorescenc

e-image-analysis-intro (Part III)

RECOMMEND! It is a pleasant reading!

Ref [2]

More on reference slide…

When to do deconvolution?

• Wide field microscopy (WFM)

Less affected by out-of-focus light:

• Confocal laser scanning microscopy (CLSM)

• Two-photon excitation microscopy (TPEM)

• Selective plane illumination microscopy (SPIM)

Super-resolution fluorescence microscopy

• Stimulated emission depletion microscopy (STED)

• …

All microscopy techniques that

include directly or indirectly a

convolution in their image

formation processes can benefit

from image deconvolution.

Any 2D or 3D image obtained

from almost any fluorescence

microscope is expected to be

deconvolved before being analysed.

Why to do deconvolution?

• Attenuation of the out of focus light - increase contrast

• Reduce noise

• Increase of the spatial resolution

Deconvolution Algorithms Used in Biological Fluorescence Image Processing

Ref [2]

Deblurring - subtractive

Nearest neighbour

No neighbours

Linear inverse filter

Regularized Inverse filter

Object smoothness

e.g. Wiener filter

Constrained Iterative

Nonnegative

Blind deconvolution

Not for

quantitative

intensity

measurements

Do not count for

noise

Quantitative

No PSF as input

May not be

absolutely

quantitative

Linear Deconvolution: Inverse Filter Deconvolution

A very simple modelfor the PSF H

(Gaussian std = 1 pixel)

1

4,1·10-8

1

2,4·107

H power spectrum (log display)overlaid with raw values

H-1 power spectrum (log display)overlaid with raw values

1( , ) ( , )F u v H u v−=

As convolution in the spatial domain can be performed as a multiplication in the frequency

domain, inverse filtering can be performed as a division in the frequency domain!

But in practice…

Noise enhancement ruins our efforts!

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450

Inverse Filter Deconvolution

Original image Blurred image

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450

H +N

H-1

Original image S

H is a Gaussian with std = 2 pixels

50 100 150 200 250 300 350 400 450

50

100

150

200

250

300

350

400

450 Noise std = 10-4 Noise std = 10-12

S after convolution by H

No noise

Second try: Regularized Inverse

A very simple modelfor the PSF H

(Gaussian std = 1 pixel)

1

4,1·10-8

H power spectrum (log display)overlaid with raw values

(H-1)reg (1% clipping)power spectrum (log display)

overlaid with raw values

1

2,4·107

1

100

1 1

1

( , ), ( , )( , )

, ( , )

H u v H u v tF u v

t H u v t

− −

−

≤= >

Regularized Inverse Filter Deconvolution

Original image Blurred image

H +N

(H-1)trunc

Original image S S after convolution by H

Restoration of Blurred, Noisy Image Using regularized inverseNoise std = 10-4

Third Try: Wiener Filter The Golden Linear Deconvolution Trade-off

2*

2 2 2

( , ) ( , )( , )

( , ) · ( , ) ( , )

H u v S u vF u v

H u v S u v N u v=

+

( )estS F S H N= ⊗ ⊗ +

.estE S S= −Minimizing the expectation of ||E|| over all possible noise realizations assuming a

white Gaussian noise:

Coming back to:

Bands free of noise: |N(u,v)| = 0 ���� F(u,v) = H(u,v)-1 (inverse filter)

Strong noise bands: |N(u,v)|���� ∞ ���� F(u,v) ���� 0 (cut-off)

Intermediate bands: best trade-off

Wiener filter

Wiener filter attenuates frequencies dependent on their signal-to-noise ratio.

Wiener Deconvolution

Restoration of Blurred, Noisy Image Using Wiener filter for known noise varianceRestoration of Blurred, Noisy Image Using regularized inverseRegularized inverse filter resultNoise std = 10-4

Wiener filter resultNoise std = 10-4

Non-Linear Deconvolution

The best deconvolution algorithms for 3D microscopy are typically non-linear.

Principle of Maximum A Priori algorithms (MAP):

The second equality comes from Bayes theorem.

In the optimization S is usually constrained to be positive and somehow spatially smooth (TV

regularization term) � Pr(S).

The statistical distribution of the noise has to be known to derive the maximum likelihood

term Pr(R|S) � the algorithm is tuned to a particular noise (e.g. Poisson or Gaussian noise).

There is usually no known analytical solution to the problem, the algorithms proceeds by

iterations (candidate Si at iteration i) to refine the estimate of the data at each iteration.

The Richardson-Lucy algorithm is among the most well known MAP deconvolution algorithm.

Some algorithms also simultaneously estimate the PSF from the sampled image (blind

deconvolution).

( ) Pr( | )Pr( )arg max Pr( | ) arg max .

Pr( )MAP

S S

R S SS S R

R= =

Quantification with deconvolution

• Ideally: relocate signal to the point of origin in 3D, thus conserve the sum of fluorescence signal. It improves quantification!

• In practice: different algorithms have more or less compromises

• Quantitative intensity measurements, e.g. intensity ratio: controls, also report on un-deconvolved data for comparison

• Quantitative positional or structural analysis, e.g. centroid, tracking, volume analysis, (object based) colocalisation, etc: relatively less critical the choice

• For all analysis:

• Deconvolution process comparable between datasets

• Compare with control/un-deconvolved data

• Understand algorithm used and choose most suitable

• Report possible artifacts and confirm it, if possible

Software tools

PSF generators

Deconvolution

Deconvolution tools (not exhaustive!)

Parallel iterative deconvolution (fiji.sc/Parallel_Iterative_Deconvolution): 4 deconvolution algorithms

Parallel spectral deconvolution (fiji.sc/Parallel_Spectral_Deconvolution)

Not iterative, no constraint e.g. nonnegativity

Iterative Deconvolve 3D (fiji.sc/Iterative_Deconvolve_3D) :

non-negative, iterative, similar to WPL algorithm. The execution is way slower on modern (multicore) computers

but the memory requirement is less stringent

DeconvolutionLab (http://bigwww.epfl.ch/algorithms/deconvolutionlab/ ): different algorithms including a custom

version of the thresholded Landweber algorithm

Commercial software

- SVI Huygens

- MC AutoquantX

- …

Fiji plugins

I could not comment on commercial software, due to access issue.

Original AutoquantX (30IT, bead PSF)

Huygens (50IT, bead distilled PSF)

PID (WPL, Wiener Gamma 0.1, 50IT, bead PSF)

Original PID (WPL, 50IT, true PSF) AutoquantX (30IT, true PSF) Huygens (50IT, distilled true PSF)

Original AutoquantX (30IT, bead PSF)

Huygens (50IT, bead distilled PSF)

PID (WPL, 50IT, bead PSF)

Courtesy of Sébastien Tosi (IRB

Barcelona)

+ Microscope specific

PSF

+ depth-varying PSF

+ supports spinning

disk M.

+ Visually appealing

results

- Expensive & Closed

source

+ Free & Open source

& full control

+ Reasonably fast

+ Support for

spatially-variant

PSF (un-tested)

- High memory usage

- Visually less

crispy

+ Fast convergence

+ Robust algorithms

+ Very simple to use

+ Visually appealing

results

+ 2D mode for thin

samples

- Expensive & Closed

source

Examples

Theoretical PSF generator

PSF generator:

http://bigwww.epfl.ch/algorithms/psfgenerator/#download

>15 models

Diffraction PSF 3D:

http://fiji.sc/Diffraction_PSF_3D

using Fraunhofer diffraction

Parallel Iterative Deconvolution

The plugin provides 4 deconvolution

algorithms:

- Wiener Filter Preconditioned Landweber

(WPL)

- Modified Residual Norm

Steepest Descent (MRNSD)

nonnegative

- Conjugate Gradient for Least Squares

(CGLS)

- Hybrid Bidiagonalization Regularization

(HyBR)

regularized

Parallel Iterative Deconvolution

attempts to reduce artifacts from features near the boundary of the imaging volume.

stops the iteration if the changes appear to be increasing. Increase the low pass filter size if this problem occurs. fraction of the largest Fourier

coefficent of the PSF

higher value increases

convergence

affects the scaling of the result

Example

Image courtesy: K Peng

PSF (Fiji -> Plugins -> Diffraction PSF 3D)

Image 1 Image 2

Deconvolution (Fiji -> Plugins -> Parallel Iterative Deconvolution -> 3D Iterative Deconvolution)

Image 1 Image 3

Original

DeconvolvedImage 1 Image 3

Objects look brighter -> higher contrast

Better separation between close objects

Deconvolved + background subtractedImage 1 Image 3

Other resources/tools:A MatlabMatlabMatlabMatlab software

• http://www.unife.it/prin/software/sgp_deblurring_boundary.zip

Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy,

Scientific Reports 2013.

Other resources/tools:Fiji Squassh – segmentation / colocalization

• deconvolved (subpixel) segmentations in 2D&3D through prior knowledge of PSF

• Intensity within each object is homogeneous

• Bright foreground & dark background

• Noise model: Gaussian (wide field) or Poisson (confocal)

• (more robust) Joint deconvolution-segmentation procedure

Rizk et al., Segmentation and quantification of subcellular structures in fluorescence microscopy images using Squassh, Nature Protocol, 2014.

Summary

• Deconvolution is a computational technique allowing to (partly) compensate for theimage distortion created by an optical system

• Correct deconvolution should improve:

attenuation of the out of focus light

quantitative measurements

the spatial resolution

• Incorrect deconvolution could:

Introduce (more) artifacts -> reduce image quality

• It works best for thin (<50 um), optically transparent, fixed, bright samples.

• Challenging for live microscopy: short exposure (limit motion blur), objective adaptedto medium (limit spherical aberrations).

References (to name a few…)

Good reviews (overviews):

1. Waters, Accuracy and precision in quantitative fluorescence microscopy, JCB 2009

2. Parton et al., Lifting the fog: Image restoration by deconvolution, Cell biology 2006

3. Pawley, Chapter 25: “Image enhancement by deconvolution”, Handbook of biological confocal microscopy, 2006

4. McNally et al., Three-Dimensional Imaging by Deconvolution Microscopy, Methods 1999

Technical articles:

5. Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy, Scientific Reports 2013

6. Bertero et al., Image deconvolution, Proc. NATO A.S.I. 2004

7. Thiébaut, Introduction to image reconstruction and inverse problems, Proc. NATO A.S.I. 2002

On the web:

8. Olympus microscopy center (overview): http://www.olympusmicro.com/primer/digitalimaging/deconvolution/deconvolutionhome.html

9. Textbook: http://blogs.qub.ac.uk/ccbg/fluorescence-image-analysis-intro

10. http://fiji.sc/Deconvolution_tips