The Camera Topic 1 Week 1 – Jan. 9 th , 2019 1
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
The CameraTopic 1
Week 1 ndash Jan 9th 2019
1
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
2
Camera Obscura
3
Camera Obscurabull Latin ldquoDark Roomrdquo
4
Camera Obscura
bull Aristotle (350 BC) writes about itbull Photos from August 2017 Solar Eclipse
5
The Pinhole Camera
ldquopinholerdquo aperture
optical axis
world point
projection of world point
image plane
7
Modern Camera
bull Cross-section of the Canon EOS Mbull Compound Lens CMOS sensorhellipbull Same optical principals
Source dpreviewcom
8
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
2
Camera Obscura
3
Camera Obscurabull Latin ldquoDark Roomrdquo
4
Camera Obscura
bull Aristotle (350 BC) writes about itbull Photos from August 2017 Solar Eclipse
5
The Pinhole Camera
ldquopinholerdquo aperture
optical axis
world point
projection of world point
image plane
7
Modern Camera
bull Cross-section of the Canon EOS Mbull Compound Lens CMOS sensorhellipbull Same optical principals
Source dpreviewcom
8
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Camera Obscura
3
Camera Obscurabull Latin ldquoDark Roomrdquo
4
Camera Obscura
bull Aristotle (350 BC) writes about itbull Photos from August 2017 Solar Eclipse
5
The Pinhole Camera
ldquopinholerdquo aperture
optical axis
world point
projection of world point
image plane
7
Modern Camera
bull Cross-section of the Canon EOS Mbull Compound Lens CMOS sensorhellipbull Same optical principals
Source dpreviewcom
8
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Camera Obscurabull Latin ldquoDark Roomrdquo
4
Camera Obscura
bull Aristotle (350 BC) writes about itbull Photos from August 2017 Solar Eclipse
5
The Pinhole Camera
ldquopinholerdquo aperture
optical axis
world point
projection of world point
image plane
7
Modern Camera
bull Cross-section of the Canon EOS Mbull Compound Lens CMOS sensorhellipbull Same optical principals
Source dpreviewcom
8
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Camera Obscura
bull Aristotle (350 BC) writes about itbull Photos from August 2017 Solar Eclipse
5
The Pinhole Camera
ldquopinholerdquo aperture
optical axis
world point
projection of world point
image plane
7
Modern Camera
bull Cross-section of the Canon EOS Mbull Compound Lens CMOS sensorhellipbull Same optical principals
Source dpreviewcom
8
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
The Pinhole Camera
ldquopinholerdquo aperture
optical axis
world point
projection of world point
image plane
7
Modern Camera
bull Cross-section of the Canon EOS Mbull Compound Lens CMOS sensorhellipbull Same optical principals
Source dpreviewcom
8
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Modern Camera
bull Cross-section of the Canon EOS Mbull Compound Lens CMOS sensorhellipbull Same optical principals
Source dpreviewcom
8
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Simple Camera with Lens
ldquothinrdquo lens
world point
projection of world point
image plane
optical axis
= imagedistance = object distance
plane centre (origin)
9
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane
optical axis
= imagedistance = object distance
10
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Simple Camera with Lens ndash Distant
ldquothinrdquo lens
distantworld point
projection of world point
image plane(world point out
of focus)
optical axis
= imagedistance = object distance
11
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
= focallengthoflens = imagedistance = object distance
equiv limrarr
optical axis
12
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Simple Camera with Lens ndash Infinity focus
ldquothinrdquo lens
infinitely farworld point
projection of world point
image plane
= infin
$ = focallengthoflens = imagedistance = object distance
optical axis
1$ =
1 +
1ThinLensLaw
$
13
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Modern Camera
bull In practice the image plane is changed by moving lens elements rather than moving the sensorbull This is the ldquofocusingrdquo mechanism
Source dpreviewcom
14
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Focusing
ldquothinrdquo lens
world pointimage plane
optical axis
= imagedistance = object distance
bull Imagine slowly moving the image planebull What does the image of a fixed nearby world point look like
15
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
16
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Depth of Field
bull This effect is called ldquodepth of fieldrdquo in photography (DoF)bull Range of distances over
which image is in ldquoperfect focusrdquo
17
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Depth of Field
bull But why do we see more than just what is exactly at the distance we focused on
18
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Depth of Field
bull DoF = range of distances where blur lt 1 sensor pixelbull Things that affect DoFbull pixel sizebull aperture bull lens focal length
bull Cellphone camerabull wide-angle lens (short focal
length)bull need to fake DoF (portrait
mode)
19
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Modelling Defocus Blur
equiv blur circle diameter of scene pointrsquos image on sensor planeDoF equiv range of distance in scene where lt sensor pixel size 20
Hasinoff amp Kutulakos PAMI 10
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Portrait Mode Faking Depth of Field
21
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
22
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Aperture
bull The relative size of the area in which light is collected through the lensbull Typically adjustable with
aperture lsquobladesrsquobull You can tell how many aperture
blades a lens has from lens flarecopy Taylor Bennett23
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Aperture
bull Expressed as ltvaluegt (f-stop)bull eg this lens is 50mm and f18bull f18 is maximum aperture
bullegravemax = )+- asymp 278 mm
copy Taylor Bennett24
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Shutter Speed
bull The duration (∆) of the exposurebull How long we allow photons to
hit the sensorbull Often expressed as fractions of
a second (ie 11000s)
25
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Equal Exposures Aperture and DoFbull Photons prop ∆bull ie get correct exposure
with different aperture and exposure timesbull However get different DoF
bull uarr darr ∆ rArr small DoFbull darr uarr ∆ rArr large DoF
05s
2s
darr uarr ∆(small aperture long exposure)
uarr darr ∆(large aperture short exposure)
D
2D 26
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
ISO Film Speedamp Sensor Sensitivitybull The sensitivity of filmsensor
to lightbull Often expressed by ISO film
speed (ie ISO 400)bull For a given exposurebull High ISO egrave brighter imagebull High ISO egrave higher noise
bull In a digital camera translates to sensorrsquos signal gain setting
27
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
28
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
What do we see
bull We model filmsensors based on our own visual perceptionbull Everything on Earth has
evolved in the context of the sunrsquos spectral outputbull Digital sensors often have
wider spectral sensitivity and are restricted to visible (IR cut filters)
29
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
What is Colour
bull Rod cells which are very highly sensitive to photos used in dark No colour visionbull Cone cells 3 types have different spectral
sensitivity roughly correspond to ldquoRGBrdquo
30
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Color image acquisitionbull All sensor pixels have same
response curve ndash ie are monochromaticbull Typically each pixel is made
sensitive to one of R G or B by placing filters over individual pixelsbull Typical Bayer filter has 25 red
25 blue and 50 greenbull Full-colour images by
computationally filling in missing RGB ldquodemosaicingrdquo
31
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Cross-section of aCMOS Image SensorBack-illuminated structureAka back-side illuminated (BSI)CMOS sensor
1 Retina2 Nerve fibers3 Optic nerves4 Blind spot
Vertebrate vs Cephalopod32
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
RAW vs Developed Images
The color image before ldquodevelopingrdquo (linear RAW image)
33
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
RAW vs Developed Images
The color image before ldquodevelopingrdquo (contrast-enhanced)
34
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
RAW vs Developed Images
The color image after ldquodevelopingrdquo Demosaicing+Intensity mapping
35
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
36
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Digital Sensors Photons agrave Digital Value
bull Arriving photons cause photo-electrons (due to photoelectric effect)bull Charge accumulates as more photons hit the photo-diodebull After exposure time amplifier converts charge to measurable voltagebull This voltage is converted to digital reading by an A-to-D converter
(aka ldquodata numberrdquo or DN)
37
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Photo-electrons to Radiant Power (Flux)
Φ = $
amp ) +()) )
pixel footprint wavelength
incident spectral irradiance(photonss at given wavelength)
spatial response at collection site (unitless)
quantum device efficiency(electrons collected per incident photon at given wavelength)
(DN)
38
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Quantum Efficiency Curves
(Sony 2012)39
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Lighting Levels vs Average Photon Counts
(Assuming Q = 05 ( = 1m2 ∆ = 150 sec surface albedo = 05 aperture = 21)
Φ∆ =Cossairt et al ldquoWhen Does Computational Imaging Improve Performancerdquo
IEEE transactions on Image Processing (TIP) May 2012
Illuminanceirradiance
Φ Φ∆(DN)
40
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Digital Sensors Photons agrave Digital Value
Φ Φ∆ + amp
amp = black level non-photoelectric (ie electrons) current from photo diode( = saturation current maximum non-discarded current from photodiode) = amplifier gain electronsDN or ISO (see httpclarkvisioncomarticlesiso)
min(Φ∆ + amp ()min Φ∆ + amp (
)
black level saturationcurrent gain
DN = min Φ∆ + amp ()
Note DN obtained above has a linear relationship (up to saturation) with flux
(DN)
41
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Linear Images Donrsquot Look good
42
bull The human visual system (HVS) doesnrsquot have a linear response
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Gamma Correction
43
bull The human visual system (HVS) doesnrsquot have a linear responsebull DNs are passed
through a ldquogamma functionrdquo to compensate for HVSbull = ()(
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Digital Sensors Gamma Correction
Φ Φ∆ + amp min(Φ∆ + amp )min Φ∆ + amp
black level saturationcurrent gain
DN = min Φ∆ + amp
2 34 = 2( min Φ∆ + amp )gamma correction
This is also called the camera response function
(DN)
44
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Digital Sensors Camera Response Functions
Grossberg et al Modeling the Space of Camera Response Functions PAMI 200445
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
46
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
bull Scene points of interest are ldquoout of focusrdquobull not within the Dof
Defocus Blur
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
subject in focus subject out of focus
47
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Motion blur
bull Camera moves significantly during exposure timebull More likely withbull Long exposuresbull Long focal length
(zoom)
4 sec f11 (ISO 100) 4 sec f11 (ISO 100)
camera on tripod camera hand-held
48
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Pixel noise
4 sec f11 (ISO 100) 115 sec f11 (ISO 1600)
ideally-exposed photo under-exposed photo
49
bull Incorrect exposure not enough photons reaching sensorbull High ISO (gain)
causes noise
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Whatrsquos Going on in These Photoshttpspetapixelcom20141013math-behind-rolling-shutter-phenomenon
wwwsilent9com Joel Johnson50
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Rolling Shutter vs Global Shutter
httpsandoroxinstcomlearningviewarticlerolling-and-global-shutter
51
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Rolling Shutter Timing Diagram
httpswwwmatrix-visioncomglossariohtml
52
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Topic 1 The Camera
bull Pinhole lens-based cameras amp image blurbull Basic camera controlsbull Color image acquisitionbull Image formation from photons to digital numbersbull Key image artifactsbull Understanding image noise
53
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Digital Sensors Sources of Noise
Free electrons due to thermal energy
Depends on temperature measured in
electronssec independent of Φ∆
Photon arrival timing is random
Depends on total photon arrivals ie
Φ∆
Noise from readout
electronics
IndependentOfΦ∆
Amplifier A-to-D quantization
noise
Independentof Φ∆
(DN)
54
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Shot noise is Poisson distribution with mean Φ∆bull Poisson k events in ∆ mean + = -
12-
bull received photons = k = 3∆4 123∆4
bull Largest source of noise for high exposures
(DN)
55
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Sources of Noise Photon (aka Shot) Noise
Photon arrival timing is random
bull Forlargeenoughmean- (Φ∆1)Poisson(-) asymp Normal(9 = - lt = -)bull Can approximate with Normal distribution
for large Φ∆1httpwwwboostorgdoclibs1_55_0libsmathdochtmlmath_toolkitdist_refdistspoisson_disthtml
(DN)
56
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Sources of Noise Dark Current Noise
Free electrons due to thermal energy
bull Depends on temperaturebull Poisson distribution with mean ∆bull is thermal electron rate (electronss)
bull $ received thermal electrons = k = amp∆() +
amp∆
(DN)
57
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Sources of Noise Readout Noise
Noise from readout
electronics
bull Normal distribution with = 0 = ampbull Only depends on characteristics of electronics
(DN)
58
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Sources of Noise ADC amp Quantization Noise
Amplifier ADC quantization
noise
bull Normal distribution with = 0 = amp(bull Amplifier noise (depends on gainISO) is largest
source of noise for low exposures
(DN)
59
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Sources of Noise Putting it all Together
Free electrons due to thermal energy
Photon arrival timing is random
Noise from readout
electronics
Amplifier A-to-D quantization
noise
Poisson = Φ∆Large Φ∆
asymp Normal = 0 =
Normal( = 0 0 = 03)Poisson = 5∆Large D∆
asymp Normal = 0 =
Normal( = 0 0 = 0789)
(DN)
60
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Sources of Noise Eg Canon EOS 5D Mark 2
61httpwwwclarkvisioncom
$ + $() sdot log
01 234536247min 234536247 = log
$ + $() sdot
(DN)
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Putting It All Together
mean()= min() + + + - (variance()= + + - + () + 123 + 14563 sdot 83
mean(-9)= minlt=gtlt5gt variance(-9)= =gt
A +5gtA +
ltBCAA + 14563
62
(DN)
Photon term amp dark current term are additive Hasinoff et al CVPR2010
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Quantifying the Effect of Noise the SNR
bull Signal-to-noise ratio (SNR) = 10 logamp()+ -
01+2) -
mean(34)= minlt=gtltgt
A
variance(34)= =gt+ gt
+ ltDE
+ FGHI
63
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Quantifying the Effect of Noise Example
64Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Putting It All Together
65
bull Most common (but inaccurate) simplificationsbull Ignore photon + dark currentbull Ignore camera response function
mean()= min()+- (0 variance()= +
2 +-2 +
()4522 + 67-89
65
Hasinoff et al CVPR2010
Related Documents
