Mehdi Rezagholizadeh: Image Sensor Modeling: Color Measurement at Low Light Levels
Post on 11-Aug-2015
27 Views
Preview:
Transcript
Color Perception at Low Signal
to Noise Levels
By:
Mehdi REZAGHOLIZADEH
MCGILL UNIVERSITY
DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING
Supervisor:
Prof. James J. CLARK
Brief Overview
DECEMBER 2014
Statement of Problems
• Developing a Fast and Accurate Color
Constancy Method
• Color Appearance Modeling at Low Light
Levels
• Image Sensor Modeling and Color
Measurement at Low Light Levels
2
Definition of Color Constancy
• Discounting the illuminant effect on the color
of objects
• Color constancy is a great feature of our
visual system
3Computational Color Constancy
Importance of Color Constancy• Applications:
� Object Recognition
� Image Enhancement
� Robot Vision
� Object Tracking
� Photography and Film Industry
The last three cases are
real-time applications of color constancy.
4Computational Color Constancy
Problem of Color Constancy
5Computational Color Constancy
Problem of Color Constancy
• Image Formation Model:
�� = �� � � �, � � ��� � : the illuminant spectrum
�(�, ): the surface spectral reflectance function
at location .
�(�): the sensor spectral sensitivity
��: the sensor response of ith channel
6Computational Color Constancy
Problem of Color Constancy• The Transformation Imposed by a Change in Illumination:
�� = �� � � �, � � ��Givens:
�(�): the sensor spectral sensitivity
��: the sensor response of ith channel
Unknowns:
� � : the illuminant spectrum
�(�, ): the surface spectral reflectance function at location .
Spectral Color Constancy Approaches try to find the entire spectrum of the illuminant and the surface spectral reflectance function.
It is an ill-posed problem.
7Computational Color Constancy
Methods of Color Constancy
Computational Color Constancy
Spectral Methods
Non-spectral Methods
Static Methods
Gamut-Based
Learning-Based
8Computational Color Constancy
Non-Spectral Methods:• MAIN Objective of this problem is to obtain:
��� = �� �, � � ��It is equivalent to obtaining the sensor responses when � � = 1.
• Assuming that color of the illuminant can be estimated:
��� = �� � � � ��• The Transformation Imposed by an illuminant can be
obtained through an Over-simplification:
��� ≅ �����
= �� � � �, � � ���� � � � ��
9Computational Color Constancy, March. 2014
Non-Spectral Methods:
Over-simplification leads to:
- Estimating the illuminant color
rather than
- Estimating the entire spectrum of the illuminant �(�)• Corrective Transformation:
���������=
1��� 0 0
0 1���
0
0 0 1���
������
10Computational Color Constancy, March. 2014
Problem II: Color Appearance Modeling at Low Light Levels
• Color Appearance Model (CAM):
• An ideal color appearance model:
The output resembles human perception in all conditions including different light levels
• Lack of a good color appearance model
for low light conditions
Color Perception at Low Signal to Noise Levels 11
TransformTransformTristimulus
values (RGB)
Perceptual attributes of
color:
lightness, hue, chroma
Biophysical Background
• Our eye can work in three different modes:
1- Photopic condition (Luminance>5 cd/m2 )
2- Mesopic condition (0.005<Luminance<5 cd/m2 )
3- Scotopic condition (Luminance<0.005 cd/m2 )
• Photopic Condition: (High Light Levels)
Color Perception at Low Signal to Noise Levels 12
• Mesopic Condition: (Low Light Levels)• Scotopic Condition: (Very Low Light Levels)
Background and Preliminaries • Existing Models for Mesopic & Scotopic Vision:
� Modeling Blue Shift in Moonlit scenes [1]– Addresses scotopic vision by adding some blue to the initial image
– The output of this algorithm does not look natural and realistic
� Cao’s Model of Mesopic Vision [2]– It is a two stage model based on the gain control and cone opponent
mechanisms
– Model is fitted to the psychophysical experiment data
� iCAM06 Tone Compression Model for Mesopic Vision [3]– iCAM06 includes rod responses in a linear fashion
� Shin’s Color Appearance Model [4]– Boynton two-stage model is fitted to the behavioral experiment data
13
[1] S. M. Khan and S. N. Pattanaik, “Modeling blue shift in moonlit scenes by rod cone interaction,” Journal of VISION, vol. 4, no. 8,
2004.
[2] D. Cao, J. Pokorny, V. C. Smith, and A. J. Zele, “Rod contributions to color perception: linear with rod contrast," Vision research, vol.
48, no. 26, pp. 2586-2592, 2008.
[3] J. Kuang, G. M. Johnson, and M. D. Fairchild, “iCAM06: a rened image appearance model for HDR image rendering," Journal of
Visual Communication and Image Representation, vol. 18, no. 5, pp. 406 -414, 2007.
[4] J. Shin, N. Matsuki, H. Yaguchi, and S. Shioiri, “A color appearance model applicable in mesopic vision," Optical review, vol. 11, no.
4, pp. 272-278, 2004.
Physics & Color Perception
Color Perception at Low Signal to Noise Levels 14
Problem
• Lack of a good color appearance model (CAM) for the low light conditions
Physics
• The basic physical principles governing the probabilistic nature of color perception at low light levels
Analysis
• Photon Detection and Color Perception at low light levels
Proposed Method: Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• Under very low light conditions:
The photoreceptor responses more uncertain
• Hypothesis:
Color Perception at Low Signal to Noise Levels 15
Visual Processing Center reconstructs a part of the information being lost in the projection of light spectra into the space
of photoreceptor responses
Proposed Method: Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• The spectral theory of color perception [Clark and Skaff,
2009]:
�Provides a tool to address the issues of uncertain
measurements
� Estimates the spectral power distributions corresponding
to these uncertain measurements.
Color Perception at Low Signal to Noise Levels 16
Proposed Method: Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• Spectral Model of Mesopic Vision
� Clark and Skaff proposed a spectral model for color perceptionwhich is valid for photopic conditions
� During the mesopic condition, both cones and rods contribute to thevision
� Given the measurement vector �, we can model the rod intrusioninto the perception as follows:
�� = � � ��� � + ���� � ! � �� + "# $ ∈ {',(, �}
- +,(-) and +.(-): cone and rod spectral sensitivity functions respectively
- /(-): normalized mesopic spectral power distribution
- �: intensity factor
- 0 [0, 1]: a parameter which determines relative rod intrusion
- 1 = [�3 �4 ��]: a diagonal matrix specifies the relative contribution of rod response to each conechannel.
- ": additive noise
17Color Perception at Low Signal to Noise Levels
Proposed Method: Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
• Spectral Model of Mesopic Vision
� Given the measurement vector �, we can model the rod intrusioninto the perception as follows:
�� = � � ��� � + ���� � ! � �� + "# $ ∈ {',(, �}
18
19
Proposed Method: Maximum Entropy Spectral Modeling Approach to the
Low Light Levels Color Appearance Modeling
� An exponential family is employed to estimate ! � :
!̂ � = exp(< � � , ; > −>(;))� � = �� � + �1� �;: parameter vector which should be estimated
>(;): normalizing function
� Parameters can be estimated as follows:
;? = minC { DE − D FG(DE − D)} − HI(;)}D = �/� normalized measurement
I(;): entropy function corresponding to !̂(�)A: positive definite matrix
K: regularization factor
• Spectral Model of Mesopic Vision
Color Perception at Low Signal to Noise Levels
Results:
• Simulation of Munsell patches
– surrounded by a white background
– viewed under different light levels from scotopic to
photopic.
Color Perception at Low Signal to Noise Levels 20
Image Sensor Modeling:
Color Measurement at Low Light Levels
By:
Mehdi REZAGHOLIZADEH
James J. CLARK
MCGILL UNIVERSITY
November 2014
22nd Color and Imaging Conference
The Presentation Outline:
Image Sensor Modeling: Color Measurement at Low Light Levels 22
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
Introduction:Motivation
Importance of Studying low light levels:
• Color Measurement at low light level becomes more uncertain due to the low signal to noise ratio
• Most of the theories, measures, models and methods in color science are developed for high intensities
• The quality of the human color vision at low light levels is much better than existing handy cameras
23Image Sensor Modeling: Color Measurement at Low Light LevelsKirk, Adam G., and James F. O'Brien. "Perceptually based tone
mapping for low-light conditions." ACM Trans. Graph. 30.4 (2011): 42.
Introduction:Statement of the Problem
24
Problem:
• What is the impact of noise at low lightlevels on the color measurements ofimaging devices?
Image Sensor Modeling: Color Measurement at Low Light Levels
Applications of the Study:
� Spectral Imaging
� Image Processing
� Low Light Photography
� Characterizing the Noise of Image Sensors
�Developing Denoising and Enhancement Algorithms
� Photon Limited Imaging (biosensors, astronomy, etc)
25Image Sensor Modeling: Color Measurement at Low Light Levels
What Next…
26
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
Image Sensor Modeling: Color Measurement at Low Light Levels
Physical Background
27
• Simulating the effect of Photon Noise (given the high
intensity description of the light):
• For each bin:
L M(��), N � O PQ RSTU VQW!
0
2
4
6
8
104
00
42
0
44
0
46
0
48
0
50
0
52
0
54
0
56
0
58
0
60
0
62
0
64
0
66
0
68
0
70
0
Av
era
ge
Ph
oto
n C
ou
nt
: g
(λ)
Wavelength (nm)
ᵟ
Image Sensor Modeling: Color Measurement at Low Light Levels
Physical Background
28
• A set of Poisson distributions (one for each bin)
characterizes the targeted light.
• To estimate the spectral radiance at a lower intensity:
• The estimated quantal spectral radiance:
'YZ[ �� = \]Z(��)^
0
10
g(λ
)
Wavelength (nm)
Image Sensor Modeling: Color Measurement at Low Light Levels
_ = lowintensityhighintensity Draw samples from
hijk l m n -j op qrl -j ~hijk l m n -j��
Simulation:
How Does Spectral Power Distribution Change with Intensity?
• The estimated spectral power distribution at
different intensities. _ = 5 m 10u��1vww
^ � 5Nxw � 0.2{|}
Color Perception at Low Signal to Noise Levels 29
Simulation:
How Does Spectral Power Distribution Change with Intensity?
• The estimated spectral power distribution at
different intensities.
Color Perception at Low Signal to Noise Levels 30
_ � 5 m 10u��1vww^ � 5Nxw � 0.2{|}
Simulation:
How Does Spectral Power Distribution Change with Intensity?
• The estimated spectral power distribution at
different intensities.
Color Perception at Low Signal to Noise Levels 31
_ � 5 m 10u�~1vww^ � 5Nxw � 0.2{|}
Image Sensor Modeling
32
• Image sensor pipeline (for a single channel):
Noise Model
Photon Shot Noise
Dark Current Noise
Read Noise
Quantization Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling
33
• Image sensor pipeline (for a single channel):
Noise Model
Photon Shot Noise
Dark Current Noise
Read Noise
Quantization Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling
34
• Image sensor pipeline (for a single channel):
Noise Model
Photon Shot Noise
Dark Current Noise
Read Noise
Quantization Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling
35
• Image sensor pipeline (for a single channel):
Noise Model
Photon Shot Noise
Dark Current Noise
Read Noise
Quantization Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling
36
• Image sensor pipeline (for a single channel):
Noise Model
Photon Shot Noise
Dark Current Noise
Read Noise
Quantization Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling: Noise Model
37
• Variations in the number of emitted photons
• Can be modeled by a Poisson DistributionPhoton Shot Noise
• The current produced inside the image sensor
• ��� �� (�, �)~L�${( ��� �� �)Dark Current
Noise
• The noise in the readout circuit
• � S��~�(0, � S��)Read Noise
• The error introduced in the quantization stepQuantization
Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling: Pixel Measurement Model
38
Output of the Image Sensor
• �� �, � � \�ST m ���� � m �'YZ[ �, �, � �S� � �� � � m ��� �� �, ��
Measured Voltage
• �] � �, � � �� �, � � � S�� �, ��
Raw Output
Image
• �� �, � � q m �r� �, �� ��
Image Sensor Modeling: Color Measurement at Low Light Levels
What Next…
39
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Dataset and Preparation
� Dataset: “A data set for Color Research”
� By: Barnard et al.
� Includes:
- The Sony DXC-930 sensor sensitivity curves
- The spectra and color measurements of 598 color samples
made by the Sony camera
40[1] K. Barnard, L. Martin, B. Funt, and A. Coath, “A data set for color research,” Color Research & Application, vol. 27, no. 3, pp.
147-151, 2002.
Experiments & Results:Dataset and Preparation
� Preparation:
− 20 samples from the 598 color measurements are selected
for our experiments
− By scaling the initial spectra, the luminance values of color
samples are set to 100
41
− The luminance of each color
sample is modified by applying
the intensity factor, F.
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Caveats
42
• Temperature is assumed constant, hence the dark noise
parameters are fixed during the experiments.
• Noise model is additive
• The Sony DXC-930 camera is nearly linear for most of its
range, provided it is used with gamma disabled.
• Raw output images are considered for our analysis.
• The effects of reset noise, photodetector response
nonuniformity (PRNU), dark signal nonuniformity(DSNU) are
considered negligible.
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Analysis
43
Experiments:
Scenario I: Ideal Image Sensor
Scenario II: Effects of Dark Current
Scenario III: Real Image Sensor Model
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Scenario I: Ideal Image Sensor
44
Assumptions:
• Sensor is ideal (no internal noise in the model)
• Photon shot noise may corrupt the measurements
• log _ ∈ &0, =7,=8,=9,=10,=11,=12,=13,=14)
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Scenario I: Ideal Image Sensor
45Image Sensor Modeling: Color Measurement at Low Light Levels
Chromaticity of Measured Samples at
Different Light Levels
Magnified Result of the Data Point Indexed 3
at Different Intensity Factors
Experiments & Results:Scenario II: Effects of Dark Current
46
Assumptions:
• Only photon shot noise and dark noise may corrupt the measurements
• Only boundary color patches are used (index: 1-13)
• _ ∈ &1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001)
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Scenario II: Effects of Dark Current
47Image Sensor Modeling: Color Measurement at Low Light Levels
Chromaticity of Measured Samples at
Different Light Levels
Magnified Result of the Data Point Indexed 3
at Different Intensity Factors
Experiments & Results:Scenario III: Real Image Sensor Model
48
Assumptions:
• A model of real image sensor is considered
• _ ∈ &1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001)
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Scenario III: Real Image Sensor Model
49Image Sensor Modeling: Color Measurement at Low Light Levels
Chromaticity of Measured Samples at
Different Light Levels
Magnified Result of the Data Point Indexed 3
at Different Intensity Factors
Experiments & Results:Comparing the Three Scenarios
50Image Sensor Modeling: Color Measurement at Low Light Levels
Scenario I Scenario II Scenario III
Experiments & Results:Comparing the Three Scenarios
51Image Sensor Modeling: Color Measurement at Low Light Levels
Scenario I Scenario II Scenario III
What Next…
52
Conclusion
Experiments and Results:
Preparation Caveats Analysis Experiment Scenarios
Solution: Image Sensor Modeling
Physical Background Noise Model Pixel Measurement Model
Introduction:
Motivation Statement of the Problem: Color Measurement at Low Light Levels
Image Sensor Modeling: Color Measurement at Low Light Levels
Conclusion
53
− Photon noise
− read noise
− quantization error
The physical limitation imposed by the photon noise
Dark current dominates the other sensor noise types
in the image sensor
Image Sensor Modeling: Color Measurement at Low Light Levels
Uncertain measurements distributed
around the noise free measurements
Dark
current
noise
dynamic
effects
on color
measur
ements
Shifting
chromaticities
towards the
camera black
point
1
2
3
4
Prevents stable measuring of color (even for an ideal image sensor)
Image Sensor Modeling: Color Measurement at Low Light Levels 54
Thank You for Your Attention!
Questions…
Image Sensor Modeling
55
• Image sensor pipeline (for a single channel):
Noise Model
Photon Shot Noise
Dark Current Noise
Read Noise
Quantization Noise
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling: Pixel Measurement Model
56
Output of the
Image Sensor
• \�ST: conversion gain (volts/|u)
• ����: saturation function of the sensor
• �S� � : the quantum efficiency function of the sensor
• 'YZ[: the quantal radiance at the intensity factor F (photons/sec/x�/sr/nm)
�� �, � � \�ST m ���� � m �'YZ[ �, �, � �S� � �� � � m ��� �� �, ��
Image Sensor Modeling: Color Measurement at Low Light Levels
Image Sensor Modeling: Pixel Measurement Model
57
Output of the
Image Sensor
• �� �, � �\�ST m ���� � m �'YZ[ �, �, � �S� � �� � � m ��� �� �, ��
Measured
Voltage
• �r� �,� � �� �,� � p.��� �, ��
Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Scenario I: Ideal Image Sensor
58Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Scenario II: Effects of Dark Current
59Image Sensor Modeling: Color Measurement at Low Light Levels
Experiments & Results:Scenario III: Real Image Sensor Model
60Image Sensor Modeling: Color Measurement at Low Light Levels
top related