1 Digital Camera and Scanner Performance Evaluation: Standards and Software Peter D. Burns & Don Williams Updated course on imaging performance measurement methods for digital image capture devices and systems. • several ISO measurement protocols for camera resolution, tone- transfer, noise, etc. • underlying sources of variability in system performance • measurement for emerging standards • how standards can be adapted to evaluate capture devices • evaluation of capture devices from cell phone cameras to scientific detectors • required elements of software tools We will also address various challenges to reliable evaluation and system comparison using actual test data and software tools. SPIE/IS&T Electronic Imaging Symposium, 2014 Course SC807 2 Feb. 2014, 8:30 AM-12:30 PM Digital Camera and Scanner Performance Evaluation: Standards and Software, Peter Burns & Don Williams OBJECTIVES This course will enable you to: • Understand the difference between imaging performance and image quality • Interpret and apply the different flavors of each ISO performance method • Identify sources of system variability, and understand resulting measurement error • Distill information-rich ISO metrics into single measures for quality assurance • Adapt standard methods for use in factory testing • Select software elements (with Matlab examples) for performance evaluation programs • Be aware of upcoming standard measurement protocols Burns/Williams, EI2014 2 Course SC807
41
Embed
Digital Camera and Scanner Performance Evaluation: Standards …burnsdigitalimaging.com/teaching/EI2014/EI_SC807_Burns.pdf · 2014-02-05 · Digital Camera and Scanner Perf ormance
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Digital Camera and Scanner Performance Evaluation: Standards and Software
Peter D. Burns & Don Williams
Updated course on imaging performance measurement methods for digital image capture devices and systems.
• several ISO measurement protocols for camera resolution, tone-transfer, noise, etc.
• underlying sources of variability in system performance
• measurement for emerging standards
• how standards can be adapted to evaluate capture devices
• evaluation of capture devices from cell phone cameras to scientific detectors
• required elements of software tools
We will also address various challenges to reliable evaluation and system comparison using actual test data and software tools.
– Image Noise (10 pages)• Random, fixed pattern, two-dimensional, streaks• Standard noise performance measurements and testing
Break3. Why results vary and ways to control variation
– Test plans– Requirements for evaluation tools
4. Standards Rodeo
4
What makes a digital image ?you push the button and …..
Burns/Williams, EI2014: Introduction
imageformation
capture processing displayhuman-to-machine
interface
intended renderingsharpeningde-noisingcompression
detector sizeshutteringreadout
renderingsize
environmentshutter lagcamera motion
opticsfocus
Object/scene
lightsource
Imaging Performance Metrics quantify how an imaging system or component acts on, modifies, or limits the effective optical characteristics of an input scene.
Once digitally captured, the image is encoded as N-bit (M channel) digital files
3
Burns/Williams, EI2014 5
Digital Capture:Why measure imaging performance?
Standard testing, e.g., ISO– Aimed at hardware evaluation or benchmarking– Identify the influence of image processing, e.g., sharpening– Marketing leverage– Quality control and industry compliance
System testing– Actual product usage– Robustness testing – image processing as used, or in default signal
path, e.g., sRGB metric – Scanner as instrument
Metadata population– Enablement of enhanced product features and ease of use
Image Quality – Required as input for image quality
Burns/Williams, EI2014: Introduction
6
A shopping list of digital camera imaging (capture) performance features *
__________________________* From : “Proposal for a Standard to Test Mobile Phone Cameras”
High Priority• OECF – tone reproduction• White balancing • Dynamic range (related scene contrast) • Used digital values • Noise, signal to noise ratio • Resolution (limiting resolution center, corner) • Sharpness
Recommended• Spatial distortion • Shading / vignetting• Chromatic aberration • Color reproduction • Shutter lag • Aliasing artifacts • Compression rates • Exposure and exposure time accuracy
May be tested if available and applicable. • Optical stabilization • Auto focus accuracy and constancy • Metadata (Exif, IPTC) • Watermarking • Spectral sensitivities • Bit depth of raw data • Detailed noise analysis • Color resolution (quantization)
Burns/Williams, EI2014: Introduction
4
7
How is imaging performance measured?
• Signal
Any response that provides valued information
• Noise
Any response that detracts from a desired signal
• Signal-to-Noise ratio
A good-to-bad proportionality measure
Burns/Williams, EI2014: Introduction
8
Vocabulary – Image Speak- staying dry in a storm of idioms -
Resolution
Exposure
Sharpening
Delta E
White Balance
Dynamic Range
GammaExposure
Depth offield
Flare
Shading
Noise
Geometric Distortion
Wobble
Aliasing
Signal Noise
Lateral ColorError Sharpness
Acutance
Burns/Williams, EI2014: Introduction
5
9
Imaging Performance vs. Image Quality
Image Quality: Task dependent, but can almost always be correlated to some weighted combination of imaging performance metrics.
• Aerial Reconnaissance – high SFR and low noise• Health Imaging – tone control, low noise, SFR dependent on task• Consumer Imaging – color and tone, moderate SFR.
Imaging performance specifies the underlying design needed to deliver perceived image quality.
a) Peak-to-absolute zero – The differencebetween CV of interest and zero.
b) Peak-to-peak – Relative differencebetween the CV of interest and thelowest effective CV.
c) Incremental slope – The differencebetween the CV of interest and the CV ofan incrementally small change in density. The slope of the OECF curve for any given density. Indicates how well a device
Hsat = Minimum focal plane exposure that yields the highest unclipped output signal( lux-seconds)
• Appropriate for controlled illumination environments where the best possible image quality is desired
• Intended to prevent clipping artifacts associated with image sensor saturation
2 – signal – speed
0
50
100
150
200
250
0.000
aver
age
gree
n co
unt v
alue
( 8
bit)
Exposure at detector
Correlates the highest exposure to the best SNR and the best image quality for a given camera. Does not mean that the same Ssat will yield equivalent image quality between cameras
• Currently what we think most camera manufacturers are using to report speed
• Number of pixels • Sampling frequency – dpi, ppi• Limiting resolution, Resolving power*• Spatial Frequency Response – SFR*• Modulation Transfer Function – MTF*
*Related Standards – ISO 16067-1, 16067-2, 12233, & 15529 all use these metrics.
2 – signal – resolution
Resolution and Spatial Frequency Response
What is it?The ability of an imaging component or system to distinguishfinely spaced detail : ResolutionGenerally, the ability to maintain the relative contrast of finely spaced detail. Spatial Frequency Response
1- Rayleigh Criteria - Find frequency of first 10% SFR occurrence in both vertical and horizontal directions. ( clip at 0.5 cycles/pixel)
2- Efficiency- Normalize each by digital limit of 0.5 cycles/pixel.3- Take product of vertical and horizontal components.4- Multiply by camera’s finished file size
ISO SFR Method• Based on edge-gradient method• Intended for digital capture devices• Alignment immunity• Localized features• Open source software• Extendable to any angular direction• Ease of target generation; several imaging mechanisms or steps can be
described in terms of spread functionsMTF and SFR
Spread functions and MTF indicate the imaging or transfer of signal detail and sharpness
A number of derivative metrics can be calculated from SFR for process control & monitoring as well as image quality.
For quality-assurance control charts, a good measure of image resolution is the SFR-based sampling efficiency.
1. Transform the captured image array of the target field to one encoded as proportional to luminance.
2. Compute the power-spectral density as the square of the amplitude of the two-dimensional DFT of the array.
3. Divide this array, frequency-by-frequency, by the modeled spectrum for the specific target to yield a two-dimensional array as the square of the effective MTF.
4. Compute the square-root, frequency-by-frequency, of this vector.
5. Compute the one-dimensional MTF vector by a radial-average of this array
• Noise is a general term applied to any response that detracts from a desired signal. Usually, errors or unwanted fluctuations in images. Given that it can have many sources, it can take several forms.
• Random noise due to film grain or low exposure to a detector. Often looks like “salt and pepper” noise.
• Often reported as an RMS number (standard deviation)
The extent of energy over which a digital capture device can reliably detect signals: reported as either a normalized ratio (xxx:1) or in equivalent optical density units, , R= reflectance.
• ISO 21550 – Dynamic Range for Digital Film Scanners• ISO 15739 – Noise Measurement for Electronic Still Pictures Cameras
Signal Detection → OECF
The mean (average) change in count level associated with a particular input density level.
Reliability of Detection → Noise
The root mean square (std. dev.) count level associated with a particular input density level
Measurements usually require some level of both accuracy and precision.
– Accuracy: average error from a standard
– Precision: variability about the average readingIs your watch set to the correct time?
Factors that influence measurements
– Test target feature location estimation
– Signal processing
– Signal quantization
– Spatial sampling
– Image noise
XXXX
X bias
X
X
X X
X
variation
Burns/Williams, EI2014: Performance Variation
58
Variation
Actual performancee.g., Optical quality across the image field, or with lens zoom position
Measurement errorEstimation of sample statisticse.g. due to signal quantization, sampling
Sampling:variation in estimating parameters (statistics) from test images
Source of imaging performance variability for a digital camera?• Scene exposure.• Image compression parameters.• Camera shake.
Burns/Williams, EI2014: Performance Variation
30
59
Measurement error example
• Simplest way to quantify measurement variation is by repeating the analysis• This edge SFR example is for one test file and one edge, varying the
location of the region of interest, N = 10
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Frequency, cy/pixel
SF
R
Single file, edge (N=10)
ave
+1 std
-1 stdmax
min
Burns/Williams, EI2014: Performance Variation
60
ISO Protocols vs. Specifications:
ISO Imaging Performance provide protocols on how and what to measure with respect to imaging performance ( e.g. tone, noise, SNR, Resolution, etc.) but …..
The acceptable levels and tolerances for each of those is left to the user because they are largely use-case dependent. Resolution and color aims and tolerances could be quite different whether one is doing cell phone vs. portrait photography
Useful features in software for imaging performance evaluation• Image metadata (header and tag) reading.• Use with multi-attribute test targets.• Testing of results against specified tolerance values.
cycles/mm, cy/mm, mm-1 - Engineering, film photography
line-pairs/mm - Graphics
1 -
2 - 1 line width
1 line pair
39
Burns/Williams, EI2014 77
Frequency Unit Conversion Table
LW/PH LP/mm L/mm Cycles/mm Cycles/pixel
LW/PH1
⁄ [ 2 picture height (mm)]
⁄ picture height (mm)
⁄ [ 2 picture height (mm)]
⁄ [ 2 # vert. pixels ]
LP/mm [2 picture
height (mm)]
1 2.0 1 pixel pitch (mm)
L/mm picture height
(mm) 0.5 1 0.5
[pixel pitch (mm) ⁄2]
Cycles/mm [2 picture
height (mm)]
1 2.0 1 pixel pitch (mm)
Cycles/pixel [2 # vert.
pixels] ⁄ pixel pitch
(mm)
[2 ⁄ pixel pitch (mm)]
⁄ pixel pitch (mm) 1
LW/PH = Line width per picture heightLP/mm = Line pairs per millimetreL/mm = Lines per millimetreTo convert from left column units to top row units, use operation at their row/column intersection.(e.g., 5 LP/mm 2.0 = 10.0 L/mm )
Burns/Williams, EI2014 78
Appendix IV: ISO 12233, SFR and MTF
The ISO standard for measuring digital still camera resolutions specified a particular method, based on a modified form of edge gradient analysis (EGA) The ISO standard, however, refers to the camera resolution metric as
Spatial Frequency Response (SFR), rather than an MTF There are three basic reasons for this
– Established photographic standards measure MTFs using other methods (sine waves), and the results often differ from EGA results
– The term MTF has been associated with the measurement of systems or subsystems* for which it (approximately) uniquely describes the signal transfer from input to output). For many electronic imaging systems, however, the results will vary with system and image conditions.
– EGA measurement of a system MTF requires compensation for the input target edge characteristics. While this is possible, the original ISO standard did not require it.
___________________
* Linear systems, or those with combinations of linear and point-wise nonlinear subsystems
40
Burns/Williams, EI2014 79
Appendix V: Limiting Resolution Estimation from SFR
EvaluateSFR
ii SFRf ,Find criticalfrequency
pp SFRf where,
InterpolateSFR
'whereSFR
test file,critical value, f
EvaluateSFR
ii SFRf ,Find criticalfrequency
pp SFRf where,
InterpolateSFR
'whereSFR
test file,critical value, f
Burns/Williams, EI2014 80
Two dimensional example
Example of two-dimensional symmetric SFR and corresponding 50% and 10% response contours
41
Burns/Williams, EI2014 81
Appendix VI: ISO Camera Resolution Efficiency Rating Proposal
Electronic still picture cameras are normally marketed using a “megapixel” value, which normally provides thenumber of effective pixels of the image sensor. This can be confusing or misleading, since it is a value relatedto the number of addressable photoelements on the image sensor, and is not a based on any type of resolutionmeasurement.
There is interest in developing a single-value resolution numeric for reporting the measured resolution of electronicstill picture cameras. One proposed metric is the “resolution efficiency rating.”
1. Determine the maximum resolution in LW/PH for the horizontal (RH), vertical (RV), and +/- 45° (R+45 , R-45 )directions.2. Calculate individual directional efficiencies ( EH , EV , E+45 , E+45 ) by normalizing the maximum resolutions of
item #1 by the captured image’s picture height. If any normalized value is greater than 1.0, assign that value to 1.0.3. Combine E+45 and E-45 efficiencies into an equally weighted average diagonal value ED. 4. Calculate the resolution efficiency rating (ER) as the product of 100, ED and the average of EH and EV.
Resolution Efficiency Rating ER = 100 ( ED (EH + EV) /2 )
Example using the CIPA tool: 2048 pixel high x 3072 pixel wide, 6.0 MPixel camera file