Digital Camera and Scanner Performance Evaluation: Standards …burnsdigitalimaging.com/teaching/EI2014/EI_SC807_Burns.pdf · 2014-02-05 · Digital Camera and Scanner Perf ormance
Post on 02-Aug-2020
3 Views
Preview:
Transcript
1
Digital Camera and Scanner Performance Evaluation: Standards and Software
Peter D. Burns & Don Williams
Updated course on imaging performance measurement methods for digital image capture devices and systems.
• several ISO measurement protocols for camera resolution, tone-transfer, noise, etc.
• underlying sources of variability in system performance
• measurement for emerging standards
• how standards can be adapted to evaluate capture devices
• evaluation of capture devices from cell phone cameras to scientific detectors
• required elements of software tools
We will also address various challenges to reliable evaluation and system comparison using actual test data and software tools.
SPIE/IS&T Electronic Imaging Symposium, 2014Course SC807
2 Feb. 2014, 8:30 AM-12:30 PM
Digital Camera and Scanner Performance Evaluation: Standards and Software,
Peter Burns & Don Williams
OBJECTIVES
This course will enable you to:
• Understand the difference between imaging performance and image quality
• Interpret and apply the different flavors of each ISO performance method
• Identify sources of system variability, and understand resulting measurement error
• Distill information-rich ISO metrics into single measures for quality assurance
• Adapt standard methods for use in factory testing
• Select software elements (with Matlab examples) for performance evaluation programs
• Be aware of upcoming standard measurement protocols
Burns/Williams, EI2014 2
Course SC807
2
Burns/Williams, EI2014 3
Course Outline
1. Introduction– Measurement and sources of variability
2. Imaging Performance Measurement– Signal
• Opto-Electronic Conversion Function (OECF) • White Balance
– Color Evaluation• SFR/MTF & Summary measure: • Resolution, Acutance, Sampling Efficiency
– Image Noise (10 pages)• Random, fixed pattern, two-dimensional, streaks• Standard noise performance measurements and testing
Break3. Why results vary and ways to control variation
– Test plans– Requirements for evaluation tools
4. Standards Rodeo
4
What makes a digital image ?you push the button and …..
Burns/Williams, EI2014: Introduction
imageformation
capture processing displayhuman-to-machine
interface
intended renderingsharpeningde-noisingcompression
detector sizeshutteringreadout
renderingsize
environmentshutter lagcamera motion
opticsfocus
Object/scene
lightsource
Imaging Performance Metrics quantify how an imaging system or component acts on, modifies, or limits the effective optical characteristics of an input scene.
Once digitally captured, the image is encoded as N-bit (M channel) digital files
3
Burns/Williams, EI2014 5
Digital Capture:Why measure imaging performance?
Standard testing, e.g., ISO– Aimed at hardware evaluation or benchmarking– Identify the influence of image processing, e.g., sharpening– Marketing leverage– Quality control and industry compliance
System testing– Actual product usage– Robustness testing – image processing as used, or in default signal
path, e.g., sRGB metric – Scanner as instrument
Metadata population– Enablement of enhanced product features and ease of use
Image Quality – Required as input for image quality
Burns/Williams, EI2014: Introduction
6
A shopping list of digital camera imaging (capture) performance features *
__________________________* From : “Proposal for a Standard to Test Mobile Phone Cameras”
Dietmar Wueller, Image Engineering, http://digitalkamera.image-engineering.de/
High Priority• OECF – tone reproduction• White balancing • Dynamic range (related scene contrast) • Used digital values • Noise, signal to noise ratio • Resolution (limiting resolution center, corner) • Sharpness
Recommended• Spatial distortion • Shading / vignetting• Chromatic aberration • Color reproduction • Shutter lag • Aliasing artifacts • Compression rates • Exposure and exposure time accuracy
and constancy • ISO speed
Optional• Detailed macro mode testing (shortest shooting distance, max. scale, distortion)
• Flash capabilities (uniformity• Image frequency • Video capabilities
(pixel count, resolution, frame rate, low light behavior)
May be tested if available and applicable. • Optical stabilization • Auto focus accuracy and constancy • Metadata (Exif, IPTC) • Watermarking • Spectral sensitivities • Bit depth of raw data • Detailed noise analysis • Color resolution (quantization)
Burns/Williams, EI2014: Introduction
4
7
How is imaging performance measured?
• Signal
Any response that provides valued information
• Noise
Any response that detracts from a desired signal
• Signal-to-Noise ratio
A good-to-bad proportionality measure
Burns/Williams, EI2014: Introduction
8
Vocabulary – Image Speak- staying dry in a storm of idioms -
Resolution
Exposure
Sharpening
Delta E
White Balance
Dynamic Range
GammaExposure
Depth offield
Flare
Shading
Noise
Geometric Distortion
Wobble
Aliasing
Signal Noise
Lateral ColorError Sharpness
Acutance
Burns/Williams, EI2014: Introduction
5
9
Imaging Performance vs. Image Quality
Image Quality: Task dependent, but can almost always be correlated to some weighted combination of imaging performance metrics.
• Aerial Reconnaissance – high SFR and low noise• Health Imaging – tone control, low noise, SFR dependent on task• Consumer Imaging – color and tone, moderate SFR.
Imaging performance specifies the underlying design needed to deliver perceived image quality.
Performance Metrics
Signal
OECF – tone, colorSFR / MTF
Noise
Total RMS noiseStructured noise
Artifacts
Quality Metrics
Subjective Correlates
SharpnessGraininess
ColorfulnessOver sharpness
Objective Utility Correlates
NIIRSBriggs scores
Resolving powerOCR detection
Burns/Williams, EI2014: Introduction
Burns/Williams, EI2014: Imaging Performance Measurement 10
SIGNAL
• Opto-Electronic Conversion Function (OECF)
• Spatial Frequency Response (SFR)
6
Burns/Williams, EI2014 11
Signal – Metrics we’ll discuss
OECF
- Speed, Exposure
- White Balance
- Color Encoding
Spatial Frequency Response (SFR)
- Sampling Frequency
- Resolution, Sampling Efficiency
- Sharpening, Acutance
Burns/Williams, EI2014: Imaging Performance Measurement
Burns/Williams, EI2014 12
Large Area Image Capture – OECF*
• Large area image capture behavior is measured by OECF• The conversion relationship between optical reflectances (gray levels) of a
source object and corresponding electronic (digital) count values in a digitized image file; a luminance-to-signal mapping relation.
Uniformstepsused
Input Signal ( stimulus)
Out
put
Sig
nal
(res
pons
e)
*Electro-optical Conversion Function
Burns/Williams, EI2014 Imaging Performance Measurement
7
13
0
50
100
150
200
250
0.000 0.200 0.400 0.600 0.800 1.000
neutral ( gray) reflectance
aver
age
gree
n co
unt v
alue
( 8
bit)
LightLow density
Darkhigh density
• Important link between the physical characteristics of the source object and its digital image file
• A prerequisite for facilitating image reproduction and image quality and color management tasks.
Some important items to keep in mind on OECF
*ISO 16067-1,16067-2, 21550, 12233, 15739, 14524
• Allows performance evaluation in a common “input referred” metric* for other metrics like resolution, dynamic range
• Measures signal ‘encoding’
• There is no good or bad OECF….but characteristics like clipping and strong curvatures with multiple inflection points should be avoided
Burns/Williams, EI2014: Imaging Performance Measurement
Burns/Williams, EI2014 14
Some common input/output OECF variables
Stimulus ( Input )
PhotonsReflectance
DensityExposure ( Lux-seconds)
Log exposureLuminance ( L*)
Response (Output)
Electrons, Current8 bit Count value (CV)
16 bit Count value (CV)Log CV
Burns/Williams, EI2014: Imaging Performance Measurement
8
15
How OECF Measures Signal
Some ways signal is measured:
a) Peak-to-absolute zero – The differencebetween CV of interest and zero.
b) Peak-to-peak – Relative differencebetween the CV of interest and thelowest effective CV.
c) Incremental slope – The differencebetween the CV of interest and the CV ofan incrementally small change in density. The slope of the OECF curve for any given density. Indicates how well a device
can detect small density differences. (ISO)
OECF for Film Scanner
0
50
100
150
200
250
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
Density
CV
a) b)
c) slope
Burns/Williams, EI2014: Imaging Performance Measurement
16
Camera SpeedISO 12232
2 – signal - speed
• Exposure Index (EI)
• Saturation-based speed (Ssat)
• Noise-based speed (Snoise)
Camera speed is intended to indicate the exposure range (or the lower exposure limit) over which a camera
can deliver a useful digital image.
Burns/Williams, EI2014: Imaging Performance Measurement
9
17
Saturation-Based Speed, Ssat
Ssat= 78 / Hsatwhere
Hsat = Minimum focal plane exposure that yields the highest unclipped output signal( lux-seconds)
• Appropriate for controlled illumination environments where the best possible image quality is desired
• Intended to prevent clipping artifacts associated with image sensor saturation
2 – signal – speed
0
50
100
150
200
250
0.000
aver
age
gree
n co
unt v
alue
( 8
bit)
Exposure at detector
Correlates the highest exposure to the best SNR and the best image quality for a given camera. Does not mean that the same Ssat will yield equivalent image quality between cameras
• Currently what we think most camera manufacturers are using to report speed
Burns/Williams, EI2014: Imaging Performance Measurement
18
Snoisex= 10 / Hs/nx
where Hs/nx= exposure which yields a camera
incremental SNR of x=40 ( excellent ) or x=10 ( acceptable ).
Based on an objective correlation tosubjective judgments of noise level acceptability.
Noise-based Speed
Currently, the best speed based path for comparing image quality equivalence across cameras
Burns/Williams, EI2014: Imaging Performance Measurement
10
19
Color MetrologyJust a three channel OECF ?
• Keep the neutrals neutral. 85% of good color imaging is maintaining the neutrality of gray patches –white balance
• Color capture easier to manage for scanners because the device illumination is known.
.Burns/Williams, EI2014: Imaging Performance Measurement
20
Spatial Frequency Response (SFR)
NEXT
Burns/Williams, EI2014: Imaging Performance Measurement
11
21
…and what are all these related terms?
• Number of pixels • Sampling frequency – dpi, ppi• Limiting resolution, Resolving power*• Spatial Frequency Response – SFR*• Modulation Transfer Function – MTF*
*Related Standards – ISO 16067-1, 16067-2, 12233, & 15529 all use these metrics.
2 – signal – resolution
Resolution and Spatial Frequency Response
What is it?The ability of an imaging component or system to distinguishfinely spaced detail : ResolutionGenerally, the ability to maintain the relative contrast of finely spaced detail. Spatial Frequency Response
Burns/Williams, EI2014: Imaging Performance Measurement
22
In digital imaging two factors dictate the level of spatial resolution or detail that can ultimately be captured.
– Quantity: sampling frequency ( i.e. dots per inch, Mpixels )
• Sets the maximum resolution limit that can be achieved.
– Quality: SFR – Spatial Frequency Response
• Defines the real level of resolution that optics, environmental factors, hardware, and image processing impose on the captured image.
Resolution for Digital Capture Devices
Burns/Williams, EI2014: Imaging Performance Measurement
12
23
Point Spread Function and image sampling
• Several imaging mechanisms influence spread function descriptions
– Optical : focus, diffraction, aberrations, radiation scatter, filters
– Mechanical : sampling aperture size, vibration, motion
– Digital filtering : de-screening, unsharp masking
• Some do not *
– Image noise, film grain, detector noise
– Sampling artifacts
– Signal quantization and clipping (thresholding)
*...but make spread function and SFR measurements difficult
Burns/Williams, EI2014: Imaging Performance Measurement
24
Hubble Space Telescope Examplesame # pixels, different resolution
Before lens modification(oops !)
After lens modification(that’s better)
Burns/Williams, EI2014: Imaging Performance Measurement
13
25
2 – signal – MTF
Pro ConDirect
Sine waves Simple, intuitive analysis Large spatial extentLimited sampling
Costly capture targets
Indirect
Point/Line Spread Fundamental features Low signal strength
Function
Square Waves Simple targets Large spatial extent
Edge gradients Compact features Noise sensitiveSimple targets
Noise fields Small signal Spectrum estimation
MTF/SFR Measurement Techniques
Burns/Williams, EI2014: Imaging Performance Measurement
26
*Spatial Frequency Response
Spatial Frequency Response - SFR
Test Pattern
Incr
easi
ng s
patia
l fre
quen
cy
Lens
Imaged Test
Pattern
imagedlight
distribution
SFR or ( MTF)
Increasing spatial frequency
The relative change in contrast of increasingly finer spatial features
Burns/Williams, EI2014: Imaging Performance Measurement
14
27
Theory: Frequency response by edge-gradient analysis
Theory: Estimate the line spread function from an ideal edge measurement
DerivativeFourier
transform
Line spreadfunction
Modulationfunction
Edgeresponse
Fouriertransform
Line spreadfunction
Modulationfunction
ISO 15529 Method: Principles of measurement of MTF of sampled imaging systems
Technical background
Burns/Williams, EI2014: Imaging Performance Measurement
28
How is SFR measured for digital capture?
All you need is an image of a good slanted edge target.
Burns/Williams, EI2014: Imaging Performance Measurement
15
29
y = mx + bProjection of data along the edge effectively increases sampling.
x
signal
ISO method: Edge profile
Burns/Williams, EI2014: Imaging Performance Measurement
30
ISO Method: Resolution Charts
• OECF• SFR
Inputexposure
Outputsignal
2 – signal – targets and tools
Burns/Williams, EI2014: Imaging Performance Measurement
16
31
Different SFR shapes and their meaning
Limiting resolution will generallycorrespond to the spatial frequency having a 0.10 SFR level
Increasing spatial frequency. Finer detail
0.0
0.5
1.0SF
R re
spon
se
Sharpened
Normal
Low contrast - Flare
Burns/Williams, EI2014: Imaging Performance Measurement
32
How limiting resolution can fail to predict quality and why the SFR is better.
2 – signal – MTF
Burns/Williams, EI2014: Imaging Performance Measurement
SFRs for ISO 12233 TargetSame Resolution, Different SFRs
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Frequency (cyc/pixel)
Sp
atia
l Fre
qu
ency
Res
pon
se
right image middle image left image
17
33
Derivative Measures of SFR
• Color fringing vs. misregistration
• Sharpening
• Acutance
• Aliasing
• Flavors of resolution ( 10% SFR, 50% SFR, half -Nyquist)
• Flare
Burns/Williams, EI2014: Imaging Performance Measurement
34
Well-behaved SFRs
- Linear falloff to half-sampling frequency- Tightly grouped color SFRs- Similar form in both directions
Horizontal direction Vertical direction
Burns/Williams, EI2014: Imaging Performance Measurement
18
Burns/Williams, EI2014: Imaging Performance MeasurementBurns/Williams, EI2014 35
Anatomy of the SFR- strange, but explainable, shapes -
Typical Optical SFR- focal plane referred -
0
0.2
0.4
0.6
0.8
1
1.2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
cycles/pixel
SF
R
- half sampling - 166 cy/mm for 3 micron sensor250 cy/mm for 2 micron sensor
Typical Sharpening Filter SFR(e.g. sparsely populated 1x5 FIR filter )
0
0.5
1
1.5
2
2.5
3
3.5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
cycles/pixel
SF
RCombined delivered file SFR
Yes, these shapes are real and explainable !
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
cycles/pixel
SF
R
36
What’s the difference ?
• Resolution – the maximum spatial frequency of utility for an imaging system (limiting resolution).
• Sharpening- a class of image processing operations that enhances the contrast of selective spatial frequencies, usually visually important ones.
• Acutance – An integrated and weighted SFR ratio that is usually well correlated to sharpness.
• Sharpness – The visually perceived quality of being crisp or of containing detail.
Burns/Williams, EI2014: Imaging Performance Measurement
19
37
Sharpness vs. Resolution
0.5
1.0
0.1
0.25 0.5
Frequency, cy/pixel
SF
R
high sharpness, low resolution
0
0.5
1.0
0.1
0.25 0.5
low sharpness, high resolution
0.5
1.0
0.1
0.25 0.5
Frequency, cy/pixel
SF
R
high sharpness, low resolution
0
0.5
1.0
0.1
0.25 0.5
low sharpness, high resolution
0.5
1.0
0.1
0.25 0.5
Frequency, cy/pixel
SF
R
high sharpness, low resolution
0
0.5
1.0
0.1
0.25 0.5
low sharpness, high resolution
0.5
1.0
0.1
0.25 0.5
Frequency, cy/pixel
SF
R
high sharpness, low resolution
0
0.5
1.0
0.1
0.25 0.5
low sharpness, high resolution
Burns/Williams, EI2014: Imaging Performance Measurement
38
SFR and sampling efficiency
HVAC example
Advertised BTUs × Rated Efficiency = Usable BTUs
( 70K BTUs × 80% = 56K BTUs )the percentage of fuel that is converted to useable energy
Digital Camera example
# Delivered MPixels × Resolution Efficiency = # Effective MPixels( 6 MPixels × 80% = 4.8 Mpixels )
the percentage of optically resolvable pixels
Burns/Williams, EI2014: Imaging Performance Measurement
20
39
1- Rayleigh Criteria - Find frequency of first 10% SFR occurrence in both vertical and horizontal directions. ( clip at 0.5 cycles/pixel)
2- Efficiency- Normalize each by digital limit of 0.5 cycles/pixel.3- Take product of vertical and horizontal components.4- Multiply by camera’s finished file size
6.0 MPixel camera SFRs
0
0.2
0.4
0.6
0.8
1
1.2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
theo
retic
al m
axim
um f
requ
ency
A
B
C
D
Cameras A&B:
6.0 MP x (0.5/0.5)2 = 6.0 MP[ E=100% ]
Cameras C&D:
6.0 MP x (0.3/0.5)2 = 6.0 MP[ E = 36% ]
Sampling efficiency for digital cameras
Burns/Williams, EI2014: Imaging Performance Measurement
Burns/Williams, EI2014 40
Some Popular Sharpening Options
21
41
..and their SFR responses
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.1 0.2 0.3 0.4 0.5 0.6
relative frequency
SF
R
reference SFR moderate strong
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.1 0.2 0.3 0.4 0.5 0.6
relative frequency
SF
R
reference SFR moderate strongreference SFR moderate strong
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.1 0.2 0.3 0.4 0.5 0.6
relative frequency
SF
R
reference SFR moderate strong
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.1 0.2 0.3 0.4 0.5 0.6
relative frequency
SF
R
reference SFR moderate strong
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.1 0.2 0.3 0.4 0.5 0.6
relative frequency
SF
R
reference SFR moderate strongreference SFR moderate strong
SFRs from identically specified sharpness settingsRadius=0.5 ; Amount=350
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 0.2 0.4 0.6 0.8 1
Spatial frequency
SF
R
Software A Software B
SFRs from identically specified sharpness settingsRadius=0.5 ; Amount=350
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 0.2 0.4 0.6 0.8 1
Spatial frequency
SF
R
Software A Software B
SFRs from identically specified sharpness settingsRadius=0.5 ; Amount=350
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 0.2 0.4 0.6 0.8 1
Spatial frequency
SF
R
Software A Software B
SFRs from identically specified sharpness settingsRadius=0.5 ; Amount=350
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 0.2 0.4 0.6 0.8 1
Spatial frequency
SF
R
Software A Software B
SFRs from identically specified sharpness settingsRadius=0.5 ; Amount=350
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 0.2 0.4 0.6 0.8 1
Spatial frequency
SF
R
Software A Software B
Burns/Williams, EI2014: Imaging Performance Measurement
42
How to Measure Sharpness
• Via the SFR shape and magnitude
• Summary sharpness metrics such as
– Location/magnitude of maximum SFR value
• Visual weighting > acutance
• Sharpness = f(acutance)
0
)( )( dCSFSFRA
0
)( dCSFAr
CMT Acutance
rAAAcutance =
Visual contrast Modulation Transfer (CMT)
Cycle/degree
0
0.5
1
1.5
2
2.5
3
3.5
0 5 10 15 20 25 30 35 40 45
Burns/Williams, EI2014: Imaging Performance Measurement
22
43
SFR measurement
Challenges for today’s cameras• Degree of Over-sharpening• Level of SFR field non-uniformity• Aliasing• Texture
Proposed New ISO 12233 Resolution Targets – Rev. 2
CIPA – maximum resolutionNot recommended for SFR
OECF, Slanted EdgesEdge-based SFR Polar sine waves
Sine-based SFR
Burns/Williams, EI2014: Imaging Performance Measurement
44
SFR Summary
ISO SFR Method• Based on edge-gradient method• Intended for digital capture devices• Alignment immunity• Localized features• Open source software• Extendable to any angular direction• Ease of target generation; several imaging mechanisms or steps can be
described in terms of spread functionsMTF and SFR
Spread functions and MTF indicate the imaging or transfer of signal detail and sharpness
A number of derivative metrics can be calculated from SFR for process control & monitoring as well as image quality.
For quality-assurance control charts, a good measure of image resolution is the SFR-based sampling efficiency.
Burns/Williams, EI2014: Imaging Performance Measurement
23
45
Dead-Leaves Texture MTF Measurement
Method is aimed at providing an effective MTF for image fluctuations (signals) that are influenced by adaptive or signal-dependent image processing
For example adaptive noise cleaning, which could leave edge untouched, but reduce detail in important ‘textured regions’
filterednoisy
Burns/Williams, EI2014: Imaging Performance Measurement
46
Dead-leaves target and amplitude spectrum
Noise-free target
Similar to image features Amplitude spectrum
Burns/Williams, EI2014: Imaging Performance Measurement
24
47
Dead-Leaves MTF Measurement
1. Transform the captured image array of the target field to one encoded as proportional to luminance.
2. Compute the power-spectral density as the square of the amplitude of the two-dimensional DFT of the array.
3. Divide this array, frequency-by-frequency, by the modeled spectrum for the specific target to yield a two-dimensional array as the square of the effective MTF.
4. Compute the square-root, frequency-by-frequency, of this vector.
5. Compute the one-dimensional MTF vector by a radial-average of this array
Burns/Williams, EI2014: Imaging Performance Measurement
48
Dead-leaves target and amplitude spectrum
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-1.5
-1
-0.5
0
0.5
1
cy/pixel
Log
spec
trum
radial average spectrum
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-1.5
-1
-0.5
0
0.5
1
cy/pixel
Log
spec
trum
x
ydiagonal
Cross-sections
• Single result indicates NPS estimation variation
• Cross-section results can indicate direction differences
Burns/Williams, EI2014: Imaging Performance Measurement
25
49
NOISE
Burns/Williams, EI2014: Imaging Performance Measurement
50
Taxonomy - Noise
Burns/Williams, EI2014: Imaging Performance Measurement
26
51
Image Noise
• Noise is a general term applied to any response that detracts from a desired signal. Usually, errors or unwanted fluctuations in images. Given that it can have many sources, it can take several forms.
• Random noise due to film grain or low exposure to a detector. Often looks like “salt and pepper” noise.
• Often reported as an RMS number (standard deviation)
• Structured banding
• Compression artifacts
Burns/Williams, EI2014: Imaging Performance Measurement
52
Two-Dimensional Random: Example
025.0R
Fine-grain noise ->high frequency fluctuations
R = print reflectance [0-1]
025.0R
Course-grain noise ->low frequency fluctuations
3 – noise
Burns/Williams, EI2014: Imaging Performance Measurement
27
53
Noise
All of these patches have the same average and standard deviation
Burns/Williams, EI2014: Imaging Performance Measurement
54
ISO/TC42 Standards for Digital Imaging Noise Metrology
ISO 15739 – Noise for Digital Still CamerasISO 21550 – Dynamic Range for Digital Film Scanners
Commonalities
• Assume an additive noise model; σ2total = σ2
fixed pattern + σ2temporal
• Attempt to discriminate noise contributions with replicate images
• Requires OECF measurement for scene referred metrology
• Dynamic Range reported as a linear ratio, not decibels (dB)
Differences
15739 21550
• Focuses on noise metrology • Focuses on dynamic range metrology
• Uses peak-to-peak definition • Uses Incremental signal definition for
of signal for dynamic range dynamic range calculation
• Provides for frequency weighted • Assumes 100% maximum transmission
visual noise output (informative)
• Assumes 140% luminancemax
• Photoshop plug-in available athttp://www.i3a.org
3 – noise
Burns/Williams, EI2014: Imaging Performance Measurement
28
55
Dynamic Range – Cameras and Scanners*
The extent of energy over which a digital capture device can reliably detect signals: reported as either a normalized ratio (xxx:1) or in equivalent optical density units, , R= reflectance.
• ISO 21550 – Dynamic Range for Digital Film Scanners• ISO 15739 – Noise Measurement for Electronic Still Pictures Cameras
Signal Detection → OECF
The mean (average) change in count level associated with a particular input density level.
Reliability of Detection → Noise
The root mean square (std. dev.) count level associated with a particular input density level
High noise ≡ low reliability
Noise
Signal
3 – noise – dynamic range
RD 10log
Burns/Williams, EI2014: Imaging Performance Measurement
56
Incremental Signal-to-Noise Ratio – Scanners
Incremental Signal
0
20
40
60
80
100
120
140
160
180
0 0.5 1 1.5 2 2.5 3 3.5 4
Density
Incr
emen
tal S
igna
l Incremental SNR
0.1
1
10
100
1000
0 0.5 1 1.5 2 2.5 3 3.5 4
Density
Incr
emen
tal S
NR
Measured RMS noise (CV)for film scanner
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 0.5 1 1.5 2 2.5 3 3.5 4
Density
RM
S no
ise
(CV
)
3 – noise – dynamic range
Burns/Williams, EI2014: Imaging Performance Measurement
29
57
Measurement Requirements
Measurements usually require some level of both accuracy and precision.
– Accuracy: average error from a standard
– Precision: variability about the average readingIs your watch set to the correct time?
Factors that influence measurements
– Test target feature location estimation
– Signal processing
– Signal quantization
– Spatial sampling
– Image noise
XXXX
X bias
X
X
X X
X
variation
Burns/Williams, EI2014: Performance Variation
58
Variation
Actual performancee.g., Optical quality across the image field, or with lens zoom position
Measurement errorEstimation of sample statisticse.g. due to signal quantization, sampling
Sampling:variation in estimating parameters (statistics) from test images
Source of imaging performance variability for a digital camera?• Scene exposure.• Image compression parameters.• Camera shake.
Burns/Williams, EI2014: Performance Variation
30
59
Measurement error example
• Simplest way to quantify measurement variation is by repeating the analysis• This edge SFR example is for one test file and one edge, varying the
location of the region of interest, N = 10
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Frequency, cy/pixel
SF
R
Single file, edge (N=10)
ave
+1 std
-1 stdmax
min
Burns/Williams, EI2014: Performance Variation
60
ISO Protocols vs. Specifications:
ISO Imaging Performance provide protocols on how and what to measure with respect to imaging performance ( e.g. tone, noise, SNR, Resolution, etc.) but …..
The acceptable levels and tolerances for each of those is left to the user because they are largely use-case dependent. Resolution and color aims and tolerances could be quite different whether one is doing cell phone vs. portrait photography
Useful features in software for imaging performance evaluation• Image metadata (header and tag) reading.• Use with multi-attribute test targets.• Testing of results against specified tolerance values.
How & What vs. Pass or Fail
Burns/Williams, EI2014: Performance Variation
31
61
Test plans – just a few words
• Don’t collect too much data
– Limit the experiment
• Control the variables
– Operator, lighting, zoom position, image processing
• Distinguish benchmarking from monitoring
• Distinguish capability from performance
• Be reasonable with test targets designs
– Any target can be designed to make a camera look bad
• Test under conditions that match expected usage.
• Prioritize the metrics exposure, SFR, noise/artifacts
• For quality-assurance control charts, a good measure of image resolution is the SFR-based sampling efficiency (See Appendix VI)
Burns/Williams, EI2014: Performance Variation
62
ISO TC42/WG18 Standards Rodeo
PWI - Preliminary Work Item
NP– New Proposal
AWI - Approved new Work Item
WD - Working Draft
CD - Committee Draft
FCD - Final Committee Draft
DIS - Draft International Standard
FDIS - Final Draft International Standard
PRF - Proof of a new International Standard
IS - International Standard
Periodic Review and updates
Stages of an ISO standard
32
Burns/Williams, EI2014 63
ISO TC42/WG18 Standards Rodeo
PWI - Preliminary Work Item
NP– New Proposal
AWI - Approved new Work Item
WD - Working Draft
CD - Committee Draft
FCD - Final Committee Draft
DIS - Draft International Standard
FDIS - Final Draft International Standard
PRF - Proof of a new International Standard
IS - International Standard
Periodic Review and updates
Stages of an ISO standard
Burns/Williams, EI2014 64
ISO TC42/WG18 Standards Rodeo
TC42 – PhotographyWG18 – Digital Cameras
12232 Determination of exposure index, ISO speed ratings, standard output sensitivity, and recommended exposure index
14524 Methods for measuring opto‐electronic conversion functions (OECFs)
12233 Electronic still‐picture cameras ‐‐ Resolution measurements
16067-1 Reflective Scanners – Resolution measurements
16067-2 Film Scanners – Resolution measurements
21550 Film scanners ‐ Dynamic range measurements
15739 Noise
The usual suspects for imaging performance
33
Burns/Williams, EI2014 65
12231 - Vocabulary
15781 ‐Measuring shooting time lag, shutter release time lag, shooting rate, and start‐up time
12234 ( multiple parts) ‐ Removable memory
20462 ( multiple parts) ‐ Psychophysical experimental methods for estimating image quality
22028 ‐ Extended colour encodings for digital image storage, manipulation and interchange
IEC 61966 ‐ Colour measurement and management
Important, but not for imaging performance
Burns/Williams, EI2014 66
18841 - Flare measurement techniques for digital camera systems (AWI)
19093 ‐Measuring low light performance (NP)
19567 ‐ Low contrast fine detail measurements
TS 19247 ‐ Guidelines for reliable camera testing
17321 (Parts 1,2 3,& 4) ‐ Colour characterisation of digital still cameras
17850 ‐ Geometric distortion measurements
17957 ‐ Shading measurements
18383 ‐ Specification guideline
19084 ‐ Lateral chromatic displacement measurement
New Kids on the Block
Under development
34
Burns/Williams, EI2014: Standards Rodeo 67
What’s missing ?
Depth of Field / Focus ?
White Balance ?
Sharpening ?
Aliasing Potential ?
Suggestions?
68
ISO TC42/WG18 Standards Rodeo
TC42 – PhotographyWG18 – Digital Cameras
12232 Determination of exposure index, ISO speed ratings, standard output sensitivity, and recommended exposure index
14524 Methods for measuring opto‐electronic conversion functions (OECFs)
12233 Electronic still‐picture cameras ‐‐ Resolution measurements
16067-1 Reflective Scanners – Resolution measurements
16067-2 Film Scanners – Resolution measurements
21550 Film scanners ‐ Dynamic range measurements
15739 Noise
The usual suspects for imaging performance
Burns/Williams, EI2014: Standards Rodeo
35
69
12231 - Vocabulary
15781 ‐Measuring shooting time lag, shutter release time lag, shooting rate, and start‐up time
12234 ( multiple parts) ‐ Removable memory
20462 ( multiple parts) ‐ Psychophysical experimental methods for estimating image quality
22028 ‐ Extended colour encodings for digital image storage, manipulation and interchange
IEC 61966 ‐ Colour measurement and management
Important, but not for imaging performance
Burns/Williams, EI2014: Standards Rodeo
70
18841 - Flare measurement techniques for digital camera systems (AWI)
19093 ‐Measuring low light performance (NP)
19567 ‐ Low contrast fine detail measurements
TS 19247 ‐ Guidelines for reliable camera testing
17321 (Parts 1,2 3,& 4) ‐ Colour characterisation of digital still cameras
17850 ‐ Geometric distortion measurements
17957 ‐ Shading measurements
18383 ‐ Specification guideline
19084 ‐ Lateral chromatic displacement measurement
New Kids on the Block
Under development
Burns/Williams, EI2014: Standards Rodeo
36
71
What’s missing ?
Depth of Field / Focus ?
White Balance ?
Sharpening ?
Aliasing Potential ?
Suggestions?
Burns/Williams, EI2014: Standards Rodeo
Burns/Williams, EI2014 72
Where do the Standards align
Burns/Williams, EI2014: Introduction
37
Burns/Williams, EI2014 73
Wrap Up
1. Technology used influences imaging performance
2. Imaging performance measurement is standardized for
– Large area image capture (OECF)
– Image detail capture (SFR and MTF)
– Image noise and artifacts (rms, dynamic range)
3. Visual inspection is often needed for artifacts
4. Understanding of technology modes helps interpretation of results. Area arrays, image formatting, software settings, etc.
5. Useful features in software for imaging performance evaluation
• Image metadata (header and tag) reading.
• Use with multi-attribute test targets.
• Testing of results against specified tolerance values.
Send questions to Don atinfo@imagescienceassociates.com
Or Peter atinfo@burnsdigitalimaging.com
Burns/Williams, EI2014 74
Appendix I: Important Imaging Characteristics- an imaging performance taxonomy for classifying the metrics -
Burns/Williams, EI2014: Introduction
38
Burns/Williams, EI2014 7575
Color – Its just an encoding
Same colors – different encoding
Appendix II: Digital Cameras do not Reproduce Colors, they Encode them.
Reproduction requires interpreting (decoding) the pixel values to display the image for particular audience or purpose.
Burns/Williams, EI2014 76
Appendix III: Spatial Frequency Units
pixel
cyN
ph
pixel
pixel
cy
ph
lp Npixelsph
line-pairs/picture height – Video, digital systems4 -
3 – cycles/pixel – digital processing, filter design
5 – line widths per picture height, LW/PH – Video,12233 standard
6 – cycles/degree (CPD) – visual acuity, image quality
mm
cy
mm
lp
mm
cy
mm
lp
mm
l
22lines/mm = or
cycles/mm, cy/mm, mm-1 - Engineering, film photography
line-pairs/mm - Graphics
1 -
2 - 1 line width
1 line pair
39
Burns/Williams, EI2014 77
Frequency Unit Conversion Table
LW/PH LP/mm L/mm Cycles/mm Cycles/pixel
LW/PH1
⁄ [ 2 picture height (mm)]
⁄ picture height (mm)
⁄ [ 2 picture height (mm)]
⁄ [ 2 # vert. pixels ]
LP/mm [2 picture
height (mm)]
1 2.0 1 pixel pitch (mm)
L/mm picture height
(mm) 0.5 1 0.5
[pixel pitch (mm) ⁄2]
Cycles/mm [2 picture
height (mm)]
1 2.0 1 pixel pitch (mm)
Cycles/pixel [2 # vert.
pixels] ⁄ pixel pitch
(mm)
[2 ⁄ pixel pitch (mm)]
⁄ pixel pitch (mm) 1
LW/PH = Line width per picture heightLP/mm = Line pairs per millimetreL/mm = Lines per millimetreTo convert from left column units to top row units, use operation at their row/column intersection.(e.g., 5 LP/mm 2.0 = 10.0 L/mm )
Burns/Williams, EI2014 78
Appendix IV: ISO 12233, SFR and MTF
The ISO standard for measuring digital still camera resolutions specified a particular method, based on a modified form of edge gradient analysis (EGA) The ISO standard, however, refers to the camera resolution metric as
Spatial Frequency Response (SFR), rather than an MTF There are three basic reasons for this
– Established photographic standards measure MTFs using other methods (sine waves), and the results often differ from EGA results
– The term MTF has been associated with the measurement of systems or subsystems* for which it (approximately) uniquely describes the signal transfer from input to output). For many electronic imaging systems, however, the results will vary with system and image conditions.
– EGA measurement of a system MTF requires compensation for the input target edge characteristics. While this is possible, the original ISO standard did not require it.
___________________
* Linear systems, or those with combinations of linear and point-wise nonlinear subsystems
40
Burns/Williams, EI2014 79
Appendix V: Limiting Resolution Estimation from SFR
EvaluateSFR
ii SFRf ,Find criticalfrequency
pp SFRf where,
InterpolateSFR
'whereSFR
test file,critical value, f
EvaluateSFR
ii SFRf ,Find criticalfrequency
pp SFRf where,
InterpolateSFR
'whereSFR
test file,critical value, f
Burns/Williams, EI2014 80
Two dimensional example
Example of two-dimensional symmetric SFR and corresponding 50% and 10% response contours
41
Burns/Williams, EI2014 81
Appendix VI: ISO Camera Resolution Efficiency Rating Proposal
Electronic still picture cameras are normally marketed using a “megapixel” value, which normally provides thenumber of effective pixels of the image sensor. This can be confusing or misleading, since it is a value relatedto the number of addressable photoelements on the image sensor, and is not a based on any type of resolutionmeasurement.
There is interest in developing a single-value resolution numeric for reporting the measured resolution of electronicstill picture cameras. One proposed metric is the “resolution efficiency rating.”
1. Determine the maximum resolution in LW/PH for the horizontal (RH), vertical (RV), and +/- 45° (R+45 , R-45 )directions.2. Calculate individual directional efficiencies ( EH , EV , E+45 , E+45 ) by normalizing the maximum resolutions of
item #1 by the captured image’s picture height. If any normalized value is greater than 1.0, assign that value to 1.0.3. Combine E+45 and E-45 efficiencies into an equally weighted average diagonal value ED. 4. Calculate the resolution efficiency rating (ER) as the product of 100, ED and the average of EH and EV.
Resolution Efficiency Rating ER = 100 ( ED (EH + EV) /2 )
Example using the CIPA tool: 2048 pixel high x 3072 pixel wide, 6.0 MPixel camera file
Maximum horizontal resolution : RH = 1970 LW/PH Horizontal Resolution Efficiency: EH= 1970/2048 = 0.96Maximum vertical resolution : RV = 1980 LW/PH Vertical Resolution Efficiency: EV = 1980/2048 = 0.97Maximum +45° resolution : R+45 = 1500 LW/PH Diagonal Resolution Efficiency: ED = (1500 + 1500 )/2 = 0.73Maximum -45° resolution : R-45 = 1500 LW/PH
Resolution Efficiency Rating = 100 [(0.73 (0.96 + 0.97)) /2 ] = 70.6
Burns/Williams, EI2014 82
Sampling efficiency Summary
• Sampling efficiency measure is considered as an extension of thecurrent ISO 12233 standard revision effort.
• Based on the ratio of a 10% SFR-spatial frequency bandwidth to thebandwidth implied by the image sampling alone
• Not intended to include the influence of sampling artifacts and imagenoise
• Provides guidance when considering the level of image signal detaillikely to be delivered by a camera
• Sampling efficiency provides a convenient factor to adjust advertisedvalues to yield an effective megapixel value.
top related