Colour Image Processing Allan Hanbury PRIP, Vienna University of Technology Favoritenstraße 9/1832 A-1040 Vienna, Austria [email protected] http://www.prip.tuwien.ac.at/~hanbury
Jan 20, 2016
Colour Image Processing
Allan Hanbury
PRIP, Vienna University of TechnologyFavoritenstraße 9/1832A-1040 Vienna, Austria
[email protected]://www.prip.tuwien.ac.at/~hanbury
Contents
• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images
Contents
• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images
Physical Background
• Visible light: a narrow band of electromagnetic radiation → 380nm (blue) 780nm (red)
• Wavelength: Each physically distinct colour corresponds to at least one wavelength in this band.
• Pure Colours: Pure or monochromatic colours do not exist in nature.
• Spectrum: Intensity asa function of wavelength.
• The colour of an object: is the product of the spectrum of the incident light with the light absorption and/or reflection properties of the object.
From http://fuse.pha.jhu.edu/~wpb/spectroscopy/basics.html
Human colour perception
• The human eye does not perceive individual light wavelengths.
• It contains three types of colour receptor (cones) which integrate over parts of the spectrum:
From http://math.ucr.edu/home/baez/physics/General/BlueSky/blue_sky.html
• It is therefore possible to characterise a psycho-visual colour by specifying the amounts of three primary colours: red, green and blue, mixed together.
• This leads to the standard RGB space used in television, computer monitors, etc.
• We specify the levels of R, G and B in the range [0, 1], but they can easily be extended to other ranges (8-bit integers for example).
(0,0,0)
(1,1,1)
RGB
Problems with Processing Colour Images
• When processing colour images, the following problems (amongst others) have to be dealt with:– The images are vectorial → 3 numbers are associated with
each pixel.
– The colours recorded by a camera are heavily dependent on the lighting conditions.
Lighting conditions
Image taken lit by a flash. Image taken lit by a tungsten lamp.
• The lighting conditions of the scene have a large effect on the colours recorded.
• The following four images of the same scene were acquired under different lighting conditions:
Dealing with Lighting Changes
• Knowing just the RGB values is not enough to know everything about the image.– The R, G and B primaries used by different devices are
usually different.
• For scientific work, the camera and lighting should be calibrated.
• For multimedia applications, this is more difficult to organise:– Algorithms exist for estimating the illumination colour.
Contents
• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images
Processing Vectorial Images• A vectorial image has a vector at each pixel. For
colour images, these vectors each have 3 components.• Vectorial images with larger numbers of components
also exist, e.g. in satellite imagery.• There are two ways one can process vectorial images:
– Marginal processing.
– Vectorial processing.
f (x, y) {0,1,…, N} f (x, y) [ {0,…,N}, {0,…,N}, {0,…,N} ]
Gre
ysca
le
Co
lou
r
Scalar process.
Red
Scalar process.
Green
Scalar process.
Blue
Red
Green
Blue
Marginal Processing
• Each channel is processed separately:
Red
Green
Blue
Vectorial process.
Vectorial Processing
Red
Green
Blue
• The colour triplets are processed as single units:
The Problem of False Colours
FalseColours !!
Marginal Median
VectorialMedian
The Problem of False Colours
0B
0V
100R
0B
100V
0R
100B
100V
100R
Marginalmedian
0B
100V
100R
Vectorialmedian
0B
100V
0R
Contents
• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images
Alternative Colour Spaces
• Various other colour representations can be calculated from the RGB representation.
• This can be done for:– Decorrelating the colour channels:
• principal components.
– Bringing colour information to the fore:• Hue, saturation and brightness.
– Perceptual uniformity:• CIELuv, CIELab, …
Red
Green
Blue
T-1
Pro
cess
ing
Processing Strategy
T
Red
Green
Blue
Colour spaces
• RGB (CIE), RnGnBn (TV - National Television Standard Comittee)• XYZ (CIE)• UVW (UCS de la CIE), U*V*W* (UCS modified by the CIE)• YUV, YIQ, YCbCr• YDbDr• DSH, HSV, HLS, IHS• Munsel colour space (cylindrical representation)• CIELuv• CIELab• SMPTE-C RGB• YES (Xerox)• Kodak Photo CD, YCC, YPbPr, ...
Contents
• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images
3D-polar Coordinate Colour Spaces• These spaces use a cylindrical (3D-polar) coordinate system to
encode the following three psycho-visual coordinates:– Hue (dominant colour seen)
• Wavelength of the pure colour observed in the signal.
• Distinguishes red, yellow, green, etc.
• More the 400 hues can be seen by the human eye.
– Saturation (degree of dilution)• Inverse of the quantity of “white” present in the signal. A pure colour has
100% saturation, the white and grey have 0% saturation.
• Distinguishes red from pink, marine blue from royal blue, etc.
• About 20 saturation levels are visible per hue.
– Brightness• Amount of light emitted.
• Distinguishes the greylevels.
• The human eye perceives about 100 levels.
3D-polar Coordinate Colour Spaces• The transformation of the RGB colour space to a hue, saturation
and brightness colour space is essentially a conversion from a set of rectangular coordinates to a set of (3D-polar) cylindrical coordinates.
H
HSV
IHS
HSI
HLS
triangle• Yet there are many such spaces described in books.
• How does one choose which one to use?
Idea Behind the Transformation• In the RGB space, the vectors [R, G, B] specify the amount of each
red, green and blue primary in the colour.– For convenience, we take R, G, B [0, 1]– The valid coordinates form the RGB cube [0, 1] [0, 1] [0, 1].
• The basic idea behind the transformation to the hue, saturation and brightness coordinate system is to place a new axis between [0, 0, 0] and [1, 1, 1], and to specify the colours in 3D-polar coordinates based on this axis.– This new axis passes through all the grey points (R = G = B), so we call it the
achromatic axis.
Achromatic axis
Brightness
Hue
H
Saturation
[0,0,0]
[1,1,1]
Achro
mati
c
axis
Basic Problem• Many of the spaces were originally developed for
easy numerical specification of colours in computer graphics applications.
• Due to its brightness function, the “natural” shape of the HSV colour gamut is a cone.
• However, with this shape, there are many coordinates which are not valid.
• In order to avoid costly verification of the validity of specified coordinates, these gamuts were often artificially expanded into cylinders.
• These cylindrical spaces have often been unwittingly carried over into image processing applications.
Con
ic H
SV
Cyl
indr
ical
HS
V
Achromatic axis
H=0°H=180°
Standard RGB → HSV Transform• max = sup(R, G, B) min = inf(R, G, B)• L = max
•
•
• If Httt
•
H = Ht
otherwise0
0 if maxmax
minmaxS
maxB
maxG
maxR
H
minmaxGR
minmaxRB
minmaxBG
t
if4
if2
if
• The same problem occurs for the HLS transformation.
• Here the double cone has been artificially expanded at both ends.
Con
ic H
LS
Cyl
indr
ical
HL
S
H=0°H=180°
Standard RGB → HLS Transform
• max = sup(R, G, B) min = inf(R, G, B)
•
•
•
• If Httt
•
H = Ht
2minmaxL
21
2
21
if
if
L
LS
minmaxminmax
minmaxminmax
c
maxB
maxG
maxR
H
minmaxGR
minmaxRB
minmaxBG
t
if4
if2
if
R, G and B are between 0 and 1.
Removal of the Brightness Dependence of the Saturation (1)
• HSV model
• But the HSV brightness LHSV = max(R, G, B)
• So the saturation with the brightness-dependence removed is
otherwise0
0,,max if,,max
,,min,,max
HSV
BGRBGR
BGRBGRS
BGRBGRS ,,min,,maxNCHSV
Removal of the Brightness Dependence of the Saturation (2)
• HLS model
• where the brightness
otherwise
,,min,,max2
,,min,,max
if,,min,,max
,,min,,max,,min,,max if0
21
HLSHLS
BGRBGR
BGRBGR
LBGRBGR
BGRBGRBGRBGR
S
2
,,min,,max BGRBGRLHLS
• The brightness dependence is removed by using
• to give
HLSLSS 21
HLSNCHLS 21
BGRBGRS ,,min,,maxNCHLS
Same as the non-cylindrical saturation obtained for HSV!
• Consider the colour image below. Not all the pixels which appear white have RGB coordinates of exactly [1, 1, 1] (similar for black pixels).
“Le chanteur”, Joan Mirò (bottom half inverted)
• These small variations are amplified in the saturation images due to the expansion of the cones into cylinders.
HSV cylindrical saturation
HLS cylindrical saturation
• The advantage of this measure of saturation is visible on the example image.
Le chanteur, Joan Mirò (bottom half inverted)
HSV cylindrical saturation
Compare to
Proposed Saturation
Advantages of the IHLS space
• This is the Improved HLS space.• It has the following advantages:
– The saturation is low for black and white pixels.
– The brightness is independent of the saturation (can be shown mathematically). This means that you are free to choose any brightness, luminance or lightness function.
– The saturation values can be easily compared. This is important for mathematical morphology operators.
Transformation into the IHLS space
• The full transformation is implemented efficiently as:
BGRY 0722.07152.02126.0 Or your own favourite brightness expression. BGRBGRS ,,min,,max
21
222
21
21
arccosBGRBRGBGR
BGRH
otherwise
if360
H
GBHH
Trigonometric version is more accurate.
Contents
• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images
Processing and Analysing Colour Images
• Some applications which take advantage of the good properties of the IHLS space are presented.
• The following are discussed:– Hue statistics.
– Mathematical morphology on colour images.
C
han
ge o
f re
pre
sen
tati
on
R
G
B
H
L
S Im
ag
e P
roce
ssin
g o
r A
naly
sis
Basic Idea
Chan
ge o
f re
pre
sen
tati
on
H
L
S
R
G
B
Processing the Hue Component
Colour image – “The virgin”, P. Serra (16th century)
The reds and violets are separated by a large discontinuity, even though they look similar.
Histogram of the hue
Hue component
Hue Statistics• For the brightness and the saturation, one can use standard
statistical methods for calculating the mean, standard deviation, etc.
• For data ai (i = 1, 2, …, n) distributed on the unit circle, the mean direction is that of the resultant vector obtained by adding unit vectors with directions ai.
• A measure of the variation in the directions of the data is given by the length of this vector divided by n (the mean length), which has the following characteristics:– range [0:1]– Values close to 1 the data is less spread out.
n = 3
Mean
• Given n values of the hue Hi
• The mean direction H is calculated as follows:
(1)
(2)
• The mean length R is:
(3)
n
ii
n
ii BARHBHA
1
222
1
,sin,cos
0,0if2arctan
0ifarctan
0,0ifarctan
AB
A
AB
H
AB
AB
AB
n
RR
Hue Mean which Takes the Saturation Into Account
• The previous formulation is standard in the texts on circular statistics, but it ignores the fact that not all hues have the same importance.
• We take this into account by weighting the length of each hue vector by the associated saturation value.
n = 3
• Let Si be the saturation associated with hue Hi.• We replace equation (1) by:
• To calculate the saturation-weighted mean direction HS, we replace A and B by AS and BS in equation (2).
• Equation (3) becomes:
• Note that RS remains a measure of the angular dispersion, and does not give information on the mean of the saturation.
n
iSSSii
n
iSiiS BARHSBHSA
1
222
1
,sin,cos
n
i i
SS
S
RR
1
ExampleC
olou
r Im
age
Hue
Satu
rati
on
Hue histogram
Saturation histogram
• HS = 20°• RS = 0.16W
eigh
ted
Hue threshold
0° - 40°
• H = 327°• R = 0.24
Hue threshold
307° - 347°
Mathematical Morphology in a 3D-Polar Coordinate Colour Space
• One can easily order the brightness and saturation values.
• On the other hand, the hue is defined on the circle, for which there is no obvious order (blue larger than red?, green smaller than red? ).
• Applying morphological operators to the brightness and saturation is therefore easy. – We just have to be sure that we order the vectors and not the
components (i.e. marginal order), so as to avoid introducing false colours.
– For this purpose we use the lexicographical order.
• Applying mathematical morphology to the hue is trickier, and won’t be discussed in this talk.
0 0
1 1
L S
H 0°
Lexicographical Order• This is the order in which words are arranged
in a dictionary.
Aardvark
Abacus
Abandon
.
.
.
Borough
Bough
.
.
Lexicographical Order for Vectors• For example, given two vectors x = (x1, x2, x3) and y = (y1, y2, y3):
• The lexicographical order is a total vector order. This means:– There are no pairs of vectors for which the order is unknown.
– The maximum and minimum of a set of vectors is always part of the set no false colours!
• The disadvantage is that one of the vector components has to play a dominant role.
332211
2211
11
andand
or
and
or
if
yxyxyx
yxyx
yx
yx
Angular Distance
• To enable us to work on the circle, we begin by defining an angular distance.
• Given a circle C with centre o
• We choose an arbitrary origin a0 on the circle. The points ai are then described by their curvilinear coordinate between 0 and 2 starting at a0.
• Given two points a and a, the size of the acute angle aoa is
a a
a a
aaaa
aaaaaaaaD
if2
if,
o
Lexicographical Order with Brightness at the Top Level (1)
• We define
• and
00andand
or
and
or
if
HHHHSSLL
SSLL
LL
jijiji
jiji
ji
ji cc
00andand
or
and
or
if
HHHHSSLL
SSLL
LL
jijiji
jiji
ji
ji cc
Lexicographical Order with Brightness at the Top Level (2)
• H0 is an arbitrary parameter. It does not have a major effect on the results, as, being at the third level, it is almost never taken into account.
• The erosion is defined in the standard way as follows:
f (x) = {f (y) : f (y) = inf [f (z)] , z Bx }
and the dilation:
f (x) = {f (y) : f (y) = sup [f (z)] , z Bx }
Original Image (293 418)
Ero
sion
Dila
tion
Ope
ning
Clo
sing
ExampleBrightness at the
top level
SE: Square of size 5 5
Lexicographical Order with Saturation at the Top Level
• We define
• and
00andand
or
and
or
if
HHHHLLSS
LLSS
SS
jijiji
jiji
ji
ji cc
00andand
or
and
or
if
HHHHLLSS
LLSS
SS
jijiji
jiji
ji
ji cc
Original image (293 418)
ExampleSaturation at the
top level
Ero
sion
Dila
tion
Ope
ning
Clo
sing
SE: Square of size 5 5
Colour Top-Hat
• We can define a top-hat operator for colour images similar to the one used for greyscale images.
• We simply calculate the Euclidean distance E between each corresponding pixel of the original image I and the result of one of the colour opening or closing operators:– Opening top-hat:
– Closing top-hat:
IIE ,
IIE ,
We wish to extract the greyish lines in between the mosaic tiles
Example
These lines are visible in the saturation image
Original Image
A colour morphological closing using a lexicographical order with saturation at the first level
Saturation
Top-hat operator. The pixel-by-pixel Euclidean distance between the two colour images to the left.
Summary
• A 3D-polar coordinate representation of the RGB colour space can be very useful in image processing and analysis.
• One should be careful with how the saturation is defined.
• Applications demonstrated are:– Colour statistics.
– Colour morphology.
• Further applications include:– Colour histograms.
– 2-dimensional brightness-saturation histograms (have been used for segmentation).
– …